🔀 MoE Expert Expansion — performance verification lab

Runtime expert expansion for Mixture-of-Experts models on ZeroGPU: instead of touching the weights, each token is routed through N expert slots instead of the native top-k (?), with extra ranks getting a linearly decaying influence (0.99 → D) before renormalization, optionally gated by an adaptive threshold (p ≥ T × p(rank N/2)). With expansion off the model is bit-identical to stock. Weights are loaded and quantized to 2-bit on the real GPU on first use.

📄 Paper: Agrillo, "Layer-Scoped Expert-Budget Expansion … on MMLU-Pro", Zenodo 2026 — DOI 10.5281/zenodo.22255483 · Mechanism reference: llama.cpp/moe-expansion docs · Model: Qwen/Qwen3.6-35B-A3B

Model status: checkpoint downloading — press Initialize · Qwen/Qwen3.6-35B-A3B · quant: 2-bit

System prompt

Temperature

0 1.5

Max new tokens

64 1024

Top-p

0.1 1

Enable expert expansion

Off = bit-identical to stock routing

Budget N (experts/token)

--moe-experts N; must be ≥ native top-k

2 64

Adaptive threshold T

--moe-expert-threshold T; 0 = fixed N; T ≤ 1 only expands, T > 1 can prune down to max(1, N/4)

0 2

Decay end D

--moe-expert-decay-end D (default 0.50)

0.05 0.99

Apply influence decay

Unchecked = --moe-no-expert-decay (extras at full influence)

Renormalization

--moe-expert-renorm MODE

Layer start

--moe-expert-layer-start: <1 fraction of layers, ≥1 index (default: layer 30)

Layer end

--moe-expert-layer-end: <0 last, <1 fraction, ≥1 index (inclusive)

Chat

generation stats will appear here

Expansion is a routing-time modification: expert weights are never duplicated, startup stays light, and toggling expansion never reloads the model. ⚠️ For fixed-N routing, decode compute scales with N/K.