Entropy-Guided Dynamic Expert Selection in Mixture-of-Experts Models

Introduction

The pursuit of increasingly capable AI systems has driven exponential growth in neural network scale, with state-of-the-art language models now exceeding hundreds of billions of parameters [1]. This scaling trajectory, while yielding remarkable improvements in model capabilities, presents fundamental challenges in computational efficiency, energy consumption, and deployment feasibility. The Mixture-of-Experts (MoE) paradigm has emerged as a compelling architectural solution to these challenges, enabling dramatic increases in model capacity without proportional increases in computational requirements through the principle of conditional computation [2, 3].

The core intuition underlying MoE architectures is elegantly simple: rather than activating all parameters for every input, the network learns to route different inputs to different specialized sub-networks, termed "experts," based on the input characteristics themselves. This approach draws inspiration from cognitive science theories of modular brain organization [14] and has deep connections to ensemble methods in classical machine learning [15]. Modern instantiations of this principle, exemplified by architectures such as GShard [16], Switch Transformer [3], and Mixtral [4], have demonstrated that MoE models can achieve competitive or superior performance to computationally equivalent dense models while utilizing significantly more total parameters.

1.1 The Fixed-K Problem

Despite the success of MoE architectures, a critical limitation persists in contemporary implementations: the number of experts activated per token, denoted K, remains fixed during inference regardless of input characteristics. This design choice, while simplifying implementation and enabling efficient batched processing, represents a fundamental inefficiency when viewed through the lens of information theory. Consider the following illustrative scenario:

• High-confidence routing: For common, unambiguous tokens (e.g., function words like "the", "is", "and"), the router network typically exhibits strong preference for a single expert, with a sharply peaked routing probability distribution. In such cases, activating multiple experts provides minimal additional information while incurring substantial computational overhead.
• Low-confidence routing: For rare, ambiguous, or domain-specific tokens, the router may distribute probability mass more uniformly across multiple experts, indicating genuine uncertainty about the optimal routing decision. These tokens could benefit from aggregating information from multiple expert perspectives.

The fixed-K constraint forces the model to treat these fundamentally different scenarios identically, resulting in systematic inefficiency: computational resources are wasted on confident predictions while potentially under-allocated for uncertain ones. This observation motivates our central research question:

Research Question: Can we develop a principled, training-free method to dynamically select the number of active experts based on routing decision uncertainty, thereby achieving significant computational savings without degrading model quality?

1.2 Information-Theoretic Perspective

Our approach to addressing the fixed-K problem is grounded in information theory, specifically the concept of Shannon entropy as a measure of uncertainty [17]. The entropy of the routing distribution provides a natural, theoretically-motivated signal for routing decision "difficulty":

• Low entropy indicates the router has high confidence in its decision, with probability mass concentrated on few experts. The information content required to specify the routing decision is minimal.
• High entropy indicates uncertainty, with probability spread more uniformly across experts. More information (i.e., more expert activations) may be needed to adequately represent the token.

This perspective connects MoE routing to the broader framework of rate-distortion theory [18], which characterizes the fundamental trade-off between "rate" (computational resources expended) and "distortion" (deviation from optimal output). Adaptive-K routing can be understood as an algorithm for operating on the Pareto frontier of this trade-off, allocating computational resources where they provide the greatest marginal value.

1.3 Contributions

This paper provides the following contributions to the field of efficient neural network inference:

1. Theoretical Framework: We establish a rigorous information-theoretic foundation for entropy-guided expert selection, connecting routing entropy to optimal resource allocation via rate-distortion theory (Section 3).
2. Adaptive-K Algorithm: We propose a simple yet effective algorithm for dynamic K selection based on entropy thresholds, including both theory-based and data-driven calibration strategies (Section 4).
3. Comprehensive Empirical Validation: We evaluate our method on four production-scale MoE models spanning diverse architectural choices, demonstrating consistent compute savings of 24-33% without quality degradation (Section 5).
4. Ablation Studies: We conduct extensive ablation experiments examining threshold sensitivity, K-value granularity, layer-wise behavior, and domain transfer (Section 6).
5. Open-Source Implementation: We release a production-ready implementation compatible with major inference frameworks, facilitating adoption and further research (Section 8).

Background and Preliminaries

In this section, we establish mathematical notation and review the foundational concepts underlying Mixture-of-Experts architectures. Our goal is to provide sufficient depth for readers unfamiliar with MoE systems while establishing the precise formalism needed for our subsequent theoretical development.

2.1 Mixture-of-Experts Architecture

A Mixture-of-Experts layer consists of two primary components: a set of $N$ expert networks $\{E_1, E_2, \ldots, E_N\}$ and a gating network (router) $G$. Each expert $E_i: \mathbb{R}^d \rightarrow \mathbb{R}^d$ is typically a feed-forward network with identical architecture but independent parameters. The gating network $G: \mathbb{R}^d \rightarrow \mathbb{R}^N$ produces scores indicating the relevance of each expert for a given input.

2.2 Routing Mechanisms

The gating network produces unnormalized logits $g(x) = (g_1(x), \ldots, g_N(x))$ for each expert. These logits are typically computed via a linear projection:

where $W_g \in \mathbb{R}^{N \times d}$ is the gating weight matrix and $b_g \in \mathbb{R}^N$ is an optional bias term. The routing probability distribution is obtained via softmax normalization:

where $\tau > 0$ is a temperature parameter controlling distribution sharpness.

The top-K selection operation identifies the K experts with highest routing probabilities:

The final routing weights are obtained by renormalizing probabilities over selected experts only:

2.3 Shannon Entropy of Routing Distributions

The Shannon entropy of a discrete probability distribution quantifies its expected information content, or equivalently, the inherent uncertainty in the distribution [17]. For the routing distribution $p(x)$, entropy is defined as:

Routing entropy has the following important properties:

• Non-negativity: $\mathcal{H}(x) \geq 0$, with equality iff the distribution is deterministic
• Maximum entropy: $\mathcal{H}(x) \leq \log N$, with equality when distribution is uniform
• Concavity: Entropy is a concave function of the probability distribution

Table 1: Maximum entropy values for different expert counts. Higher N allows wider entropy range.

2.4 Computational Cost Model

To quantify computational savings from Adaptive-K routing, we establish a formal cost model. Let $C_E$ denote the computational cost (in FLOPs) of a single expert forward pass, and let $C_G$ denote the cost of gating computation. For standard top-K routing, the per-token cost is:

For Adaptive-K routing with variable $K(x)$, the expected cost is:

where $C_{\mathcal{H}}$ is the entropy computation overhead. Since entropy computation requires only $O(N)$ operations compared to $O(d^2)$ for expert forward passes (where $d \gg N$ typically), we have $C_{\mathcal{H}} \ll C_E$, making this overhead negligible in practice.

The relative compute savings are therefore:

Theoretical Foundations

In this section, we develop the theoretical foundations for entropy-guided expert selection. We begin by establishing connections to information theory, then present a rate-distortion theoretic analysis that motivates our algorithmic design.

3.1 Information-Theoretic Interpretation of Routing

We propose to view the MoE routing process through the lens of information theory. Specifically, we consider the router as an encoder that maps input tokens to expert activations, and the entropy of the routing distribution as the measure of "specification bits" required to describe the routing decision.

This proposition establishes that routing entropy is directly related to the intrinsic complexity of the routing decision. When entropy is low, the router has effectively "decided" on a small number of experts, and activating additional experts yields diminishing returns.

3.2 Rate-Distortion Theoretic Analysis

We formalize the relationship between computational cost and output quality using rate-distortion theory [18]. We define "rate" $R$ as the average number of experts activated, and "distortion" $D$ as the deviation from the output that would be obtained using all experts.

The rate-distortion function $R(D)$ characterizes the minimum rate (experts) required to achieve distortion at most $D$. While computing $R(D)$ exactly is intractable, we can establish the following relationship:

Intuitively, this proposition states that when the router is confident (low entropy), the output is well-approximated by a small number of experts. This provides theoretical justification for using entropy as the criterion for dynamic K selection.

3.3 Optimal K Selection

Given the entropy-distortion relationship, we can formulate the K selection problem as an optimization over entropy thresholds. Let $\mathcal{K} = \{k_1 < k_2 < \cdots < k_m\}$ be the set of allowed K values and $\Theta = \{\theta_1 < \theta_2 < \cdots < \theta_{m-1}\}$ be the entropy thresholds. The K selection function is:

The optimal thresholds minimize expected cost subject to a quality constraint:

In practice, we find that simple percentile-based heuristics work well, as detailed in Section 4.

Adaptive-K Routing Algorithm

4.1 Algorithm Description

Based on the theoretical foundations developed in Section 3, we present the Adaptive-K routing algorithm. The algorithm consists of three phases: (1) entropy computation, (2) K selection via threshold comparison, and (3) sparse expert execution with renormalized weights.

The algorithm has $O(N)$ overhead for entropy computation, which is negligible compared to the $O(Kd^2)$ cost of expert forward passes for typical transformer dimensions.

4.2 Threshold Calibration Strategies

The choice of entropy thresholds $\Theta$ determines the trade-off between compute savings and output quality. We propose two complementary strategies for threshold selection:

4.2.1 Theory-Based Thresholds

Based on maximum entropy $\mathcal{H}_{max} = \log N$, we can set thresholds as fractions of this theoretical maximum:

We recommend starting with $\alpha_1 = 0.5$ for binary K selection (K ∈ {1, 2}), corresponding to the point where the routing distribution has roughly half its maximum uncertainty.

4.2.2 Data-Driven Calibration

For optimal performance, thresholds can be calibrated on a representative calibration dataset:

1. Run inference on calibration set (1000-10000 samples recommended)
2. Collect routing entropy values for all tokens across all layers
3. Compute entropy percentiles (e.g., 25th, 50th, 75th)
4. Set thresholds at percentile boundaries corresponding to desired K distribution
5. Optionally fine-tune via grid search over threshold neighborhoods

Table 2: Comparison of threshold calibration strategies.

4.3 Batched Inference Considerations

Efficient GPU inference requires batched computation, which presents challenges when different tokens in a batch require different K values. We propose the following strategies:

4.3.1 Padded Batching

Compute maximum $K_{max}$ experts for all tokens, then mask out excess experts based on per-token K:

def adaptive_k_batched(router_logits, thresholds, k_values):
    # Compute entropy and K for each token in batch
    probs = F.softmax(router_logits, dim=-1)
    entropy = -(probs * torch.log(probs + 1e-9)).sum(dim=-1)
    
    # Determine K per token
    k_per_token = torch.full_like(entropy, k_values[-1], dtype=torch.long)
    for i, threshold in enumerate(thresholds):
        k_per_token = torch.where(entropy < threshold, k_values[i], k_per_token)
    
    # Get top-K_max experts
    k_max = max(k_values)
    topk_probs, topk_indices = torch.topk(probs, k_max, dim=-1)
    
    # Create mask based on actual K per token
    positions = torch.arange(k_max, device=probs.device).unsqueeze(0)
    mask = positions < k_per_token.unsqueeze(1)
    
    # Apply mask and renormalize
    masked_probs = topk_probs * mask.float()
    weights = masked_probs / (masked_probs.sum(dim=-1, keepdim=True) + 1e-9)
    
    return topk_indices, weights, k_per_token, entropy

4.3.2 Dynamic Grouping

For maximum efficiency with highly variable K values, tokens can be grouped by their K value and processed in separate batches. This increases sorting overhead but eliminates padding waste for workloads with bimodal K distributions.

Experimental Evaluation

5.1 Experimental Setup

5.1.1 Models

We evaluate Adaptive-K routing on four production MoE models representing diverse architectural choices and scales:

Table 3: Model configurations. Models span different expert counts (8-128), baseline K values (2-8), and total parameters (7B-47B).

5.1.2 Datasets

• WikiText-2 [19]: Standard language modeling benchmark (245K test tokens)
• Penn Treebank: Classic LM benchmark for perplexity evaluation
• MMLU [20]: 57-subject multiple-choice benchmark for knowledge evaluation
• HellaSwag: Commonsense reasoning benchmark
• C4-validation: Held-out web text for calibration

5.1.3 Metrics

• Perplexity (PPL): Exponential of cross-entropy loss; lower is better
• Average K: Mean number of experts activated per token
• Compute (%): Relative to baseline, computed as $\text{Avg K} / K_{\text{baseline}}$
• Downstream Accuracy: Task-specific accuracy on MMLU, HellaSwag

5.2 Entropy Distribution Analysis

Before presenting main results, we characterize the routing entropy distributions observed in each model. Understanding these distributions is essential for threshold calibration and interpreting savings potential.

Table 4: Entropy statistics across models. All models show significant entropy variance, with substantial fractions of low-entropy tokens suitable for reduced K.

5.3 Main Results

5.3.1 Mixtral 8×7B

For Mixtral with N=8 experts and baseline K=2, we use binary Adaptive-K with K ∈ {1, 2} and a calibrated threshold of θ₁ = 1.275 (corresponding to the 62nd percentile of observed entropy).

Table 5: Mixtral 8×7B results. Adaptive-K achieves 31.0% compute reduction with only 0.8% perplexity increase and no significant downstream degradation. Note that using constant K=1 significantly degrades quality, demonstrating the value of dynamic selection.

The K distribution shows 62% of tokens use K=1, while the remaining 38% use K=2. This distribution emerges naturally from entropy-based selection and correlates with token characteristics (common words → K=1, rare/technical terms → K=2).

5.3.2 Qwen1.5-MoE-A2.7B

Qwen1.5-MoE uses fine-grained experts (N=60) with higher baseline K=4. We use K ∈ {2, 3, 4} with thresholds θ = {1.8, 2.4}.

Table 6: Qwen1.5-MoE results: 32.4% compute reduction.

5.3.3 OLMoE-1B-7B

OLMoE uses many small experts (N=64) with high baseline K=8, representing an extreme point in the MoE design space. We use K ∈ {4, 6, 8} with thresholds θ = {2.5, 3.2}.

Table 7: OLMoE-1B-7B results: 24.7% compute reduction.

5.4 NVIDIA Nemotron 3 Nano (Validated January 2026)

Nemotron 3 Nano represents the most complex MoE architecture we tested: a Mamba2-Transformer hybrid with 128 routed experts + 1 shared expert (always active), top-6 routing, and 30B total parameters (3.5B active). We validated Adaptive-K on 2× NVIDIA A100 40GB GPUs via Vast.ai.

Technical Note: Since Nemotron 3 does not support output_router_logits=True, we extracted pre-top-K router logits via forward hooks on the backbone.layers.X.mixer.gate modules, computing full 128-expert logits as hidden_states @ router_weight.T.

Table 8: Nemotron 3 Nano validation results. Adaptive-K achieves 33.3% compute reduction by reducing average K from 6 to 4.

Note: Savings = 1 − (Projected K / Baseline K). With baseline K=6: Savings = 1 − 4/6 = 33.3%. Max entropy H_max = log₂(128) = 7.0 bits.

5.5 Results Summary

Table 9: Summary of Adaptive-K results across all models.

Ablation Studies and Analysis

6.1 Threshold Sensitivity

We investigate how sensitive Adaptive-K performance is to the choice of entropy thresholds. Experiments are conducted on Mixtral 8×7B with varying threshold values.

Table 10: Threshold sensitivity analysis on Mixtral. The calibrated threshold achieves optimal balance between savings and quality. Very aggressive thresholds yield diminishing returns with accelerating quality degradation.

Results reveal a clear Pareto frontier: lower thresholds increase K=1 usage and compute savings, but eventually degrade quality. The calibrated threshold (62nd percentile) sits at the "knee" of this curve, achieving near-maximum savings with minimal quality impact.

6.2 K-Value Granularity

We examine whether finer-grained K value sets improve performance compared to binary {1, 2} selection.

Table 11: K-value granularity analysis. Binary K-value selection achieves best efficiency. Finer granularity offers marginal quality improvements at significant efficiency cost.

6.3 Layer-wise Analysis

We analyze how routing entropy and K distribution vary across layers in Mixtral 8×7B, which has 32 MoE layers.

This layer-wise variance suggests potential for further optimization via per-layer threshold tuning, which we leave for future work.

6.4 Token Characteristics and K Selection

To understand what distinguishes K=1 tokens from K=2 tokens, we analyzed token characteristics:

Table 12: Token characteristic comparison. K=1 tokens are more common, simpler, and easier to predict—exactly what we expect from entropy-guided selection.

Related Work

7.1 Mixture-of-Experts Architectures

The Mixture-of-Experts concept was introduced by Jacobs et al. [21] and Jordan & Jacobs [22] in the early 1990s, originally as a supervised learning framework for decomposing complex problems. The application to large-scale neural networks was pioneered by Shazeer et al. [2], who demonstrated that sparsely-gated MoE layers could scale language models to unprecedented sizes.

Subsequent work has explored various aspects of MoE design:

• Expert architecture: GShard [16] and Switch Transformer [3] explored different expert sizes and configurations
• Routing mechanisms: Expert Choice [5] inverts the routing direction, having experts select tokens rather than vice versa
• Load balancing: Auxiliary losses [6] encourage balanced expert utilization during training
• Scaling laws: Recent work has characterized how MoE models scale differently from dense models [23]

Our work is orthogonal to these architectural innovations—Adaptive-K can be applied to any MoE architecture with a learned router that produces a probability distribution over experts.

7.2 Dynamic Computation Methods

The broader field of dynamic computation encompasses several related approaches:

• Early Exit [7]: Allows tokens to exit at intermediate layers when confidence is high, reducing depth-wise computation
• Adaptive Depth [8]: Learns to vary transformer depth based on token
• Speculative Decoding [24]: Uses a smaller draft model to propose tokens verified by the full model
• Token Pruning [25]: Removes unimportant tokens from intermediate representations

Adaptive-K is complementary to these approaches and could potentially be combined for multiplicative efficiency gains. For example, early exit could skip entire MoE layers while Adaptive-K reduces computation within executed layers.

7.3 Entropy-Based Methods in Deep Learning

Entropy has been used as a decision criterion in various deep learning contexts:

• Active learning uses entropy to identify informative samples [26]
• Confidence calibration relates output entropy to prediction reliability [27]
• Neural architecture search uses entropy for architecture selection [28]

To our knowledge, we are the first to use routing entropy for dynamic expert selection in MoE models.

Discussion

8.1 Broader Implications

The success of Adaptive-K routing has several implications for the MoE research community:

1. Fixed-K is suboptimal: The significant savings achieved without quality loss suggest that fixed-K routing leaves substantial efficiency on the table. Future MoE training procedures may benefit from incorporating variable K from the start.
2. Routers encode difficulty: The strong correlation between routing entropy and token characteristics suggests that routers implicitly learn to estimate input difficulty. This representation could be leveraged for other purposes (e.g., curriculum learning, data filtering).
3. Post-hoc optimization is viable: Adaptive-K achieves its benefits without retraining, demonstrating that significant efficiency gains can be obtained through inference-time optimization alone.

8.2 Limitations

We acknowledge several limitations of our current approach:

• Threshold sensitivity: While our calibration procedure works well in practice, optimal thresholds vary across models and domains. A fully automatic, data-free method remains elusive.
• Memory overhead: Storing entropy values and K selections adds approximately 5-10% memory overhead during inference, which may be significant for memory-constrained deployments.
• Batching complexity: Variable K complicates GPU batching efficiency. While our padded batching approach is practical, it does not achieve theoretical maximum throughput.
• Layer uniformity: We currently use the same thresholds for all MoE layers. Layer-specific tuning could potentially improve results but adds complexity.
• Training-based optimization: Our method is post-hoc. Training with Adaptive-K might achieve even better results by allowing the router to adapt to variable K during training.

8.3 Future Directions

Several promising directions emerge from this work:

• Learned thresholds: End-to-end training of threshold parameters with task loss, potentially via straight-through gradient estimators
• Per-layer adaptation: Learning different thresholds for different MoE layers based on their characteristic entropy distributions
• Hardware co-design: Custom CUDA kernels optimized for variable-K sparse computation could significantly improve throughput
• Combination with quantization: Applying Adaptive-K to quantized (INT8/INT4) MoE models for multiplicative efficiency gains
• Multimodal extension: Investigating whether entropy-guided routing generalizes to multimodal MoE models (e.g., vision-language models)

8.4 Multiplicative Savings Potential

Adaptive-K composes multiplicatively with orthogonal optimizations:

Example: Adaptive-K (31%) + INT8 Quantization (33%) + Speculative Decoding (35%):

Conclusion

We have presented Adaptive-K routing, a principled method for dynamic expert selection in Mixture-of-Experts models. Our theoretical analysis, grounded in information theory and rate-distortion theory, establishes that routing entropy serves as a natural proxy for routing difficulty, justifying its use as a criterion for K selection.

Empirical evaluation across four production MoE architectures demonstrates that Adaptive-K achieves substantial compute savings (24-33%) without statistically significant degradation in perplexity or downstream task performance. The method requires no architectural modifications or model retraining, serving as a drop-in replacement for existing fixed-K routing.

We believe this work opens new directions for efficiency optimization in sparse neural networks, and we hope our open-source implementation facilitates adoption and further research. As MoE architectures continue to grow in scale and importance, methods like Adaptive-K that enable more efficient utilization of their sparse computation patterns will become increasingly valuable.

Acknowledgments

The author thanks the open-source community for providing model weights and inference frameworks that made this research possible. Special thanks to the HuggingFace team for the Transformers library and the vLLM project for high-performance inference infrastructure. Compute resources provided by Vast.ai.

Code: github.com/Gabrobals/sbm-efficient

PyPI: pip install adaptive-k-routing