Entropy-Guided Dynamic Expert Selection in Mixture-of-Experts Models:
A Comprehensive Theoretical and Empirical Analysis

Keywords: Mixture-of-Experts, Sparse Models, Dynamic Routing, Information Theory, Computational Efficiency, Large Language Models, Entropy-Based Methods

Introduction

The quest for increasingly capable artificial intelligence systems has driven an exponential growth in neural network scale, with state-of-the-art language models now exceeding hundreds of billions of parameters [1]. This scaling trajectory, while yielding remarkable improvements in model capabilities, presents fundamental challenges in computational efficiency, energy consumption, and deployment feasibility. The Mixture-of-Experts (MoE) paradigm has emerged as a compelling architectural solution to these challenges, enabling dramatic increases in model capacity without proportional increases in computational requirements through the principle of conditional computation [2, 3].

The foundational insight underlying MoE architectures is elegantly simple: rather than activating all parameters for every input, the network learns to route different inputs to different specialized subnetworks—termed "experts"—based on the characteristics of the input itself. This approach draws inspiration from cognitive science theories of modular brain organization [14] and has deep connections to ensemble methods in classical machine learning [15]. Modern instantiations of this principle, exemplified by architectures such as GShard [16], Switch Transformer [3], and Mixtral [4], have demonstrated that MoE models can achieve performance competitive with or superior to dense models of equivalent computational cost while utilizing significantly more total parameters.

1.1 The Fixed-K Problem

Despite the success of MoE architectures, a critical limitation persists in contemporary implementations: the number of experts activated per token—denoted K—remains fixed throughout inference, regardless of the nature of the input. This design choice, while simplifying implementation and enabling efficient batched computation, represents a fundamental inefficiency when viewed through the lens of information theory. Consider the following illustrative scenario:

• High-confidence routing: For common, unambiguous tokens (e.g., function words like "the", "is", "and"), the router network typically exhibits strong preference for a single expert, with the routing probability distribution heavily concentrated. In such cases, activating multiple experts provides minimal additional information while incurring significant computational overhead.
• Low-confidence routing: For rare, ambiguous, or domain-specific tokens, the router may distribute probability mass more uniformly across multiple experts, indicating genuine uncertainty about the optimal routing decision. These tokens may benefit from aggregating information from multiple expert perspectives.

The fixed-K constraint forces the model to treat these fundamentally different scenarios identically, resulting in systematic inefficiency: computational resources are wasted on confident predictions while potentially being insufficient for uncertain ones. This observation motivates our central research question:

1.2 Information-Theoretic Perspective

Our approach to addressing the fixed-K problem is grounded in information theory, specifically the concept of Shannon entropy as a measure of uncertainty [17]. The entropy of the routing distribution provides a natural, theoretically motivated signal for the "difficulty" of a routing decision:

• Low entropy indicates that the router has high confidence in its decision, with probability mass concentrated on a small number of experts. The information content required to specify the routing decision is minimal.
• High entropy indicates uncertainty, with probability distributed more uniformly across experts. More information (i.e., more expert activations) may be required to adequately represent the token.

This perspective connects MoE routing to the broader framework of rate-distortion theory [18], which characterizes the fundamental tradeoff between the "rate" (computational resources expended) and "distortion" (deviation from optimal output). Adaptive-K routing can be understood as an algorithm for operating on the Pareto frontier of this tradeoff, allocating computational resources where they provide the greatest marginal value.

1.3 Contributions

This paper makes the following contributions to the field of efficient neural network inference:

1. Theoretical Framework: We establish a rigorous information-theoretic foundation for entropy-guided expert selection, connecting routing entropy to optimal resource allocation via rate-distortion theory (Section 3).
2. Adaptive-K Algorithm: We propose a simple yet effective algorithm for dynamic K selection based on entropy thresholds, including both theory-based and data-driven calibration strategies (Section 4).
3. Comprehensive Empirical Validation: We evaluate our method on three production-scale MoE models spanning different architectural choices, demonstrating consistent compute savings of 24-52% without quality degradation (Section 5).
4. Ablation Studies: We conduct extensive ablation experiments examining threshold sensitivity, K-value granularity, layer-wise behavior, and domain transfer (Section 6).
5. Open-Source Implementation: We release a production-ready implementation compatible with major inference frameworks, facilitating adoption and further research (Section 8).

1.4 Paper Organization

The remainder of this paper is organized as follows. Section 2 provides necessary background on MoE architectures and routing mechanisms. Section 3 develops the theoretical foundations of entropy-guided routing. Section 4 presents the Adaptive-K algorithm and implementation considerations. Section 5 describes our experimental methodology and main results. Section 6 presents ablation studies and analysis. Section 7 discusses related work in the context of our contributions. Section 8 addresses limitations and future directions, and Section 9 concludes.

Background and Preliminaries

In this section, we establish the mathematical notation and review the fundamental concepts underlying Mixture-of-Experts architectures. We aim to provide sufficient depth for readers unfamiliar with MoE systems while establishing the precise formalism required for our subsequent theoretical development.

2.1 Mixture-of-Experts Architecture

A Mixture-of-Experts layer consists of two primary components: a set of \(N\) expert networks \(\{E_1, E_2, \ldots, E_N\}\) and a gating (router) network \(G\). Each expert \(E_i: \mathbb{R}^d \rightarrow \mathbb{R}^d\) is typically a feed-forward network with identical architecture but independent parameters. The gating network \(G: \mathbb{R}^d \rightarrow \mathbb{R}^N\) produces scores indicating the relevance of each expert for a given input.

2.2 Routing Mechanisms

The gating network produces unnormalized logits \(g(x) = (g_1(x), \ldots, g_N(x))\) for each expert. These logits are typically computed via a linear projection:

where \(W_g \in \mathbb{R}^{N \times d}\) is the gating weight matrix and \(b_g \in \mathbb{R}^N\) is an optional bias term. The routing probability distribution is obtained via softmax normalization:

where \(\tau > 0\) is a temperature parameter controlling the sharpness of the distribution. Lower temperatures produce more peaked distributions, while higher temperatures yield more uniform distributions.

The top-K selection operation identifies the K experts with highest routing probabilities:

The final routing weights are obtained by renormalizing the probabilities over only the selected experts:

2.3 Shannon Entropy of Routing Distributions

The Shannon entropy of a discrete probability distribution quantifies the expected information content, or equivalently, the uncertainty inherent in the distribution [17]. For the routing distribution \(p(x)\), the entropy is defined as:

The routing entropy has the following important properties:

• Non-negativity: \(H(x) \geq 0\), with equality if and only if the distribution is deterministic (all mass on a single expert).
• Maximum entropy: \(H(x) \leq \log N\), with equality when the distribution is uniform over all \(N\) experts.
• Concavity: Entropy is a concave function of the probability distribution, which has implications for optimization.

The range of routing entropy depends on the number of experts \(N\). For reference, the maximum entropy values for common MoE configurations are:

2.4 Computational Cost Model

To quantify the computational savings achieved by Adaptive-K routing, we establish a formal cost model. Let \(C_E\) denote the computational cost (in FLOPs) of a single expert forward pass, and let \(C_G\) denote the cost of the gating computation. For standard top-K routing, the per-token cost is:

For Adaptive-K routing with variable \(K(x)\), the expected cost is:

where \(C_H\) is the overhead of entropy computation. Since entropy computation requires only \(O(N)\) operations compared to \(O(d^2)\) for expert forward passes (where \(d \gg N\) typically), we have \(C_H \ll C_E\), making this overhead negligible in practice.

The relative compute savings are therefore:

Theoretical Foundations

In this section, we develop the theoretical foundations for entropy-guided expert selection. We begin by establishing connections to information theory, then present a rate-distortion theoretic analysis that motivates our algorithmic design.

3.1 Information-Theoretic Interpretation of Routing

We propose to interpret the MoE routing process through the lens of information theory. Specifically, we view the router as an encoder that maps input tokens to expert activations, and the entropy of the routing distribution as measuring the "bits of specification" required to describe the routing decision.

This proposition establishes that routing entropy directly relates to the intrinsic complexity of the routing decision. When entropy is low, the router has effectively "decided" on a small number of experts, and activating additional experts provides diminishing returns.

3.2 Rate-Distortion Theoretic Analysis

We formalize the relationship between computational cost and output quality using rate-distortion theory [18]. Define the "rate" \(R\) as the average number of experts activated, and the "distortion" \(D\) as the deviation from the output that would be obtained using all experts.

The rate-distortion function \(R(D)\) characterizes the minimum rate (experts) required to achieve distortion at most \(D\). While computing \(R(D)\) exactly is intractable, we can establish the following relationship:

Intuitively, this proposition states that when the router is confident (low entropy), the output is well-approximated by a small number of experts. This provides theoretical justification for using entropy as a criterion for dynamic K selection.

3.3 Optimal K Selection

Given the relationship between entropy and distortion, we can formulate the K selection problem as an optimization over entropy thresholds. Let \(\mathcal{K} = \{k_1 < k_2 < \cdots < k_m\}\) be the set of allowed K values and \(\Theta = \{\theta_1 < \theta_2 < \cdots < \theta_{m-1}\}\) be the entropy thresholds. The K selection function is:

The optimal thresholds minimize expected cost subject to a quality constraint:

In practice, we find that simple heuristics based on entropy percentiles perform well, as detailed in Section 4.

Adaptive-K Routing Algorithm

4.1 Algorithm Description

Based on the theoretical foundations developed in Section 3, we present the Adaptive-K routing algorithm. The algorithm consists of three phases: (1) entropy computation, (2) K selection via threshold comparison, and (3) sparse expert execution with renormalized weights.

The algorithm has \(O(N)\) overhead for entropy computation, which is negligible compared to the \(O(Kd^2)\) cost of expert forward passes for typical transformer dimensions.

4.2 Threshold Calibration Strategies

The choice of entropy thresholds \(\Theta\) determines the trade-off between compute savings and output quality. We propose two complementary strategies for threshold selection:

4.2.1 Theory-Based Thresholds

Based on the maximum entropy \(H_{\max} = \log N\), we can set thresholds as fractions of this theoretical maximum:

We recommend starting with \(\alpha_1 = 0.5\) for binary K selection (K ∈ {1, 2}), which corresponds to the point where the routing distribution has roughly half its maximum uncertainty.

4.2.2 Data-Driven Calibration

For optimal performance, thresholds can be calibrated on a representative calibration dataset:

1. Run inference on calibration set (1000-10000 samples recommended)
2. Collect routing entropy values for all tokens across all layers
3. Compute entropy percentiles (e.g., 25th, 50th, 75th)
4. Set thresholds at percentile boundaries corresponding to desired K distribution
5. Optionally fine-tune via grid search over threshold neighborhoods

4.3 Batched Inference Considerations

Efficient GPU inference requires batched computation, which presents challenges when different tokens in a batch require different K values. We propose the following strategies:

4.3.1 Padded Batching

Compute top-\(K_{\max}\) experts for all tokens, then mask out excess experts based on per-token K:

def adaptive_k_batched(router_logits, thresholds, k_values):
    # Compute entropy and K for each token in batch
    probs = F.softmax(router_logits, dim=-1)
    entropy = -(probs * torch.log(probs + 1e-9)).sum(dim=-1)
    
    # Determine K per token
    k_per_token = torch.full_like(entropy, k_values[-1], dtype=torch.long)
    for i, threshold in enumerate(thresholds):
        k_per_token = torch.where(entropy < threshold, k_values[i], k_per_token)
    
    # Get top-K_max experts
    k_max = max(k_values)
    topk_probs, topk_indices = torch.topk(probs, k_max, dim=-1)
    
    # Create mask based on actual K per token
    positions = torch.arange(k_max, device=probs.device).unsqueeze(0)
    mask = positions < k_per_token.unsqueeze(1)
    
    # Apply mask and renormalize
    masked_probs = topk_probs * mask.float()
    weights = masked_probs / (masked_probs.sum(dim=-1, keepdim=True) + 1e-9)
    
    return topk_indices, weights, k_per_token, entropy

4.3.2 Dynamic Grouping

For maximum efficiency with highly variable K, tokens can be grouped by their K value and processed in separate batches. This adds sorting overhead but eliminates padding waste for workloads with bimodal K distributions.

Experimental Evaluation

5.1 Experimental Setup

5.1.1 Models

We evaluate Adaptive-K routing on three production MoE models representing different architectural choices and scales:

5.1.2 Datasets

• WikiText-2 [19]: Standard language modeling benchmark (245K test tokens)
• Penn Treebank: Classic LM benchmark for perplexity evaluation
• MMLU [20]: 57-subject multiple-choice benchmark for knowledge assessment
• HellaSwag: Commonsense reasoning benchmark
• C4-validation: Held-out web text for calibration

5.1.3 Metrics

• Perplexity (PPL): Exponential of cross-entropy loss; lower is better
• Average K: Mean number of experts activated per token
• Compute (%): Relative to baseline, computed as \(\text{Avg K} / K_{\text{baseline}}\)
• Downstream accuracy: Task-specific accuracy on MMLU, HellaSwag

5.2 Entropy Distribution Analysis

Before presenting main results, we characterize the routing entropy distributions observed in each model. Understanding these distributions is essential for threshold calibration and interpreting savings potential.

5.3 Main Results

5.3.1 Mixtral 8×7B

For Mixtral with N=8 experts and baseline K=2, we use binary Adaptive-K with K ∈ {1, 2} and a calibrated threshold of θ₁ = 1.275 (corresponding to the 62nd percentile of observed entropy).

The K distribution shows that 62% of tokens use K=1, while the remaining 38% use K=2. This distribution emerges naturally from the entropy-based selection and correlates with token characteristics (common words → K=1, rare/technical terms → K=2).

5.3.2 Qwen1.5-MoE-A2.7B

Qwen1.5-MoE uses fine-grained experts (N=60) with higher baseline K=4. We use K ∈ {2, 3, 4} with thresholds θ = {1.8, 2.4}.

5.3.3 OLMoE-1B-7B

OLMoE uses many small experts (N=64) with high baseline K=8, representing an extreme point in the MoE design space. We use K ∈ {4, 6, 8} with thresholds θ = {2.5, 3.2}.

5.4 Summary of Results

Ablation Studies and Analysis

6.1 Threshold Sensitivity

We investigate how sensitive Adaptive-K performance is to the choice of entropy thresholds. Experiments are conducted on Mixtral 8×7B with various threshold values.

The results reveal a clear Pareto frontier: lower thresholds increase K=1 usage and compute savings but eventually degrade quality. The calibrated threshold (62nd percentile) sits at the "knee" of this curve, achieving near-maximum savings with minimal quality impact.

6.2 K-Value Granularity

We examine whether finer-grained K value sets improve performance over binary {1, 2} selection.

6.3 Layer-wise Analysis

We analyze how routing entropy and K distribution vary across layers in Mixtral 8×7B, which has 32 MoE layers.

This layer-wise variance suggests potential for further optimization via per-layer threshold tuning, which we leave for future work.

6.4 Token Characteristics and K Selection

To understand what distinguishes K=1 tokens from K=2 tokens, we analyzed token characteristics:

6.5 K Distribution Visualization

Related Work

7.1 Mixture-of-Experts Architectures

The Mixture-of-Experts concept was introduced by Jacobs et al. [21] and Jordan & Jacobs [22] in the early 1990s, originally as a supervised learning framework for decomposing complex problems. The application to large-scale neural networks was pioneered by Shazeer et al. [2], who demonstrated that sparsely-gated MoE layers could scale language models to unprecedented sizes.

Subsequent work has explored various aspects of MoE design:

• Expert architecture: GShard [16] and Switch Transformer [3] explored different expert sizes and configurations
• Routing mechanisms: Expert Choice [5] inverts the routing direction, having experts select tokens rather than vice versa
• Load balancing: Auxiliary losses [6] encourage balanced expert utilization during training
• Scaling laws: Recent work has characterized how MoE models scale differently than dense models [23]

Our work is orthogonal to these architectural innovations—Adaptive-K can be applied to any MoE architecture with a learned router producing a probability distribution over experts.

7.2 Dynamic Computation Methods

The broader field of dynamic computation encompasses several related approaches:

• Early Exit [7]: Allows tokens to exit at intermediate layers when confidence is high, reducing depth-wise computation
• Adaptive Depth [8]: Learns to vary transformer depth on a per-token basis
• Speculative Decoding [24]: Uses a smaller draft model to propose tokens that are verified by the full model
• Token Pruning [25]: Removes unimportant tokens from intermediate representations

Adaptive-K is complementary to these approaches and could be combined for multiplicative efficiency gains. For example, early exit could skip entire MoE layers while Adaptive-K reduces compute within executed layers.

7.3 Entropy-Based Methods in Deep Learning

Entropy has been used as a decision criterion in various deep learning contexts:

• Active learning uses entropy to identify informative samples [26]
• Confidence calibration relates output entropy to prediction reliability [27]
• Neural architecture search uses entropy for architecture selection [28]

To our knowledge, we are the first to use routing entropy for dynamic expert selection in MoE models.

Discussion

8.1 Broader Implications

The success of Adaptive-K routing has several implications for the MoE research community:

1. Fixed K is suboptimal: The substantial savings achieved without quality loss suggest that fixed-K routing leaves significant efficiency on the table. Future MoE training procedures might benefit from incorporating variable-K from the start.
2. Routers encode difficulty: The strong correlation between routing entropy and token characteristics suggests that routers implicitly learn to estimate input difficulty. This representation could be exploited for other purposes (e.g., curriculum learning, data filtering).
3. Post-hoc optimization is viable: Adaptive-K achieves its benefits without retraining, demonstrating that significant efficiency gains can be obtained through inference-time optimization alone.

8.2 Limitations

We acknowledge several limitations of our current approach:

1. Threshold sensitivity: While our calibration procedure works well in practice, optimal thresholds vary by model and domain. A fully automated, data-free method remains elusive.
2. Memory overhead: Storing entropy values and K selections adds approximately 5-10% memory overhead during inference, which may be significant for memory-constrained deployments.
3. Batching complexity: Variable K complicates efficient GPU batching. While our padded batching approach is practical, it doesn't achieve theoretical maximum throughput.
4. Layer uniformity: We currently use the same thresholds across all MoE layers. Layer-specific tuning could potentially improve results but adds complexity.
5. Training-aware optimization: Our method is post-hoc. Training with Adaptive-K might achieve even better results by allowing the router to adapt to variable K during training.

8.3 Future Directions

Several promising directions emerge from this work:

• Learned thresholds: Training threshold parameters end-to-end with task loss, potentially via straight-through gradient estimators.
• Per-layer adaptation: Learning different thresholds for different MoE layers based on their characteristic entropy distributions.
• Hardware co-design: Custom CUDA kernels optimized for variable-K sparse computation could significantly improve throughput.
• Combination with quantization: Applying Adaptive-K to quantized (INT8/INT4) MoE models for multiplicative efficiency gains.
• Multi-modal extension: Investigating whether entropy-guided routing generalizes to multi-modal MoE models (e.g., vision-language models).

Conclusion

We have presented Adaptive-K routing, a principled method for dynamic expert selection in Mixture-of-Experts models based on routing entropy. Our theoretical analysis, grounded in information theory and rate-distortion theory, establishes that routing entropy serves as a natural proxy for routing difficulty, justifying its use as a criterion for K selection.

Empirical evaluation on three production MoE architectures demonstrates that Adaptive-K achieves substantial compute savings (24-52%) without meaningful degradation in perplexity or downstream task performance. The method requires no architectural modifications or model retraining, serving as a drop-in replacement for existing fixed-K routing.

We believe this work opens new directions for efficiency optimization in sparse neural networks and hope that our open-source implementation facilitates adoption and further research. As MoE architectures continue to grow in scale and importance, methods like Adaptive-K that enable more efficient utilization of their sparse computation patterns will become increasingly valuable.

Acknowledgments

The author thanks the open-source community for providing model weights and inference frameworks that made this research possible. Special thanks to the HuggingFace team for the Transformers library and to the vLLM project for high-performance inference infrastructure.

References

Appendix A: Detailed Experimental Configuration

Appendix B: SDK Usage Example

# Installation
pip install adaptive-k-routing

# Basic usage with PyTorch
import torch
from adaptive_k import AdaptiveKRouter, EntropyCalibrator

# Initialize router for Mixtral
router = AdaptiveKRouter(
    k_values=[1, 2],
    model_name="mixtral-8x7b",
    calibration_mode="percentile"
)

# Calibrate on sample data
calibrator = EntropyCalibrator(router)
with torch.no_grad():
    calibrator.calibrate(calibration_loader, percentile=62)

# Apply during inference
def forward_with_adaptive_k(hidden_states, router_logits):
    indices, weights, k_selected = router.apply(router_logits)
    # Execute only selected experts...
    return output, k_selected.float().mean()

# Monitor statistics
stats = router.get_statistics()
print(f"Avg K: {stats['avg_k']:.2f}")
print(f"Compute savings: {stats['savings']:.1%}")
print(f"K distribution: {stats['k_distribution']}")