This is a remedial run for missed papers from 07/30/2026 to 07/30/2026.

Results generated on 09/13/2026.

Personalized Daily ArXiv Papers 2026-07-31

_{Token counts are not reported for this run. 5 of 6 model calls succeeded, 2,793s of model wall clock.}

Topic Coverage:

Table of contents by topic:

MoE Training (1)

1. From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference Authors: Tianyang Zhu

Large-Scale Training Systems and Efficiency (3)

1. Explaining Data Mixing Scaling Laws Authors: Rui Dai, Shuran Zheng

2. Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting Authors: Xiang Yuan, Kaiqing Lei, Zhenyu Jin, Jun Shu, Deyu Meng, Zongben Xu

3. Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis Authors: Yuichi Kondo, Hideaki Iiduka

Architecture and Training Dynamics (8)

1. Critical attention scaling in long-context transformers Authors: Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet

2. Latent Matters: Learning Deep State-Space Models Authors: Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, Patrick van der Smagt

3. Simplifying Neural Networks During Training Authors: Lorenzo Sciandra, Samuele Fonio, Roberto Esposito

4. Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation Authors: Alexander Boesgaard Lorup

5. A Nonlinear Singular Value Theory for Neural Networks Authors: Brian Charles Brown, Mauricio Munoz, Robert Bridges, David Grimsman, Sean Warnick

6. What Makes Graph Unified? Principles and Generative Sliding-Window Transformer for Graph Foundation Models Authors: Dongxiao He, Siqi Liu, Jitao Zhao, Yawen Li, Yi Wang, Di Jin

7. Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation Authors: Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

8. Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks Authors: Mostafa Haghir Chehreghani

Efficiency, Compression, and Large-Scale Training (13)

1. Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection Authors: Arunan J

2. Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning Authors: Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu

3. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs Authors: Jinyi Liu, Wei Chen, Pengyu Chen, Xinyi Yuan, Minghe Bai, Guoquan Wu, Jun Wei

4. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference Authors: Sangjin Kim, Yuseon Choi, Jungjun Oh, Byeongcheol Kim, Hoi-Jun Yoo

5. Back from the Future: Key-Value Cache Management by Counter-Causal Surprise Authors: Stephen Gould, Anton van den Hengel

6. RAP: KV-Cache Compression via RoPE-Aligned Pruning Authors: Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, Marco Canini

7. FeatFix: Reuse What You Verify through Local Exact-Feature Correction for Faster Cached Diffusion Inference Authors: Hanshuai Cui, Zhiqing Tang, Zhi Yao, Qianli Ma, Fanshuai Meng, Weijia Jia

8. SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models Authors: Jinwei Kong, Runqi Meng, Fanyi Wang, Wentao Qiu, Haotian Hu, Yongjian Zhou, Zhenhua Ge

9. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs Authors: Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, Oana Balmau

10. LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference Authors: Feng Yang, Xinrui Ju, Keyang Zhang, Xiandong Meng, Rongqun Lin, Howard Leung, Shiqi Wang, Haoliang Li, Chris Xing Tian

11. Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models Authors: Jialuo He, Huangxun Chen

12. Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization Authors: Robert Jankowski, Pedro Almagro-Blanco, Marián Boguñá, Melanie Weber, M. Ángeles Serrano

13. Transporting Task Vectors across Different Architectures without Training Authors: Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, Simone Calderara

MoE Training (1)

1. From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference

ArXiv ID: 2607.28097

Primary Topic: MoE Training

Also Matches: Architecture and Training Dynamics

Authors: Tianyang Zhu

Abstract: Mathematically equivalent expert-reduction orders can produce observably different sparse-MoE executions. We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics. Four schemes separate operand representation from accumulator precision. At one layer-5 fork, 720 A-mode orders yield 10 continuation basins; 720 B-mode orders form 360 exact structural classes and 11 basins. Under one Chinese prompt, the B classes split into 202 layoffs, 113 hiring, and 45 other continuations. Maximum-L-infinity B-branch selection separates 12, 24, and 36 of 50 prompts by 8, 16, and 32 tokens. Across 192 persistent trajectories per scheme, P32, A, and B change every native-reference route trajectory, while C preserves routes, token sequences, and texts. A separate 192-trajectory C check matches native MoE, post-mHC, next-router, and LM states bitwise. For one controlled B branch, exact post-mHC endpoint reconstruction reproduces the measured downstream trajectory. At the next decode boundary, exact FP64 reconstruction of the branch's full persistent state yields agreement for 301 downstream post-mHC states, 301 persistent-state checkpoints, 301 routes, predictions, and text over seven steps, given the same naturally generated next input. These controls identify post-mHC as an intra-token boundary and full persistent state as a cross-token continuation boundary. Identical tokens need not imply identical autoregressive state: divergence can survive a token boundary and become visible later. These results make expert operand conversion, accumulator precision, and reduction order part of a numerical compatibility contract for sparse-MoE runtimes and hardware backends. They establish controlled causal possibility, not deployment incidence; C's order invariance is limited to evaluated six-term states and schedules.

Comment: Shows that expert-reduction order, operand conversion, and accumulator precision can change sparse-MoE routing trajectories.

Topic Match: It directly establishes numerical requirements for MoE expert kernels and persistent routing state.

Relevance: 8 Novelty: 8

Large-Scale Training Systems and Efficiency (3)

1. Explaining Data Mixing Scaling Laws

ArXiv ID: 2606.08167

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Rui Dai, Shuran Zheng

Abstract: Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures. However, a theoretical understanding of these model loss behaviors remains absent. In this work, we propose a unified framework to explain the underlying mechanics of data mixing. Our approach extends theoretical perspectives originally developed for standard neural scaling laws (e.g., Kaplan and Chinchilla) to the multi-domain setting. Based on the distributional assumption that domains overlap on fundamental skills while diverging on specialized skills, we identify two key factors that govern the domain losses of models trained on different data mixtures: \textit{Capacity Competition}, where the allocation of finite model capacity couples domain losses globally, and \textit{Noise Reduction}, where optimal weights shift toward harder-to-learn domains to minimize overall noise. Empirical evaluations show that our framework outperforms existing baselines by fitting the loss landscape with a lower Mean Relative Error and identifying higher-performing training mixtures. Most importantly, our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales using parameters fitted on smaller ones. In addition, our model achieves these results using significantly fewer parameters compared to previous empirical laws. Our code is available at https://github.com/meiqwq/Explaining-Data-Mixing-Scaling-Laws.

Comment: Explains and extrapolates data-mixture scaling through capacity competition and domain-specific noise reduction.

Topic Match: The scaling law directly informs pretraining-mixture configuration, supported by a mechanistic account of model-capacity dynamics.

Relevance: 9 Novelty: 8

2. Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting

ArXiv ID: 2607.27928

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Xiang Yuan, Kaiqing Lei, Zhenyu Jin, Jun Shu, Deyu Meng, Zongben Xu

Abstract: The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data. While manual heuristics were prevalent in early models, they increasingly fail to capture the intricate synergies between domains as data complexity grows. To overcome the issue, a dominant approach seeks to fit a proxy function mapping between domain weights and their corresponding validation losses, and then find the optimal domain weights to minimize validation losses. These methods rely on strong structural assumptions, such as rank invariance or scaling laws, which are often violated, resulting in non-negligible estimation bias. A promising approach is to directly optimize the weighting scheme from data. However, it suffers from unstable optimization trajectory and prohibitive computational overhead, limiting its potential to search better domain weights configurations. This paper presents a Bayesian domain weighting method to infer the weights from a Dirichlet distribution via introducing Gamma prior information learned from observations. Experimental results demonstrate that proposed method could achieve stable and efficient domain weights learning, and identifies optimal mixtures while consuming substantially less data than search-based function-fitting methods, revitalizing optimization-based domain weighting for large-scale applications.

Comment: Uses Bayesian domain reweighting to optimize large-scale pretraining mixtures with fewer search runs.

Topic Match: It directly changes how multi-domain pretraining runs are configured and reduces the cost of finding effective mixtures.

Relevance: 8 Novelty: 7

3. Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

ArXiv ID: 2508.03105

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Yuichi Kondo, Hideaki Iiduka

Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm under our Lyapunov-based analysis, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warmup schedule in experiments, which empirically outperformed all other strategies in convergence behavior.

Comment: Derives a convergence hierarchy for coupled learning-rate and batch-size schedules using a new Lyapunov analysis.

Topic Match: The results directly inform optimizer and batch scheduling for costly training runs.

Relevance: 7 Novelty: 6

Architecture and Training Dynamics (8)

1. Critical attention scaling in long-context transformers

ArXiv ID: 2510.05554

Primary Topic: Architecture and Training Dynamics

Authors: Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet

Abstract: As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse. While $\textit{attention scaling}$ effectively addresses this deficiency by rescaling attention scores with a polylogarithmic factor $β_n$, theoretical justification for this approach remains lacking. We analyze a simplified yet tractable model that magnifies the effect of attention scaling. In this model, attention exhibits a phase transition governed by the scaling factor $β_n$: insufficient scaling collapses all tokens to a single direction, while excessive scaling reduces attention to identity, thereby eliminating meaningful interactions between tokens. Our main result identifies the critical scaling $β_n \asymp \log n$ and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.

Comment: Derives the critical logarithmic attention-score scaling that avoids both rank collapse and identity attention at long context.

Topic Match: This is a direct theoretical analysis of a consequential long-context attention mechanism.

Relevance: 9 Novelty: 8

2. Latent Matters: Learning Deep State-Space Models

ArXiv ID: 2602.23050

Primary Topic: Architecture and Training Dynamics

Authors: Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, Patrick van der Smagt

Abstract: Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.

Comment: Introduces constrained training and an extended-Kalman VAE for learning faithful latent state-space dynamics.

Topic Match: The core contribution combines a state-space architecture with a training objective designed to recover underlying dynamics.

Relevance: 8 Novelty: 7

3. Simplifying Neural Networks During Training

ArXiv ID: 2607.27854

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Lorenzo Sciandra, Samuele Fonio, Roberto Esposito

Abstract: Understanding and exploiting the training dynamics of overparameterized deep neural networks remains a central challenge in modern machine learning. Recent evidence on Neural Collapse (NC) shows that class representations and classifiers exhibit highly structured geometry, while the Tunnel Effect suggests that only a subset of layers is essential for feature extraction. We combine these two perspectives and propose an NC-inspired training framework for simplifying deep networks during training. Our method monitors representation dynamics through the Inverse Fisher Criterion, a stable and efficient proxy for the variability collapse behavior, to identify both the split point between feature extraction and classification and the training stage at which simplification becomes viable. We then replace the trailing layers with a lightweight classification head and continue training the reduced model. Experiments on image-classification benchmarks across MLP, VGG, and ResNet architectures show that the proposed method achieves substantial parameter reductions while maintaining accuracy comparable to that of the full model. Code to reproduce the experiments can be found at: https://github.com/LorenzoSciandra/NNS.

Comment: Uses representation-collapse dynamics to determine when trailing network layers can be replaced during training.

Topic Match: The central mechanism couples training-dynamics monitoring with adaptive architectural simplification.

Relevance: 7 Novelty: 7

4. Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation

ArXiv ID: 2607.28495

Primary Topic: Architecture and Training Dynamics

Authors: Alexander Boesgaard Lorup

Abstract: Stage-replay diagnostics reconstruct intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix. We audit that assumption at a whole reasoning-stage boundary in a Qwen2.5-derived system. A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides. In BF16, replicas remain exact while the constructions differ on 166 suffixes and 20 correctness labels; the accuracy difference is only one point (paired 95% CI [-3.5, +5.5]). A fixed-prefix 2x2 holds all 200 token states constant while crossing construction and precision. The BF16 disagreements recur, whereas FP32 produces no decoded disagreement (95% Wilson upper bound 1.88%). A prospective bridge makes token-by-token incremental and retained live caches bit-exact on 12/12 rows; an all-200 saved-ledger audit reproduces every retained trajectory and comparison fingerprint. Bidirectional transplantation of all 48 key/value layers makes every tested divergent continuation follow its cache donor, both on a selected set at the primary checkpoint (24/24) and an outcome-blind replication at a later checkpoint (43/43). Exact-token replay can therefore be repeatable without preserving live-state fidelity. On the tested states, boundary K/V cache is a causally sufficient carrier of the divergent trajectory, while numerical precision moderates its behavioral expression.

Comment: Establishes KV-cache state and numerical precision as causal controls of fixed-prefix replay divergence.

Topic Match: The paper provides mechanistic insight into persistent transformer state and precision-sensitive computation.

Relevance: 6 Novelty: 8

5. A Nonlinear Singular Value Theory for Neural Networks

ArXiv ID: 2605.06938

Primary Topic: Architecture and Training Dynamics

Authors: Brian Charles Brown, Mauricio Munoz, Robert Bridges, David Grimsman, Sean Warnick

Abstract: Recently Brown et al. [2025] established a singular value decomposition (SVD) for maps (especially nonlinear) satisfying certain norm conditions. We prove that most modern neural architectures admit this nonlinear SVD (NLSVD) representation---with no change in input--output behavior---and enumerate the classes covered. In this factorization the network is a left-invertible nonlinear map followed by a final linear layer. Moreover, the left-invertible factor is norm-preserving, so distances in the embedding (activations before the final linear layer) calibrate directly to distances in input space. We introduce a flexible architecture that yields an explicit decomposition at training time, a data-driven algorithm for estimating the representation from trained models, and the mathematical foundations for nonlinear analogues of row and null spaces in neural networks. Empirical case studies illustrate uses of the theory for latent-space pullback (visualization and data generation), bias detection, and membership-inference robustness under training. Altogether, these foundations support new approaches to core problems in neural-network analysis.

Comment: Establishes a nonlinear SVD factorization for broad neural-architecture classes and supplies an explicit trainable construction.

Topic Match: The work provides a fundamental factorization and computational mechanism for neural architectures.

Relevance: 6 Novelty: 8

6. What Makes Graph Unified? Principles and Generative Sliding-Window Transformer for Graph Foundation Models

ArXiv ID: 2607.27966

Primary Topic: Architecture and Training Dynamics

Authors: Dongxiao He, Siqi Liu, Jitao Zhao, Yawen Li, Yi Wang, Di Jin

Abstract: Graph Foundation Models (GFMs) have recently emerged as a promising paradigm for general-purpose graph learning, aiming to learn reusable knowledge that generalizes across diverse graph domains and downstream tasks, reducing the need for specific model development. Achieving this goal requires reconciling the substantial heterogeneity in node features, graph structures, and semantic information across domains. Among them, heterogeneous node features constitute a fundamental input-level barrier, as their dimensionality and semantics vary substantially across datasets. Existing studies typically project or map heterogeneous node features into a fixed-dimensional space, often implicitly equating dimensional uniformity with effective feature unification. Yet dimensional consistency alone does not ensure that the unified features preserve informative semantics and capture transferable patterns that can support cross-domain knowledge transfer. To bridge this conceptual gap, we distill four desiderata for cross-domain graph feature unification: formal uniformity, cross-domain transferability, information preservation, and backbone compatibility. Guided by these principles, we propose SliGFM, a graph foundation model built upon topology-aware sliding-window feature encoding and generative reconstruction. SliGFM orders feature dimensions by topological smoothness and scans the reordered features with a shared sliding-window feature encoder, transforming heterogeneous features into a common space of ordered fixed-dimensional feature tokens. This formulation enables a smoothness-aware transformer to capture transferable relational patterns among feature tokens within each node, while the generative reconstruction objective encourages preservation of the original feature information.

Comment: Introduces topology-aware sliding-window feature tokenization and generative reconstruction for graph foundation models.

Topic Match: Its core contribution is a new tokenization and transformer mechanism for heterogeneous graph features.

Relevance: 6 Novelty: 7

7. Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

ArXiv ID: 2606.13657

Primary Topic: Architecture and Training Dynamics

Authors: Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

Abstract: On-policy distillation (OPD) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student-generated trajectories and dense token-level teacher supervision. Yet how this hybrid training regime shapes a model remains poorly understood. We characterize the sparsity and geometry of OPD parameter updates across several language and vision-language model pairs and application settings. OPD updates are small and coordinate-sparse at checkpoint precision, while remaining distributed across layers and modules. This sparse support is operationally meaningful: masked training on the discovered subnetwork nearly recovers full-training performance. At the matrix level, the updates are numerically full-rank but spectrally concentrated. Their visible supports avoid coordinates emphasized by the source's principal structure and favor low-magnitude source coordinates, while the source singular-value spectra change little. Together, these findings show that OPD exhibits important weight-space signatures of on-policy post-training despite using dense teacher supervision.

Comment: Characterizes the coordinate sparsity and spectral geometry of on-policy distillation updates and validates their effective subnetwork.

Topic Match: Its strongest fit is mechanistic training-dynamics analysis of how parameter updates are distributed.

Relevance: 6 Novelty: 7

8. Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks

ArXiv ID: 2607.28185

Primary Topic: Architecture and Training Dynamics

Authors: Mostafa Haghir Chehreghani

Abstract: Oversmoothing is a fundamental limitation of deep graph neural networks (GNNs), where repeated message passing causes node representations to become increasingly similar, eventually collapsing toward a low-dimensional subspace. This phenomenon limits the effective depth of message-passing architectures and motivates the search for mechanisms that preserve representation diversity. In this paper, we study a recurrent graph neural network in which independent Gaussian noise is injected after every propagation step and analyze the resulting architecture as a stochastic dynamical system. Under a standard global contraction assumption on the deterministic update, we prove that the hidden representations form a geometrically ergodic Markov chain admitting a unique invariant probability measure. Our main theoretical result establishes an explicit positive lower bound on the expected stationary Dirichlet energy, proportional to both the noise variance and the spectral gap of the underlying graph. Consequently, the stationary representations cannot collapse onto the constant manifold, providing a rigorous guarantee that asymptotic oversmoothing is prevented in the sense of non-vanishing Dirichlet energy. Our analysis reveals persistent stochastic perturbations as a fundamentally different mechanism for combating oversmoothing, complementing existing deterministic approaches based on residual connections, normalization, and graph rewiring. Finally, numerical experiments on both linear and nonlinear recurrent graph neural networks closely match the theoretical predictions, illustrating the emergence of a stationary distribution and the predicted dependence of the limiting Dirichlet energy on the noise intensity.

Comment: Proves that persistent Gaussian perturbations maintain nonzero representation diversity in recurrent GNN dynamics.

Topic Match: The core result explains how a stochastic architectural mechanism prevents oversmoothing.

Relevance: 6 Novelty: 7

Efficiency, Compression, and Large-Scale Training (13)

1. Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

ArXiv ID: 2607.27680

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Arunan J

Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood. Existing generalization results provide upper bounds of the form O~(sqrt(rd/n)) or O~(rd/n), but a matching lower bound is missing, and the question of how to choose the LoRA rank r has no formal answer. Both gaps are closed here. A local Rademacher argument establishes an upper bound of O~(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA, whenever the target adaptation has rank at most r. A matching minimax lower bound of Omega(rd/n) is then proved via a Fano-type packing of the rank-r subspace of R^{d x d}; the bound applies to any estimator whose output lies in the rank-r LoRA class. Combining the two yields a rank-selection dichotomy. For the constrained empirical risk minimizer, the optimal rank equals the intrinsic rank r, and over-ranking strictly hurts. For adaptive estimators of the nuclear-norm-then-truncate type, over-ranking is harmless and the rate saturates at Theta~(r d / n) regardless of r. Taken together, the three results characterize the statistical complexity of LoRA fine-tuning within the well-specified locally quadratic regime, and identify the empirically observed over-parameterization penalty as a property of unregularized empirical risk minimization rather than of the LoRA class itself. Predictions of the theory are verified on a synthetic trace-regression benchmark and on real LoRA fine-tuning across three (model, task) configurations covering DistilBERT and RoBERTa on SST-2 and MRPC. All configurations exhibit the predicted U-shape in validation loss, with two showing statistically significant loss inflation at large ranks (paired permutation p = 0.016).

Comment: Derives matching LoRA sample-complexity bounds and distinguishes when excessive adaptation rank harms generalization.

Topic Match: The theory directly informs low-rank adaptation rank selection and its efficiency-generalization tradeoff.

Relevance: 8 Novelty: 8

2. Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning

ArXiv ID: 2606.03328

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu

Abstract: Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects. We show that this conclusion is an averaging artifact: at 60\% SparseGPT sparsity, calibration strategies separated by only 2.85 points in averaged commonsense accuracy differ by 51.9 points in Code retention. Across 15 sources, capability-decomposed analysis reveals an opposing pattern: calibration perplexity is positively associated with General retention but negatively associated with Math or Code retention, leaving no evaluated single source uniformly strong across capabilities. This finding motivates capability-balanced multi-source calibration. Under the same calibration budget, a balanced real-data mixture outperforms every evaluated single source on LLaMA-3.1-8B, beating C4 by 18.8 points; the advantage grows with sparsity and persists on LLaMA-3.1-70B. Because the original pretraining data of advanced LLMs are often inaccessible, we further introduce Information-Guided Self-Calibration for Pruning (IGSP). Using only the base model and evaluation taxonomy, IGSP generates capability-stratified pools and selects low-redundancy samples within capability-specific perplexity ranges, outperforming Self-Cal and SGS by up to 4.8 points. Together, these results recast calibration as a capability-coverage problem and identify multi-source design as a practical principle for preserving capabilities in high-sparsity LLM pruning.

Comment: Shows that capability-balanced calibration data substantially improves LLM pruning at high sparsity.

Topic Match: The core contribution improves capability preservation under aggressive post-training sparsification.

Relevance: 8 Novelty: 7

3. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

ArXiv ID: 2607.27591

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jinyi Liu, Wei Chen, Pengyu Chen, Xinyi Yuan, Minghe Bai, Guoquan Wu, Jun Wei

Abstract: Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present \emph{Prox}, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a $1.99\times$ end-to-end decoding speedup at 70\% FFN sparsity, and is compatible with quantization and sparse attention.

Comment: Uses quantized proxy computation to select sparse SwiGLU channels without first evaluating the dense intermediate state.

Topic Match: The work introduces a concrete activation-sparsity mechanism that materially accelerates LLM inference.

Relevance: 8 Novelty: 7

4. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

ArXiv ID: 2607.27704

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Sangjin Kim, Yuseon Choi, Jungjun Oh, Byeongcheol Kim, Hoi-Jun Yoo

Abstract: As large language models (LLMs) continue to demonstrate exceptional capabilities across various domains, the challenge of achieving energy-efficient and accurate inference becomes increasingly critical. This work presents LightRot, a lightweight rotation scheme and dedicated hardware accelerator designed for low-bit LLM inference. The proposed architecture integrates Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit to address key challenges in low-bit quantization, including the energy overhead of rotation operations. The proposed accelerator, implemented in a 28nm CMOS process, achieves a peak energy efficiency of 27.4 TOPS/W for 4-bit inference, surpassing prior state-of-the-art designs. Unlike conventional approaches that rely on higher-precision inference or evaluate on basic language modeling tasks like GPT-2, LightRot is optimized for advanced models such as LLaMA2-13B and LLaMA3-8B. Its performance is further validated on MT-Bench, demonstrating robust applicability to real-world conversational scenarios and redefining benchmarks for chat-based AI systems. By synergizing algorithmic innovations and hardware efficiency, this work sets a new paradigm for scalable, low-bit LLM inference, paving the way for sustainable AI advancements.

Comment: Combines grouped local rotations, outlier alignment, and dedicated FHT hardware for accurate 4-bit LLM inference.

Topic Match: The core algorithm-hardware design directly advances low-bit model compression and inference efficiency.

Relevance: 8 Novelty: 7

5. Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

ArXiv ID: 2607.27600

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Stephen Gould, Anton van den Hengel

Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years. Computational demands of large language models (LLMs) and their multi-modal variants during output generation can be partially alleviated by caching previous key and value calculations needed by subsequent scaled dot-product attention operations. However, this leads to another problem: the size of the resulting KV cache grows linearly with context length and quickly consumes all available GPU memory when either the prompt or the generated output are long. KV cache management periodically prunes entries from the cache thereby reducing its memory footprint while attempting to retain sufficient information for accurate generation. A by-product is faster inference speed. We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant and their associated keys and values can be removed from the cache. To score entries for eviction we run the model on the tokens in their original order, reusing the key and value representations already stored in the KV cache, and applying a counter-causal attention mask so that each position attends only to its future context. This is in-distribution, tied directly to the actual cache contents, and requires no additional training. To further reduce cost, we additionally propose a fast single-layer approximation that restricts the counter-causal pass to the last transformer layer, achieving a significant speedup per refresh cycle at marginal accuracy cost. We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods. Reference code is available at https://github.com/metacognitionai/counter_causal.

Comment: Evicts KV-cache entries by measuring how predictably past tokens can be reconstructed from future context.

Topic Match: The new counter-causal eviction criterion directly reduces long-context inference memory and computation without retraining.

Relevance: 8 Novelty: 7

6. RAP: KV-Cache Compression via RoPE-Aligned Pruning

ArXiv ID: 2602.02599

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, Marco Canini

Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \r{ho} = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.

Comment: Prunes KV projection dimensions in RoPE-aligned pairs to preserve positional semantics under cache compression.

Topic Match: The core mechanism directly compresses transformer KV state while preserving model quality.

Relevance: 8 Novelty: 6

7. FeatFix: Reuse What You Verify through Local Exact-Feature Correction for Faster Cached Diffusion Inference

ArXiv ID: 2607.27842

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hanshuai Cui, Zhiqing Tang, Zhi Yao, Qianli Ma, Fanshuai Meng, Weijia Jia

Abstract: Diffusion models are widely used to generate high-quality images and videos, but their iterative denoising process remains computationally intensive. A growing class of training-free accelerators reduces this cost by reusing cached intermediate features or forecasting future ones. To control draft drift, these methods sometimes compute an exact block feature for verification. Yet the resulting exact feature is typically used only to measure discrepancy or guide a later decision and is then discarded. We find that this previously computed feature can instead be reused for correction. Forwarding it at the verification site resets the local draft residual and reduces downstream feature error. Based on this observation, we introduce FeatFix, a local exact-feature correction method for cached diffusion inference. FeatFix operates at a fixed sparse set of layer--timestep sites. At each selected site, it replaces the complete draft block output with the exact output computed from the same incoming state, avoiding token- or channel-level partial replacement and full-timestep recomputation. Experiments across four image and video backbones show that FeatFix consistently accelerates generation, achieving a speedup of up to $6.70\times$ over Vanilla while maintaining competitive output quality.

Comment: Reuses exact verification features as local corrections to stabilize cached diffusion inference.

Topic Match: The method introduces a reusable cache-correction mechanism that substantially lowers diffusion inference cost.

Relevance: 7 Novelty: 7

8. SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

ArXiv ID: 2607.24787

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: MoE Training

Authors: Jinwei Kong, Runqi Meng, Fanyi Wang, Wentao Qiu, Haotian Hu, Yongjian Zhou, Zhenhua Ge

Abstract: Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory. Although expert offloading alleviates memory pressure by moving inactive experts to host memory or storage, it introduces a routing-dependent transfer bottleneck: required experts are known only after native top-(K) routing, which serializes routing, expert loading, and expert execution during inference. To address this bottleneck, we propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded MoE inference. SpecPrefetch uses a shared lightweight adapter to predict next-layer expert candidates only for asynchronous transfer, while the frozen native router still determines the final executed experts. By separating transfer prediction from execution routing, SpecPrefetch reduces exposed expert-loading latency without changing pretrained routing semantics, so prediction errors affect transfer efficiency rather than model outputs. In addition, a window-aware scheduler prioritizes feasible transfers under cache and bandwidth constraints. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with substantially fewer trainable parameters than learned predictor baselines. On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to (20\%) over a compute-optimized offloading runtime, demonstrating practical benefits for storage-constrained MoE deployment. The code and model weights are available at https://github.com/wei390/SpecPrefetch.

Comment: Predicts upcoming MoE experts solely for asynchronous prefetch while preserving the frozen native execution router.

Topic Match: Its main contribution reduces offloaded-MoE runtime cost, with expert prediction and scheduling as secondary MoE mechanisms.

Relevance: 7 Novelty: 7

9. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

ArXiv ID: 2607.28848

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, Oana Balmau

Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.

Comment: Harvests idle inference capacity for SLO-safe LoRA fine-tuning through latency-aware scheduling.

Topic Match: The primary contribution materially reduces fine-tuning hardware cost through a new co-serving systems design.

Relevance: 7 Novelty: 7

10. LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference

ArXiv ID: 2607.27952

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Feng Yang, Xinrui Ju, Keyang Zhang, Xiandong Meng, Rongqun Lin, Howard Leung, Shiqi Wang, Haoliang Li, Chris Xing Tian

Abstract: Multimodal foundation models are reshaping edge-cloud visual intelligence from task-specific feature pipelines into token-based interfaces, where edge devices encode visual inputs into tokens for a general-purpose cloud MLLM. However, dense visual-token sequences increase cloud-side inference costs. Existing pruning methods mainly target centralized inference: vision-driven methods can operate before cloud execution but are typically query-agnostic, whereas query-guided methods often rely on internal states of the target MLLM and cannot determine token relevance before transmission. Compact guidance models offer an alternative, but existing designs may require costly attention aggregation or auxiliary generation. We propose LAST, a training-free framework for query-dependent visual token pruning in edge-cloud collaborative MLLM inference. LAST uses a compact edge-side VLM as a guidance proxy and derives a lightweight importance signal from the last query token's attention to visual tokens. Under causal attention, the last query token can attend to the full visual sequence and the entire query context, enabling query-aware pruning without cloud-model access, autoregressive generation, or costly aggregation over multiple query positions. LAST then retains a diverse set of query-relevant visual tokens under a fixed token budget. We evaluate LAST on 11 multimodal benchmarks under multiple token budgets against pruning methods with different guidance strategies. Experiments show that LAST consistently achieves the strongest performance, preserving 95.4% of the full-token accuracy while retaining only 12.5% of the visual tokens, with low edge-side selection overhead and reduced cloud-side computation.

Comment: Prunes visual tokens before transmission using query-conditioned attention from a compact edge-side proxy model.

Topic Match: The core mechanism materially reduces multimodal-model token and cloud-compute costs.

Relevance: 7 Novelty: 6

11. Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models

ArXiv ID: 2603.05950

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Jialuo He, Huangxun Chen

Abstract: Visual token reduction is critical for accelerating Vision-Language Models (VLMs), since visual inputs are represented as token sequences that introduce substantial computational overhead in the LLM backbone. However, most pruning pipelines treat efficiency primarily as a token selection problem and retain a fixed visual token budget across inputs, overlooking the substantial variation in image information density. We propose E-AdaPrune, an energy driven adaptive pruning framework that determines an image specific token budget from the singular value spectrum of the visual feature matrix and passes this budget to existing token selectors. By preserving a certain proportion of spectral energy, our method allocates more tokens to information dense scenes while assigning fewer tokens to redundant scenes, without introducing additional learnable parameters. We evaluate E-AdaPrune across four VLM backbones, three token selectors, and nine benchmarks under matched average token budgets. Results show that E-AdaPrune removes a substantial amount of redundant computation from simple cases and converts the saved budget into larger gains on information rich cases. Notably, on SQA$^\mathrm{I}$ with Qwen2.5-VL-3B, E-AdaPrune uses 35.8\% fewer tokens for simple cases with only a 0.52\% relative performance decrease. The saved budget is redirected to hard cases, which receive 52.5\% more tokens and achieve a 1.94\% relative performance improvement.

Comment: Selects an image-specific visual-token budget from feature-spectrum energy.

Topic Match: Adaptive token pruning directly reduces VLM computation while introducing input-dependent dynamic computation.

Relevance: 7 Novelty: 6

12. Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization

ArXiv ID: 2607.27767

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Robert Jankowski, Pedro Almagro-Blanco, Marián Boguñá, Melanie Weber, M. Ángeles Serrano

Abstract: Graph neural networks (GNNs) can operate on large graphs but become infrastructure-sensitive at the scale of millions of nodes and typically require scalable training techniques for even larger graphs. This raises a central question: when can a model trained on a smaller, scaled-down replica of a graph be deployed on the full-resolution graph without retraining? We introduce a zero-shot transfer protocol in which a GNN is trained on a graph coarse-grained by geometric renormalization (GR), and the resulting weights are transferred directly to the original network. Across synthetic and real-world networks, training on GR scaled-down replicas preserves much of the original-scale predictive performance while significantly reducing training cost. We further find that learned representations and predictive trajectories remain aligned across scales. These findings suggest that structural similarity may be more important than network size in determining GNN transferability, opening a path toward scale-equivariant graph architectures.

Comment: Reduces GNN training cost by transferring weights from geometrically coarse-grained graphs to full-resolution graphs without retraining.

Topic Match: The method materially reduces training scale through structural renormalization, though only for graph models.

Relevance: 6 Novelty: 7

13. Transporting Task Vectors across Different Architectures without Training

ArXiv ID: 2602.12952

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, Simone Calderara

Abstract: Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains unexplored. In this work, we introduce Theseus, a training-free method for transporting task updates across heterogeneous-width models. Rather than matching parameters, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically. Code is available at https://github.com/apanariello4/merge-and-rebase.

Comment: Transfers task updates across model widths through activation-space alignment and closed-form functional matching.

Topic Match: Training-free reuse of adaptations lowers retraining cost while depending on cross-architecture representation geometry.

Relevance: 6 Novelty: 7

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains