This is a remedial run for missed papers from 08/04/2026 to 08/04/2026.

Results generated on 09/13/2026.

Personalized Daily ArXiv Papers 2026-08-05

_{Token counts are not reported for this run. 7 of 8 model calls succeeded, 2,854s of model wall clock.}

Topic Coverage:

Table of contents by topic:

Large-Scale Training Systems and Efficiency (3)

1. Muon Meets Mamba: Spectral Optimization for State Space Models Authors: Arslan Battalov, Karim Kramin, Alexander Markotenko, Sofia Sinitsina

2. When RL Meets Adaptive Speculative Training: A Unified Training-Serving System Authors: Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, Zelei Shao, Yubo Wang, Yinghui Liu, Qingyang Wu, Avner May, Sri Yanamandra, Ce Zhang, Tri Dao, Percy Liang, Ben Athiwaratkun, Shuaiwen Leon Song, Chenfeng Xu, Xiaoxia Wu

3. Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models Authors: Lele Zheng, Weifeng Kong, Xinyi Zhang, Ke Cheng, Tao Zhang, Yulong Shen

Architecture and Training Dynamics (15)

1. $π$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling Authors: Pike D. Liu, Chang Liu, Yanxuan Yu

2. Attention-based representations for multi-task computation Authors: Daniel Hsu, Mingyue Xu

3. Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining Authors: Yang Bai, Linyuan Wang, Haoyang Jiang, Nuolin Sun, Libin Hou, Bin Yan

4. On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds Authors: Jiaxin Deng, Junbiao Pang

5. MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models Authors: Tong Ling, Hang Lei, Feng Xiao, Changhui Sun, Jiahang Xie, Hao Liu, Lu Liu, Yanlong Du

6. Should the Boundary Term Be Learned in Reflected Diffusion? Conormal Trace and Reflection Masking Authors: Ziyue Wang, Takafumi Kanamori

7. Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise Authors: Juyan Zhang, Rhys Newbury, Xinyang Zhang, Tin Tran, Dana Kulic, Michael Burke

8. Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering Authors: Haotian Xia, Zilin Xiao, Junbo Zou, Vicente Ordonez, Hanjie Chen

9. The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner Authors: Simon Lam-Muir

10. Energy-Tweedie: Score meets Score, Energy meets Energy Authors: Andrej Leban

11. Compound and Parallel Modes of Tropical Convolutional Neural Networks Authors: Mingbo Li, Liying Liu, Charles Wiranto, Ye Luo

12. Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability Authors: Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi

13. 1-Lipschitz Neural Networks on Hadamard Manifolds Authors: Davide Murari, Marta Ghirardelli, Ben Adcock, Elena Celledoni, Brynjulf Owren, Carola-Bibiane Schönlieb

14. Equivariant Music Transformer Authors: Zixun Guo, Simon Dixon

15. Omega-S: A Functional Resilience Index for LLM Fine-Tuning Authors: Alberto Acedo

Efficiency, Compression, and Large-Scale Training (19)

1. Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse Authors: Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani

2. Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss Authors: Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús

3. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes Authors: Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan, Teeradaj Racharak, Naoya Inoue

4. LaPrune: Controllable Differentiable Sparsity at Million Scale Authors: Jakub Antczak, Joanna Wojciechowicz, Łukasz Struski, Jacek Tabor

5. Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging Authors: Siming Fu, Zheming Fu, Ruizhe He, Hualiang Wang, Jie Huang, Xiaoxiao Ma, Mingchen Zhong, Weihu Huang, Xiaoxuan He, Haojun Xu

6. x-Prediction Is All You Need:Training-Free Accelerated Generation via Endpoint Decodability Authors: Xin Peng, Ang Gao

7. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving Authors: Xiang Li, Pengcheng Wang, Huazheng Wang, Saurabh Bagchi

8. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs Authors: Yu Luo, Bo Dong, Wenhua Cheng, Haihao Shen

9. MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers Authors: Maoliang Li, Haojing Chen, Jiayu Chen, Zihao Zheng, Xinhao Sun, Hailong Zou, Xiang Chen

10. Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension Authors: Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar

11. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding Authors: Shuang Liang, Hao Mark Chen, Zhiwen Mo, Qianzhou Wang, Guoyu Li, Lingxiao Ma, Wayne Luk

12. Rex: A Family of Reversible Exponential (Stochastic) Runge-Kutta Solvers Authors: Zander W. Blasingame, Chen Liu

13. FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation Authors: Shenghui Li, Thiemo Voigt

14. Interpretable Adaptive Sampling for LLM Test-Time Scaling Authors: Mobina Kashaniyan, Ali Jannesari

15. When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding Authors: Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

16. Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models Authors: Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

17. Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs Authors: Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang, Zhenyu Lu, Bin Chen, Ke Chen, Yaowei Wang

18. Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectives Authors: Zhanglu Yan, Zhenyu Bai, Kaiwen Tang, Weng-Fai Wong

19. EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning Authors: Srinivas Anumasa, Dianbo Liu

Large-Scale Training Systems and Efficiency (3)

1. Muon Meets Mamba: Spectral Optimization for State Space Models

ArXiv ID: 2608.03941

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Arslan Battalov, Karim Kramin, Alexander Markotenko, Sofia Sinitsina

Abstract: Muon is a recent optimizer that orthogonalizes the update to each weight matrix with a Newton-Schulz iteration, which performs steepest descent under the spectral norm. Almost all the evidence for it comes from Transformer models, and its behavior on state-space models is largely unreported. We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon. The benefit is localized. Muon on the output projection alone beats Muon on the input projection or on both. The advantage is mainly one of token efficiency. It holds on two corpora and two token budgets, and persists when training continues well past the compute-optimal point. Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.

Comment: Isolates Muon's token-efficiency gain to Mamba's output projection under controlled pretraining.

Topic Match: The main result concerns optimizer placement and efficiency, with architecture-specific dynamics explaining its scope.

Relevance: 8 Novelty: 6

2. When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

ArXiv ID: 2602.06932

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, Zelei Shao, Yubo Wang, Yinghui Liu, Qingyang Wu, Avner May, Sri Yanamandra, Ce Zhang, Tri Dao, Percy Liang, Ben Athiwaratkun, Shuaiwen Leon Song, Chenfeng Xu, Xiaoxia Wu

Abstract: Speculative decoding can significantly accelerate LLM serving, yet most deployments today disentangle speculator training from serving, treating speculator training as a standalone offline modeling problem. We show that this decoupled formulation introduces substantial deployment and adaptation lag: (1) high time-to-serve, since a speculator must be trained offline for a considerable period before deployment; (2) delayed utility feedback, since the true end-to-end decoding speedup is only known after training and cannot be inferred reliably from acceptance rate alone due to model-architecture and system-level overheads; and (3) domain-drift degradation, as the target model is repurposed to new domains and the speculator becomes stale and less effective. To address these issues, we present Aurora, a unified training-serving system that closes the loop by continuously learning a speculator directly from live inference traces. Aurora reframes online speculator learning as an asynchronous reinforcement-learning problem: accepted tokens provide positive feedback, while rejected speculator proposals provide implicit negative feedback that we exploit to improve sample efficiency. Our design integrates an SGLang-based inference server with an asynchronous training server, enabling hot-swapped speculator updates without service interruption. Crucially, Aurora supports day-0 deployment: a speculator can be served immediately and rapidly adapted to live traffic, improving system performance while providing immediate utility feedback. Across experiments, Aurora achieves a 1.5x day-0 speedup on recently released frontier models (e.g., MiniMax M2.1 229B and Qwen3-Coder-Next 80B). Aurora also adapts effectively to distribution shifts in user traffic, delivering an additional 1.25x speedup over a well-trained but static speculator on widely used models (e.g., Qwen3 and Llama3).

Comment: Continuously retrains and hot-swaps a speculative decoder from live inference feedback.

Topic Match: The main contribution is an asynchronous training-serving architecture that closes the speculator adaptation loop.

Relevance: 6 Novelty: 7

3. Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models

ArXiv ID: 2608.03277

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Lele Zheng, Weifeng Kong, Xinyi Zhang, Ke Cheng, Tao Zhang, Yulong Shen

Abstract: Differentially private zeroth-order optimization (DP-ZO) enables memory-efficient private fine-tuning of large language models using only forward evaluations. Existing aggregation-based DP-ZO methods reconstruct model updates at a fixed scale, ignoring that the strength of useful signals varies throughout training. Consequently, noise-dominated updates may receive excessive weight and degrade model utility. To address this issue, we propose SAGE, a noise-aware shrinkage method that adaptively attenuates privatized estimates according to their estimated signal quality. SAGE subtracts the known Gaussian noise variance from the observed second moment to estimate the underlying signal energy, stabilizes this estimate through temporal tracking, and compares its current signal-to-noise level with a warm-up reference to derive a bounded shrinkage factor. As pure post-processing, SAGE requires neither additional privacy budget nor model queries and introduces only constant additional state. Our theoretical analysis shows that shrinkage reduces the quadratic update-risk term faster than the linear descent term, preserving useful descent while limiting the influence of noise-dominated updates. Experiments on RoBERTa-large, OPT-1.3B, and OPT-6.7B demonstrate that SAGE outperforms existing baselines in most settings under the same privacy budgets while preserving the forward-only memory efficiency of DP-ZO.

Comment: Adaptively shrinks differentially private zeroth-order updates using estimated signal-to-noise ratio.

Topic Match: The principal contribution is an optimizer modification for stable, memory-efficient LLM fine-tuning.

Relevance: 6 Novelty: 7

Architecture and Training Dynamics (15)

1. $π$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling

ArXiv ID: 2511.10696

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Pike D. Liu, Chang Liu, Yanxuan Yu

Abstract: Sparse attention is crucial in long-context Transformers, which restricts each token to a limited neighborhood and thereby reduces the quadratic cost of full self-attention. Local windows capture nearby context effectively, yet they induce a receptive-field bottleneck for dependencies beyond the window, limiting long-range modeling under moderate depth. In this paper, we propose $π$-Attention, an \emph{online efficient} sparse attention operator: as tokens arrive, each step maintains a streaming working set of local neighbors plus a $π$-indexed long-range fetch, fused by an adaptive prior under a shared softmax. Rather than materializing a global sparse mask in advance, $π$-Attention computes attention on the live working set with hierarchy-aware IO. We analyze causal reachability and minimum depth under this online rule, and show per-step cost remains $\mathcal{O}(k)$. Experiments on language modeling, Long Range Arena, and efficiency profiling---across 4K--32K context lengths---show consistent gains over local-window and other sparse baselines, approaching dense attention quality at linear cost.

Comment: Introduces online sparse attention with a local working set and indexed long-range fetch at linear cost.

Topic Match: The primary contribution is a new attention operator, with linear-complexity execution as a second match.

Relevance: 9 Novelty: 8

2. Attention-based representations for multi-task computation

ArXiv ID: 2608.04243

Primary Topic: Architecture and Training Dynamics

Authors: Daniel Hsu, Mingyue Xu

Abstract: Multi-head attention layers produce vector representations that support multiple downstream tasks. We establish bounds on the number of heads required in two simple and concrete multi-task scenarios. In the first scenario, a vector representation is sought so that linear predictors can compute both the smallest and largest numbers in a given list. In this case, it is known two attention heads with small embedding dimension and bit precision level suffice. We prove that a single attention head requires exponentially higher embedding dimension or precision level. In the second scenario, a vector representation is sought so that a polynomial threshold function can compute the XOR of a given string of $n$ bits. This scenario is analogous to the first one for $n=2$, since XOR is readily computed by a linear function using a vector representation that encodes both the AND and the OR of the two bits. We observe that $n$-bit XOR requires the product of the number of heads and the polynomial degree to be at least $n$, and we construct multi-head attention layers that match this lower bound. These results generalize to arbitrary (symmetric) Boolean functions, where the bound is given in terms of the threshold degree.

Comment: Proves head-count and representation-size lower bounds for multi-task attention computation.

Topic Match: The central result is a mechanistic capacity theory for multi-head attention representations.

Relevance: 8 Novelty: 8

3. Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

ArXiv ID: 2608.04213

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Yang Bai, Linyuan Wang, Haoyang Jiang, Nuolin Sun, Libin Hou, Bin Yan

Abstract: Existing studies on self-supervised learning for white-box networks typically decouple the derivation of white-box networks via optimization algorithms from self-supervised learning paradigms. In this work, we instead revisit the two components from a joint perspective. The LeJEPA-based self-supervised framework assumes an isotropic Gaussian distribution as the optimal embedding distribution for downstream tasks, which is conceptually equivalent to the expansion term $R(Z)$ in the sparse rate reduction objective guiding white-box Transformer optimization. Building on this observation, we use the LeJEPA self-supervised paradigm to optimize $R(Z)$, and derive the remaining terms $R^{c}(Z\mid U_{[K]})+λ\lVert Z\rVert_{0}$ via the alternating direction method of multipliers (ADMM) into an attention-only Transformer that dispenses with the ISTA structure or MLP layers of the original design. Experimental results demonstrate that our attention-only white-box Transformer achieves classification accuracies of $88.88\%$ on CIFAR-10 and $63.54\%$ on CIFAR-100 at the Base scale under the LeJEPA self-supervised paradigm, while the original white-box Transformer CRATE achieves classification accuracies of $89.18\%$ on CIFAR-10 and $63.56\%$ on CIFAR-100. Our model achieves competitive performance while reducing the parameter count by roughly $31\%$. Beyond the white-box setting, we further investigate standard ViTs and find that replacing all MLP blocks with ReLU activations under knowledge distillation removes approximately 66\% of the parameters while preserving competitive accuracy, motivating further investigation into the potential redundancy of MLP modules in standard ViT architectures.

Comment: Derives an attention-only white-box Transformer and removes MLP blocks with large parameter savings.

Topic Match: Eliminating the Transformer MLP is primarily an architectural change with substantial compression benefits.

Relevance: 8 Novelty: 7

4. On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

ArXiv ID: 2608.03197

Primary Topic: Architecture and Training Dynamics

Authors: Jiaxin Deng, Junbiao Pang

Abstract: Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $ρ$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies $λ{\max}\leq\sqrt[3]{bΓ/(2ρη^2)}$, where $λ$ is the largest Hessian eigenvalue, $b$ is the batch size, $η$ is the learning rate, and $Γ$ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: $ρ$ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing $ρ$ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts $ρ$ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.

Comment: Derives a stability bound linking SAM flatness to perturbation radius, batch size, and learning rate.

Topic Match: It directly analyzes optimization stability and the dynamics selecting flatter minima.

Relevance: 8 Novelty: 7

5. MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

ArXiv ID: 2608.03769

Primary Topic: Architecture and Training Dynamics

Authors: Tong Ling, Hang Lei, Feng Xiao, Changhui Sun, Jiahang Xie, Hao Liu, Lu Liu, Yanlong Du

Abstract: Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.

Comment: Makes masked-diffusion positional phases explicitly depend on the evolving revealed-token configuration.

Topic Match: The core contribution is a new positional-encoding mechanism tailored to masked diffusion language models.

Relevance: 8 Novelty: 7

6. Should the Boundary Term Be Learned in Reflected Diffusion? Conormal Trace and Reflection Masking

ArXiv ID: 2608.03469

Primary Topic: Architecture and Training Dynamics

Authors: Ziyue Wang, Takafumi Kanamori

Abstract: We study score learning for reflected diffusion on bounded domains. Reflection keeps trajectories feasible but does not ensure that the learned score satisfies the boundary behavior implied by the forward process. With implicit score matching, integration by parts leaves a boundary term, and we show that it depends on one scalar at each boundary point: the diffusion- weighted normal component of the score, or conormal trace. The no-flux condition fixes this value while leaving the re- maining boundary components unrestricted; under anisotropic diffusion it generally differs from the ordinary normal score component. On hyperrectangles, our parametrization enforces the required trace without additional trainable parameters or a stochastic boundary estimator and, under regularity assump- tions, can represent the true score, whereas fixing an incorrect value creates an error that more data cannot remove. We ex- tend the construction to simplices and polygonal domains and identify reflection masking: hard reflection can keep samples feasible even when the learned trace is wrong, so post-reflection metrics may hide the error. Experiments show the clearest separation with less frequent reflection, anisotropic diffusion, and mass near intersections of constraints; under full reflection, final sample placement improves inconsistently, illustrating how hard repair can mask boundary-score errors and decouple score accuracy from downstream generation quality.

Comment: Identifies the conormal score trace required at reflected-diffusion boundaries and exposes reflection masking.

Topic Match: It provides a mechanistic analysis and parameterization of score-learning stability at constrained boundaries.

Relevance: 7 Novelty: 8

7. Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise

ArXiv ID: 2508.01597

Primary Topic: Architecture and Training Dynamics

Authors: Juyan Zhang, Rhys Newbury, Xinyang Zhang, Tin Tran, Dana Kulic, Michael Burke

Abstract: Score Matching (SM) is a powerful framework for estimating the log-density derivatives of a distribution without calculating its normalizing constants. This capability has made it a cornerstone across multiple domains, from classical sta- tistical estimation and energy-based models to modern diffusion-based generative models. In practice, these models rely almost exclusively on Denoising Score Matching (DSM) as a tractable proxy for score matching. This ubiquity naturally raises a fundamental question: Is DSM truly "score matching for free"? In this work, we demonstrate that DSM is not a perfect substitute. We prove that the denoising objective is inherently heteroscedastic, the variance of model parame- ters fluctuates unpredictably based on both noise levels and the underlying data geometry. This instability is baked into the mathematical structure of the DSM. To address this, we derive an ideal weighting function that equalizes this variance, yielding a homoscedastic generalization of DSM. Since the ideal weights are of- ten empirically inaccessible, we show that a practical approximation weighting function via Taylor expansion reduces gradient variance during training, at the cost of statistical optimality. Notably, this provides a theoretical justification for an existing heuristic weight used in Isotropic Gaussian Diffusion. We validate our theory across different perturbed distributions and for higher-order scores.

Comment: Derives heteroscedastic gradient variance in denoising score matching and a variance-equalizing weighting rule.

Topic Match: It directly explains and mitigates an optimization instability in diffusion-model training objectives.

Relevance: 7 Novelty: 7

8. Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering

ArXiv ID: 2608.04124

Primary Topic: Architecture and Training Dynamics

Authors: Haotian Xia, Zilin Xiao, Junbo Zou, Vicente Ordonez, Hanjie Chen

Abstract: Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine video benchmarks and four multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20 tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5 tokens per query. Ablations further show that grounded perception latents, rationale-supervised reasoning latents, and adaptive routing each improve accuracy.

Comment: Dynamically routes from grounded perception latents to optional latent reasoning computation.

Topic Match: Adaptive latent computation and routing are the core mechanisms, despite evaluation being video-specific.

Relevance: 7 Novelty: 7

9. The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner

ArXiv ID: 2608.03263

Primary Topic: Architecture and Training Dynamics

Authors: Simon Lam-Muir

Abstract: We test whether the "compositional ignition" reported in latent-reasoning models is real computation, an instrument artifact, or inherited from verbal training data. We grow an independent realization of a published 30M-parameter recurrent-depth reasoner from scratch (same recipe and seed), film its development, certify fidelity through a pre-registered whole-signature gate, and measure resolution in two channels at once: the vocabulary readout and the hidden state. The ignition is real and lives at the readout: arrival time rises lawfully with problem depth, resolution is sharp and holds, and the signature reproduces across two same-seed realizations with divergent training trajectories. At commitment the decision margin jumps 5.8-8.0 logits in one iteration, exceeding the 90th percentile of near-threshold non-event steps in 96% of cases; the signed margin's zero-crossing there is definitional and carries no evidential weight, so the evidence is that conditioned magnitude. The hidden-state direction snaps in raw geometry, meeting its pre-registered criterion (in the decoder's LayerNorm coordinates it attenuates just below our bar, so the composite decoder-coordinate claim is not confirmed), and then freezes in both (descriptively so in decoder coordinates; angular steps 52.9 to 1.2 degrees over eight iterations), while subsequent displacement is predominantly radial (0.961 of squared-norm) and readout-null to a measured bound (radial logit effect <=5.7e-6). An earlier velocity-trough claim is withdrawn: pre-registered normalization controls showed it coordinate-dependent. Intermediates were never recoverable through the tied readout (relay 0.00). All criteria were frozen before their data; the predictions ledger, including this paper's own withdrawn headline, ships in the companion repository.

Comment: Locates depth-dependent recurrent reasoning commitment at a sharp vocabulary-readout transition.

Topic Match: The study directly analyzes how iterative recurrent computation evolves and commits across reasoning depth.

Relevance: 7 Novelty: 7

10. Energy-Tweedie: Score meets Score, Energy meets Energy

ArXiv ID: 2512.23818

Primary Topic: Architecture and Training Dynamics

Authors: Andrej Leban

Abstract: Denoising and score estimation are classically linked through Tweedie's formula, which relates the posterior mean under Gaussian noise to the Stein score of the noisy marginal. In this work, we extend this perspective beyond Gaussian noise to a broad class of Gibbs (energy-based) noise distributions, with the generalized Gaussian family as the running example. We derive the Energy-Tweedie identity: when the denoising posterior is viewed through the lens of scoring rules, the path derivative of a kernel scoring rule defined by the noise potential recovers the Stein score of the noisy marginal. The rule's propriety is determined by the noise potential alone. Thus, the familiar correspondence between Gaussian noise, posterior means, squared loss, and Tweedie's formula is lifted to a distributional correspondence between Gibbs noise distributions, full posterior laws, kernel scoring rules, and the Energy-Tweedie identity, yielding one Tweedie-style relation for each noise potential. Among its consequences, this identity gives a posterior-samples-to-score route to score estimation, yields a principled criterion for estimating unknown noise parameters, and enables diffusion-style sampling along user-chosen paths through the noise-parameter space, supplying the score-based perspective on recent generative methods trained with scoring rules.

Comment: Generalizes Tweedie's identity to Gibbs noise through noise-specific proper scoring rules.

Topic Match: The paper supplies a new theoretical foundation for score-model objectives and diffusion-style training.

Relevance: 6 Novelty: 8

11. Compound and Parallel Modes of Tropical Convolutional Neural Networks

ArXiv ID: 2504.06881

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Mingbo Li, Liying Liu, Charles Wiranto, Ye Luo

Abstract: Convolutional neural networks (CNNs) are foundational to many state-of-the-art computer vision systems, yet their reliance on multiplication-intensive computations poses challenges for deployment on resource-constrained devices. While tropical convolutional neural networks (TCNNs) reduce this computational burden by replacing multiplications with cheaper min/maxplus operations, they often do so at the cost of reduced model accuracy. To address this tradeoff, we introduce two novel extensions of tropical convolution: compound tropical convolution (cTCNN) and parallel tropical convolution (pTCNN). These operators combine minplus and maxplus algebraic operations within a single layer to enhance representational capacity while maintaining low computational cost. We provide an open-source implementation of these operators in a PyTorch-compatible framework, featuring optimized GPU kernels developed with TileLang. Through extensive experiments on image classification and semantic segmentation benchmarks, we demonstrate that our proposed cTCNN and pTCNN layers achieve competitive performance against standard CNNs while significantly reducing the number of multiplications. Moreover, we show that hybrid models, which integrate both tropical and conventional convolutions, can further improve the accuracy-efficiency balance. Our findings suggest that these tropical convolution variants are viable and effective components for building efficient deep learning models

Comment: Introduces compound and parallel min-plus/max-plus convolution operators to improve the accuracy–arithmetic-cost trade-off.

Topic Match: Reusable convolution operators are the core contribution, with multiplication savings providing an efficiency match; evidence remains limited to CNN vision workloads.

Relevance: 7 Novelty: 6

12. Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability

ArXiv ID: 2511.19716

Primary Topic: Architecture and Training Dynamics

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi

Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise. We analyze preconditioned SGD in the geometry induced by a symmetric positive definite matrix $\mathbf{M}$, deriving bounds in which both the convergence rate and the stochastic noise floor are governed by $\mathbf{M}$-dependent quantities: the rate through an effective condition number in the $\mathbf{M}$-metric, and the floor through the product of that condition number and the preconditioned noise level. For nonconvex objectives, we establish a preconditioner-dependent basin-stability guarantee: when smoothness and basin size are measured in the $\mathbf{M}$-norm, the probability that the iterates remain in a well-behaved local region admits an explicit lower bound. This perspective is particularly relevant in Scientific Machine Learning (SciML), where achieving small training loss under stochastic updates is closely tied to physical fidelity, numerical stability, and constraint satisfaction. The framework applies to both diagonal/adaptive and curvature-aware preconditioners and yields a simple design principle: choose $\mathbf{M}$ to improve local conditioning while attenuating noise. Experiments on a quadratic diagnostic and three SciML benchmarks validate the predicted rate-floor behavior.

Comment: Preconditioner-dependent bounds jointly characterize convergence rate, noise floor, and basin stability.

Topic Match: The central contribution is a training-dynamics theory for choosing stochastic-gradient preconditioners.

Relevance: 6 Novelty: 7

13. 1-Lipschitz Neural Networks on Hadamard Manifolds

ArXiv ID: 2607.19335

Primary Topic: Architecture and Training Dynamics

Authors: Davide Murari, Marta Ghirardelli, Ben Adcock, Elena Celledoni, Brynjulf Owren, Carola-Bibiane Schönlieb

Abstract: Controlling the Lipschitz constant of a neural network is a standard way to promote robustness and stability. Most existing constraining strategies are designed for Euclidean spaces. In this work, we construct and analyze a class of 1-Lipschitz neural networks on Hadamard manifolds. Our layers are of gradient-descent type, $1$-Lipschitz, and quasi-$α$-firmly nonexpansive. The core building blocks of the proposed architecture are Busemann functions, and we exploit the properties of Busemann gradient flows to design $1$-Lipschitz geometry-preserving layers. We provide explicit constructions and examples for hyperbolic manifolds and the manifold of symmetric positive definite (SPD) matrices. We test the proposed architecture in two numerical experiments: robust classification on the Poincaré disk and masked-Wishart covariance reconstruction. On the Poincaré disk, the proposed networks yield robust classifiers under hyperbolic perturbations. On the SPD manifold, we train SPD-valued denoisers and adopt them as a Plug-and-Play prior for a masked-Wishart covariance reconstruction problem. We show improved results from the nonexpansive denoiser over static, data-only, and Log-Euclidean denoising baselines, and empirically test its convergence properties.

Comment: Constructs 1-Lipschitz, geometry-preserving neural layers from Busemann gradient flows.

Topic Match: The core contribution is a new architectural mechanism for stable non-Euclidean neural computation.

Relevance: 6 Novelty: 7

14. Equivariant Music Transformer

ArXiv ID: 2608.03920

Primary Topic: Architecture and Training Dynamics

Authors: Zixun Guo, Simon Dixon

Abstract: Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.

Comment: Imposes time- and pitch-equivariant representations through an auxiliary self-distillation objective.

Topic Match: The main contribution is an architectural inductive bias plus analysis of how equivariance changes with scale and training.

Relevance: 6 Novelty: 7

15. Omega-S: A Functional Resilience Index for LLM Fine-Tuning

ArXiv ID: 2608.03887

Primary Topic: Architecture and Training Dynamics

Authors: Alberto Acedo

Abstract: Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

Comment: Mechanistic analysis identifies node-degree variance as the effective component of a weight-only regularizer.

Topic Match: Weight-level optimization analysis is adjacent to training dynamics, but the central contribution concerns capability retention during post-training fine-tuning.

Relevance: 6 Novelty: 6

Efficiency, Compression, and Large-Scale Training (19)

1. Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

ArXiv ID: 2608.03893

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani

Abstract: Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Comment: Transfers KV caches between model sizes using per-head ridge mappings, avoiding repeated prefill computation.

Topic Match: Cross-model cache transformation is a substantive KV-reuse mechanism that reduces inference computation, with demonstrated applicability constrained by model-pair compatibility.

Relevance: 9 Novelty: 8

2. Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

ArXiv ID: 2608.03796

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús

Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-$K$ logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29\% faster per iteration, and reaching up to 41\% higher throughput on a single H200 GPU. Second, we introduce a \emph{fused, chunked KL loss} that never materialises the full vocabulary-sized logit tensor, making peak memory linear in the sequence length. This removes the memory spike that otherwise caps context length and lets us train at four times the context (32{,}768 tokens) on a single GPU. A separate output-head-only toy benchmark isolates the loss kernel and confirms its memory and iteration-rate scaling from 4K to 256K tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.

Comment: Combines cached top-K teacher logits with a fused chunked KL kernel that avoids full-vocabulary materialization.

Topic Match: Efficient model distillation is primary, while the fused loss kernel also directly improves training-system memory and throughput.

Relevance: 9 Novelty: 7

3. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

ArXiv ID: 2608.03839

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan, Teeradaj Racharak, Naoya Inoue

Abstract: Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.

Comment: Reuses verifier hidden states to retrieve semantic speculative drafts, increasing accepted draft length without additional model training.

Topic Match: The core contribution is a new speculative-draft retrieval mechanism that reduces LLM decoding cost.

Relevance: 9 Novelty: 7

4. LaPrune: Controllable Differentiable Sparsity at Million Scale

ArXiv ID: 2608.04057

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jakub Antczak, Joanna Wojciechowicz, Łukasz Struski, Jacek Tabor

Abstract: Top-$k$ selection determines which components of a sparse model remain active. Hard selection blocks gradients, while continuous relaxations often couple mask hardness to the selected mass. We introduce LaPrune, a mathematically exact-budget differentiable layer that controls the normalized second moment while preserving the selected mass. A LapSum barrier preserves the selection mass, and a normalized second-moment constraint moves the mask from a dense equal-mass allocation toward hard top-$k$ at each budget. We derive a population prediction of the saturated fraction, a near-binary limiting law, and a tight worst-case guarantee on the near-zero fraction. The normalized hardness parameter is invariant to score scale, while a fixed LapSum temperature is not.

Comment: An exact-budget differentiable top-k layer independently controls selected mass and mask hardness.

Topic Match: The core contribution is a new differentiable sparsification mechanism with exact budgets at million-scale width.

Relevance: 8 Novelty: 8

5. Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging

ArXiv ID: 2608.03316

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Siming Fu, Zheming Fu, Ruizhe He, Hualiang Wang, Jie Huang, Xiaoxiao Ma, Mingchen Zhong, Weihu Huang, Xiaoxuan He, Haojun Xu

Abstract: On-policy distillation, in which a teacher corrects samples that the student itself generates, presupposes that the two models speak the same language: identical VAE latents, matching architectures, and a common timestep grid. We ask what happens when none of this holds, as when the strongest teacher available and the student one wishes to deploy come from different model families, and find that the standard recipes have no answer: teacher latents cannot serve as targets in a foreign coordinate system, per-pixel losses against a teacher that stochastically re-draws local detail degenerate into blur or divergence, and timestep indices lose their meaning across mismatched schedules. We present Any-OPD, to our knowledge the first framework for on-policy distillation between arbitrary pairs of latent flow-matching generators. Any-OPD treats the teacher purely as a black-box sampler and connects the two models at exactly one point: a frozen, model-agnostic vision representation in which their independently decoded outputs are compared, sidestepping every assumption about latents, features, or architecture. Trajectory correspondence is recovered by matching continuous noise levels instead of step indices, and a brief anchoring phase, in which teacher samples are re-encoded through the student's own VAE, ensures the on-policy gradient measures sample quality rather than domain mismatch. Distilling the 12B FLUX.1-dev into the 2.5B SD3.5-Medium, Any-OPD lifts the student's PickScore from 0.846 to 0.884 and HPSv3 from 9.12 to 10.97, rivaling the teacher at a fifth of its size, where direct latent regression fails to train at all.

Comment: Enables cross-architecture on-policy distillation through a model-agnostic representation-space bridge.

Topic Match: The core result compresses a large flow model into a heterogeneous smaller model without shared latent spaces.

Relevance: 8 Novelty: 8

6. x-Prediction Is All You Need:Training-Free Accelerated Generation via Endpoint Decodability

ArXiv ID: 2607.06114

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Xin Peng, Ang Gao

Abstract: Diffusion and flow matching models generate high-quality samples, but their ODE samplers often need tens to hundreds of neural function evaluations (NFEs). This remains a practical challenge for released checkpoints, since many accelerators require additional design choices and training cost through retraining, distillation, or trajectory redesign. We investigate a different route based on $x$-prediction. During sampling, standard affine probability paths already expose $x_0$ information: an intermediate state and its path velocity determine a principled estimate of the clean sample. We formalize this property as \textbf{endpoint decodability} and show that the decoder is the minimum-MSE estimator $\mathbb{E}[x_0\mid x_t]$ under the usual $\ell_2$ objective. This yields \textbf{Truncated Jump Sampling} (TJS): stop the ODE at an early-exit time $t^*$ and return the decoded $x_0$. TJS requires no retraining, distillation, or architecture change. Across SDXL, SD3.5M, Z-Image-Turbo, and three class-conditional benchmarks, it reduces NFEs by 20--70\% with near-matched quality. The analysis also shows why endpoint prediction can work without straightening the trajectory, providing inference acceleration without trajectory redesign.

Comment: Decodes the clean endpoint from an intermediate flow state, cutting sampling evaluations without retraining.

Topic Match: The method directly reduces diffusion and flow-model inference cost by enabling principled early exit.

Relevance: 8 Novelty: 8

7. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

ArXiv ID: 2608.03579

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Xiang Li, Pengcheng Wang, Huazheng Wang, Saurabh Bagchi

Abstract: Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Conversely, ultra-low-rank adapters ($r \le 2$) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this problem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine-tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand. Across LLMs of varying scales, SALT recovers high-rank accuracy using $r \le 2$ residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per-adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B.

Comment: Trains shared domain centroids plus ultra-low-rank task residuals to minimize adapter swapping cost.

Topic Match: Low-rank compression is primary, with serving-aware training and memory movement providing a systems match.

Relevance: 8 Novelty: 8

8. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

ArXiv ID: 2608.04048

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Yu Luo, Bo Dong, Wenhua Cheng, Haihao Shen

Abstract: Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.

Comment: Stacks 2-bit residual corrections over a quantized base to support multiple weight precisions from one checkpoint.

Topic Match: Progressive weight quantization directly addresses LLM precision and memory trade-offs; residual stacking is a meaningful but incremental mechanism.

Relevance: 9 Novelty: 6

9. MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers

ArXiv ID: 2606.15615

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Maoliang Li, Haojing Chen, Jiayu Chen, Zihao Zheng, Xinhao Sun, Hailong Zou, Xiang Chen

Abstract: Diffusion Transformers with Mixture-of-Experts (DiT-MoE) improve model capacity under sparse activation, but diffusion inference is still bottlenecked by redundant computation across timesteps. Existing caching methods mainly operate at the token level, which becomes suboptimal in DiT-MoE because each token update is internally decomposed into multiple routed expert branches. Our analysis shows that cross-timestep redundancy in DiT-MoE is better characterized at the expert-branch level than at the whole-token level. Based on this observation, we propose MoECa, a fine-grained caching framework that performs branch-level feature reuse across timesteps. MoECa further introduces expert-aware adaptive control and synchronized cache updates across MoE and attention paths to maintain stable intermediate states. Experiments on multiple DiT-MoE models show a favorable speed--quality trade-off, with up to 2.93$\times$ speedups while preserving generation quality.

Comment: Caches routed expert branches independently to reuse DiT-MoE features across diffusion timesteps.

Topic Match: The contribution is expert-aware inference caching that materially lowers DiT-MoE generation cost.

Relevance: 8 Novelty: 7

10. Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension

ArXiv ID: 2608.03494

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar

Abstract: Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency. We present a systematic study of more than 20 initialization strategies for Hindi vocabulary extension in Nemotron-3-Nano-30B-A3B. Our comparison spans vocabulary-averaging baselines; external and learned initialization methods, including FOCUS, top-k semantic retrieval, and residual MLP mappings; subword composition; norm calibration; and input-output asymmetry. We find that subword composition methods outperform both vocabulary averaging and external/learned initialization approaches. Within subword composition, asymmetric variants achieve the lowest observed early validation loss and reveal distinct preferences for input and output embedding initialization. The best observed configuration initializes the input embedding matrix with uniform subword averaging and Hindi-specific norm calibration, and the output language modeling head with character-length-weighted subword averaging. Relative to the standard Mean-all baseline, this full initialization pipeline reaches comparable validation loss with over a 6x reduction in CPT steps and exceeds the baseline's 3,500-step MILU-Hindi accuracy after only 500 steps. Finally, we show that initialization loss and initialization bits-per-byte (Init BPB) are unreliable predictors of downstream convergence, whereas lightweight CPT, as few as 50 steps, provides a cost-effective and reliable signal for selecting the best initialization strategy.

Comment: Subword-composed embedding initialization cuts 30B-MoE continued-pretraining steps by more than sixfold.

Topic Match: The strongest contribution is reducing large-model adaptation compute through better vocabulary initialization.

Relevance: 8 Novelty: 6

11. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

ArXiv ID: 2608.02989

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: MoE Training

Authors: Shuang Liang, Hao Mark Chen, Zhiwen Mo, Qianzhou Wang, Guoyu Li, Lingxiao Ma, Wayne Luk

Abstract: Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model-task pairs spanning three MoE targets and four benchmarks, AcceptMoE's mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290 times the throughput of this baseline with all expert weights in GPU memory, and 2.06 times under physical expert offloading, while reducing host-to-device traffic by 73.6 percent to 77.1 percent.

Comment: Commitment-weighted expert eligibility dynamically shrinks MoE verification and offloading costs.

Topic Match: Its principal result is lower inference traffic and latency, with verifier-side MoE routing as the enabling mechanism.

Relevance: 7 Novelty: 7

12. Rex: A Family of Reversible Exponential (Stochastic) Runge-Kutta Solvers

ArXiv ID: 2502.08834

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zander W. Blasingame, Chen Liu

Abstract: Deep generative models based on neural differential equations have become state-of-the-art for many generation tasks. These models rely on ODE/SDE solvers that integrate from a prior distribution to the data distribution; in many applications it is also highly desirable to integrate in the inverse direction. Standard solvers, however, accumulate discretization errors that prohibit exact inversion, an inaccuracy that is unacceptable in precision-critical applications. Existing inversion methods suffer from poor stability and low order of convergence, and are strictly limited to the ODE setting. In this work, we propose Rex, a family of reversible exponential (stochastic) Runge-Kutta solvers obtained by applying Lawson methods to convert any explicit (stochastic) Runge-Kutta scheme into an algebraically reversible one for both diffusion ODEs and SDEs. Beyond a rigorous theoretical analysis -- establishing arbitrary-order convergence and a non-zero region of linear stability -- we empirically demonstrate that Rex achieves near-machine-precision reconstruction and improves Boltzmann sampling with flow models as well as image generation and editing with diffusion models.

Comment: Transforms explicit Runge-Kutta schemes into high-order algebraically reversible ODE and SDE solvers.

Topic Match: It provides a new numerical mechanism for accurate and reversible generative-model inference.

Relevance: 6 Novelty: 8

13. FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

ArXiv ID: 2608.03605

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Shenghui Li, Thiemo Voigt

Abstract: Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.

Comment: Coordinate-space recompression exactly aggregates federated LoRA updates before adaptive rank reduction.

Topic Match: Efficient low-rank recompression is primary, with federated aggregation and communication savings as systems benefits.

Relevance: 7 Novelty: 6

14. Interpretable Adaptive Sampling for LLM Test-Time Scaling

ArXiv ID: 2608.03961

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Mobina Kashaniyan, Ali Jannesari

Abstract: Test-time scaling improves LLM reasoning by generating and aggregating multiple candidate answers, yet many pipelines use fixed per-query budgets that spend the same compute on easy and difficult prompts. These fixed budgets are also difficult to inspect because they do not explain why a given prompt receives a particular number of samples. We propose adaptive} test-time scaling with a lightweight fuzzy controller that maps interpretable signals, including estimated prompt complexity and model confidence, to a per-query sampling budget. The controller assigns fewer samples to easier or more confident prompts and more samples to harder or less certain prompts, making inference-time compute inspectable rather than fixed or opaque. We evaluate under a fair-alignment protocol with matched decoding settings and controlled answer selection, and compare against best-of-$N$, compute-aware scaling, and self-certainty-based baselines on question-answering and mathematical reasoning tasks. Across models and datasets, adaptive fuzzy control improves over several standard baselines and remains close to a selector-matched full-budget control while reducing the average number of samples. These findings suggest that interpretable adaptive sampling is a practical direction for more efficient test-time reasoning in large language models.

Comment: Allocates LLM test-time samples per query using interpretable complexity and confidence signals.

Topic Match: It directly reduces inference-time scaling cost through adaptive compute allocation.

Relevance: 7 Novelty: 6

15. When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

ArXiv ID: 2608.03918

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

Abstract: Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence. Existing relevance-based methods rely on static one-shot selection with fixed frame budgets and candidate pools, while agent-based schedulers achieve adaptivity through costly multi-round reasoning and interactive search. We propose EcoFrame, a training-free framework for low-overhead query-adaptive visual evidence scheduling. EcoFrame leverages the VLM's inference feedback to determine when to increase the frame budget and where to search for additional candidate evidence. Specifically, entropy-gated budget scheduling uses output uncertainty to stop early when the current evidence is sufficient or progressively expand the frame budget otherwise. Meanwhile, attention-guided candidate proposal converts frame-level attention into a temporal prior, enabling dense local search in informative regions while preserving global coverage when attention is diffuse. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones. On Qwen2.5-VL, EcoFrame achieves an average accuracy of 64.4, surpassing BOLT at 63.5, while providing a $1.85\times$ speedup over AKS and BOLT. Compared with the agent-based A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5\times$ inference speedup. Code will be available at https://github.com/AK-DREAM/EcoFrame.

Comment: Adaptively expands frame budgets and search regions using VLM uncertainty and attention feedback.

Topic Match: The central mechanism reduces VLM inference cost through query-dependent visual-compute scheduling.

Relevance: 7 Novelty: 6

16. Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

ArXiv ID: 2608.03112

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

Abstract: Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of tokens per image, limiting their deployment on resource-constrained edge devices and in real-time surveillance applications. This challenge is further amplified in video processing, where multiple frames must be analyzed simultaneously. Existing token reduction techniques are largely developed for single-image inputs and therefore fail to account for the temporal and inter-frame redundancies present in video sequences. In addition, these methods generally rely on a fixed, uniform pruning ratio applied across all inputs, which is suboptimal because the degree of redundancy can vary significantly between different videos, necessitating content-dependent pruning levels to preserve critical information. To address these limitations, we propose a two-stage adaptive token pruning strategy specifically designed for video processing. In the first stage, we prune out the redundant frames, and in the second stage, token-level pruning is applied within the retained frames. Crucially, the pruning ratio in the second stage is determined adaptively based on the content of each video. This is achieved by analyzing the correlation structure of token embeddings to quantify redundancy, which is used to determine the ratio. Importantly, our method is entirely post-hoc and requires no additional training or fine-tuning, while achieving strong empirical gains; notably, it improves accuracy by +7\% on a video captioning benchmark at 10\% token retention, while reducing computation TFLOPs by 95\%.

Comment: Prunes redundant video frames first and then adaptively prunes tokens according to embedding redundancy.

Topic Match: Post-hoc token pruning is the central mechanism, reducing video-language-model FLOPs by up to 95 percent.

Relevance: 7 Novelty: 6

17. Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

ArXiv ID: 2608.03450

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang, Zhenyu Lu, Bin Chen, Ke Chen, Yaowei Wang

Abstract: Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.

Comment: Switches between explicit and latent multimodal reasoning using a vision-to-text attention ratio.

Topic Match: The main benefit is lower inference cost, enabled by a dynamic-computation switching mechanism.

Relevance: 6 Novelty: 7

18. Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectives

ArXiv ID: 2409.08290

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhanglu Yan, Zhenyu Bai, Kaiwen Tang, Weng-Fai Wong

Abstract: Spiking Neural Networks (SNNs) promise higher energy efficiency over conventional Quantized Artificial Neural Networks (QNNs) due to their event-driven, spike-based computation. However, prevailing energy evaluations often oversimplify, focusing on computational aspects while neglecting critical overheads like comprehensive data movements and memory accesses. Such simplifications can lead to misleading conclusions regarding the true energy benefits of SNNs. This paper presents a rigorous re-evaluation. We establish a fair baseline by mapping rate-encoded SNNs with $T$ timesteps to capacity-matched QNNs with $\lceil \log_2(T+1) \rceil$ bits. This ensures both models have comparable representational capacities, as well as similar hardware requirements, enabling meaningful energy comparisons. We introduce a detailed analytical energy model encompassing core computation and data movements. Using this model, we systematically explore a wide parameter space, including intrinsic network characteristics (SNN time window size, spike rate, QNN sparsity, model size, weight bit-level) and hardware characteristics (memory system and network-on-chip). Our analysis identifies specific operational regimes where SNNs genuinely offer superior energy efficiency. For example, under typical neuromorphic hardware conditions, SNNs with moderate time windows ($T = 5$) require an average spike rate ($s_r$) below 5.7% to outperform equivalent QNNs These insights guide the design of truly energy-efficient neural network solutions.

Comment: Derives operational regimes in which sparse SNN inference is genuinely more energy-efficient than matched quantized networks.

Topic Match: The core contribution is a detailed cost model connecting sparsity, quantization, memory traffic, and hardware energy.

Relevance: 6 Novelty: 7

19. EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

ArXiv ID: 2608.01142

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Srinivas Anumasa, Dianbo Liu

Abstract: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation. We introduce EulerLoRA, a stochastic extension of LoRA that generates multiple predictive trajectories by sampling structured variations along the rank-one components of shared low-rank adapters, while preserving the deterministic LoRA transformation in expectation. We evaluate EulerLoRA with vision transformers on CIFAR-10, CIFAR-100, and HAM10000, together with out-of-distribution detection on SVHN. Across these benchmarks, EulerLoRA achieves comparable or improved performance relative to strong LoRA-Ensemble baselines. Using two rank-20 adapters, EulerLoRA requires approximately 3 million trainable adapter parameters, compared with about 10 million for a rank-8, 16-adapter LoRA-Ensemble, corresponding to roughly 69% fewer trainable parameters. These results show that useful predictive diversity can be obtained from a small number of shared adapters.

Comment: Structured stochastic sampling over LoRA rank-one components yields uncertainty with fewer adapter parameters.

Topic Match: The method extends low-rank adaptation while substantially reducing parameters relative to adapter ensembles.

Relevance: 6 Novelty: 6

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains