This is a remedial run for missed papers from 09/01/2026 to 09/01/2026.

Results generated on 09/14/2026.

Personalized Daily ArXiv Papers 2026-09-02

_{Token counts are not reported for this run. 27 of 27 model calls succeeded, 3,556s of model wall clock.}

Topic Coverage:

Table of contents by topic:

Large-Scale Training Systems and Efficiency (4)

1. Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining Authors: Jingru Li, Yibo Fan, Huan Li

2. Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network Authors: Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing, Zeji Li, Kaijie Chen, Harikrishna Kuttivelil, Katia Obraczka

3. FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs Authors: Jiaxiang Geng, Tianjun Yuan, Pengchao Han, Ying Gao, Xianhao Chen, Bing Luo

4. Scaling an Autoregressive Transformer for Single-Cell Generation Authors: Aleksandr Sharipov, Yusif Mukhtarov, Igor Molybog

Architecture and Training Dynamics (18)

1. Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective Authors: Ruoxi Yu, Haotian Jiang, Jingpu Cheng, Penghao Yu, Qianxiao Li, Zhong Li

2. Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture Authors: Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma

3. Multi-Head Self Attention is a Parameter Identification Mechanism Authors: W. Ross Morrow

4. Flow Reasoning Models: Turning Flows Into Efficient Recurrent Reasoners Authors: Alec Helbling, Andrey Bryutkin, Mauro Martino, Duen Horng Chau, Nima Dehmamy, Hendrik Strobelt

5. FloydNet: A Learning Paradigm for Global Relational Reasoning Authors: Jingcheng Yu, Mingliang Zeng, Qiwei Ye

6. The Topological Trouble With Transformers Authors: Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne Liu

7. Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis Authors: Minsik Choi, Geewook Kim, Young Geun Kim

8. Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs Authors: Zhaoliang Chen, Jie Fu

9. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method Authors: Steffen Dereich, Thang Do, Arnulf Jentzen

10. Denoising Diffusion Generative Models Secretly Calculate Attentions Authors: Farzan Haddadi, Leila Monfared, Ebrahim Rezaii, Mohammadreza Malek-Mohammadi, Pejman Zakalvand, Narges Mokhtari

11. Instella-MoE Technical Report Authors: Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum

12. RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis Authors: Amirhosein Azarpour

13. From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion Authors: Satoshi Hayakawa

14. A Study of Hidden-State Optimization Order in Predictive Coding Networks Authors: Xueyuan Li, Danilo Vasconcellos Vargas

15. FLaG: Frequency-Domain Latent-attention Gated Pooling for Token Aggregation Authors: Kewei Li, Rongying Zhang, Xueli Wang, Xiwen Gong, Zhongjian Wang, Qiuchen Zhao, Lan Huang, Ruochi Zhang, Fengfeng Zhou

16. When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection Authors: Zhengyu Hu, Zheyuan Xiao, Linxin Song, Fengqing Jiang, Yuetai Li, Zhihan Xiong, Yue Liu, Junhao Lin, Yao Su, Lijie Hu, Kaize Ding, Teng Xiao, Radha Poovendran

17. Wave Function Backpropagation with Explicit Temporal-Interval Dynamics Authors: Byunggu Yu, Justin Kim

18. Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU Authors: Jingzhou Liu

Efficiency, Compression, and Large-Scale Training (20)

1. Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation Authors: Wentao Ye, Zhanming Shen, Zhiqing Xiao, Yao Ding, Haobo Wang, Gang Chen

2. CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing Authors: Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt, Ryan A. Rossi, Linh Ngo Van, Jieyang Chen, Thien Huu Nguyen

3. IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models Authors: Dong-Jae Lee, Sunghyun Baek, Junmo Kim

4. The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally Authors: Jundong Hu, Shekar Ramachandran

5. CacheBridge: Efficient Cross-Model KV Cache Transfer Authors: Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin

6. Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers Authors: Yan Wang

7. Stream-CQSA: Exact Out-of-Memory Recovery for Attention Authors: Yiming Bian, Joshua M. Akey

8. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing Authors: Young D. Kwon, Miles Williams, Rui Li, Alexandros Kouris, Stylianos I. Venieris

9. LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment Authors: Haonan He, Xinyue Fan

10. CAT-Flow: Curvature-Adaptive sTeps for Flow Matching Authors: Qinchan Li, Pedro Cisneros-Velarde, Keru Fu, Samuel Antunes Miranda, Sharan Vaswani, Hao Zhang

11. HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation Authors: Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos, Giuseppe Ateniese, Emanuele RodolÃ

12. mzCache: On-Device LLM Memory Management under Multitasking Authors: Hongseung Yu, Minsung Kim, Jongseok Park, Kyunghan Lee

13. MineDraft: A Framework for Batch Parallel Speculative Decoding Authors: Zhenwei Tang, Arun Verma, Zijian Zhou, Zhaoxuan Wu, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low

14. KV Cache Offloading for Context-Intensive Tasks Authors: Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev, Vyacheslav Zhdanovskiy, Yegor Yershov

15. SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models Authors: Shiyu Li, Zi-Yuan Hu, Shijia Huang, Yanyang Li, Yiwu Zhong, Liwei Wang

16. DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models Authors: Jungseob Lee, Seongtae Hong, Seungjun Lee, Jaehyung Seo, Junyoung Son, Sugyeong Eo, Chanjun Park, Hyeongju Park, Hyeonseok Moon, Heuiseok Lim

17. Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity Authors: Lei Wang, Jieming Bian, Letian Zhang, Jie Xu

18. Relational Representation Distillation Authors: Nikos Giakoumoglou, Tania Stathaki

19. TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories Authors: Daniel Agyei Asante, Yang Li

20. Control Variate Score Matching for Diffusion Models Authors: Khaled Kahouli, Romuald Elie, Klaus-Robert Müller, Quentin Berthet, Oliver T. Unke, Arnaud Doucet

Large-Scale Training Systems and Efficiency (4)

1. Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining

ArXiv ID: 2601.14603

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Jingru Li, Yibo Fan, Huan Li

Abstract: Optimizer design plays a central role in efficient language model pretraining, directly affecting optimization dynamics, convergence speed, and compute cost under fixed training budgets. Muon has emerged as a strong optimizer by orthogonalizing momentum updates, yielding a matrix-valued analogue of sign-based normalization. However, unlike Adam-style methods, Muon does not explicitly incorporate gradient-variance information into its updates. Motivated by Adam's variance-adaptive interpretation, we propose Muon-NSR and Muon-VS, two variance-adaptive Muon variants for language model pretraining. Muon-NSR applies noise-to-signal ratio (NSR) modulation before Newton--Schulz orthogonalization, whereas Muon-VS uses variance scaling (VS) without introducing any additional hyperparameters beyond those of Muon. Both methods preserve Muon's spectral normalization structure while requiring only one additional variance buffer. Experiments on Llama-style and GPT-2 pretraining across model scales from 125M to 1.2B parameters show that our methods improve over well-tuned Muon baselines and remain competitive with representative adaptive Muon-family baselines. On Llama-1.2B, Muon-VS achieves a 1.33$\times$ step-to-target speedup over a well-tuned Muon baseline, with Muon's final validation loss as the target. These results indicate that variance-adaptive modulation is a simple and effective mechanism for improving Muon-style optimizers in language model pretraining.

Comment: Variance-adaptive Muon updates deliver a reported 1.33x step-to-target speedup during pretraining.

Topic Match: Pretraining optimizer design and measured reductions in steps needed to reach a target loss directly match training_systems.

Relevance: 10 Novelty: 6

2. Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

ArXiv ID: 2609.01952

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing, Zeji Li, Kaijie Chen, Harikrishna Kuttivelil, Katia Obraczka

Abstract: Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate $O(1/(ηT))$ to an $O(η)+O(B_f^2)+O(ζ_f^2)$ neighbourhood. Here $B_f$ is the distance from the task optimum to the peers' reachable classes and $ζ_f$ measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by $40-61\times$, while isolated training does not. The sampled stationarity diagnostic has late transient exponents $0.99-1.90$ on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Comment: Derives convergence guarantees for asynchronous, heterogeneous-model gossip distillation through function-space consensus.

Topic Match: Asynchronous optimization and decentralized consensus directly concern distributed training, although applicability to large-scale pretraining remains unestablished.

Relevance: 7 Novelty: 7

3. FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

ArXiv ID: 2508.10349

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Jiaxiang Geng, Tianjun Yuan, Pengchao Han, Ying Gao, Xianhao Chen, Bing Luo

Abstract: To fine-tune large language models (LLMs) over private data, federated learning (FL) has emerged as a promising paradigm. However, the prohibitive memory and communication demands of LLMs render standard FL impractical for resource-constrained edge devices. While split federated learning (SFL) alleviates the computing burdens via model partitioning, existing frameworks still suffer from communication bottlenecks and straggler problem due to the parameter aggregation process. To address these challenges, we propose FlexP-SFT, a novel aggregation-free framework for personalized split federated fine-tuning, which fundamentally eliminates the client-side aggregation process. Crucially, to ensure robust training in the absence of global synchronization, we introduce a layer-flexible alignment strategy to balance personalization and generalization capabilities. We further formulate split-ratio selection as a resource-aware discrete optimization problem that jointly accounts for personalization accuracy and system cost. Our proposed scheme simultaneously enhances personalized performance, reduces communication overhead, and resolves the straggler problem. Extensive results show that FlexP-SFT substantially outperforms baselines in both accuracy and latency, and that the optimized split ratio achieves a better resource-accuracy trade-off than static or memory-only choices.

Comment: Removes client aggregation from split federated LLM training to reduce synchronization, communication, and straggler costs.

Topic Match: The aggregation-free split-training protocol contributes a distributed training method; its personalized edge fine-tuning setting makes it less central than large-scale pretraining systems.

Relevance: 7 Novelty: 7

4. Scaling an Autoregressive Transformer for Single-Cell Generation

ArXiv ID: 2608.02961

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Aleksandr Sharipov, Yusif Mukhtarov, Igor Molybog

Abstract: We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.

Comment: Joint parameter-and-data scaling laws derive a compute-optimal pretraining frontier.

Topic Match: The scaling analysis directly informs model-size and training-data allocation under compute constraints, although its evidence is specific to single-cell generation.

Relevance: 7 Novelty: 6

Architecture and Training Dynamics (18)

1. Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

ArXiv ID: 2510.03784

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Ruoxi Yu, Haotian Jiang, Jingpu Cheng, Penghao Yu, Qianxiao Li, Zhong Li

Abstract: Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, we investigate how the model parameters -- mainly attention heads and head dimensions -- should be allocated across layers to balance expressivity and efficiency. We first provide mathematical analysis on the role of early layers in information extraction from an approximation perspective, with a theoretical characterization on the trade-off between the number of heads and head dimension under a fixed parameter budget. In addition, we uncover and prove the \emph{saturation} behavior of softmax activations: Continuously increasing head dimensions can lead to diminishing returns in learning errors, particularly for long sequences. Supported by both theory and experiments, this saturation pattern suggests that later layers can operate more efficiently with reduced parameters. Combining these insights, we propose principled strategies for allocating attention heads and dimensions across Transformers' layers, shedding light on theoretically-grounded model efficiency of Transformer-based architectures.

Comment: Uses softmax saturation and fixed-budget theory to guide layerwise allocation of attention heads and head dimensions.

Topic Match: Architectural analysis explains diminishing returns from attention dimensions and translates that insight into more efficient parameter allocation.

Relevance: 9 Novelty: 7

2. Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

ArXiv ID: 2506.19935

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma

Abstract: Efficiently scaling Large Language Models (LLMs) necessitates exploring alternatives to dominant autoregressive (AR) methods, with Masked Diffusion Models (MDMs) emerging as candidates. However, comparing AR (typically decoder-only) and MDM (often encoder-only) paradigms is confounded by differing architectures, obscuring true algorithmic and efficiency trade-offs. This research decouples these factors by evaluating MDMs within a decoder-only framework to: (1) Equitably compare MDM (as Any-Order AR) and standard AR paradigms through discrepancies on orders. (2) Investigate MDM architectural impacts on computational efficiency. We show decoder-only MDMs, despite a larger modeling space, can achieve significant inference speedups ($\sim25\times$) and comparable perplexity with techniques like temperature annealing, offering a path to reduced inference compute. This work provides insights for developing more computationally efficient foundation models by disentangling core modeling choices from architectural influences. Code is available at https://github.com/scxue/AO-GPT-MDM.

Comment: Decoder-only masked diffusion isolates generation-order effects from architectural effects.

Topic Match: The core study disentangles foundational sequence-model formulation and architecture, while reporting substantial inference-compute savings.

Relevance: 9 Novelty: 7

3. Multi-Head Self Attention is a Parameter Identification Mechanism

ArXiv ID: 2609.01231

Primary Topic: Architecture and Training Dynamics

Authors: W. Ross Morrow

Abstract: We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads ($1/2 \to 1/(2H)$), meaning models with more heads are structurally more identified. A subtle side effect of the mathematics observation that attention can never be fully identified. Similarly we also show that some bias terms can have no effect on softmax-based attention layers in both the single- and multiple-head settings, though this is mostly a curiosity that should have a marginal effect on model size and model training/prediction efficiency. We also touch on modern improvements to transformers including RoPE and GQA from this perspective, illustrating how those as well can improve the ratio of meaningful'' parameters to all parameters. Simple numerical examples demonstrate that training can indeed involve updates that overlap model-invariant subspaces that arise from a lack of identification. As part of our experiments we use arebalancing'' approach that can ``fix'' updates that overlap unindentified subspaces but do not try to present evidence this should actually be adopted. Instead we simply view our numerical results as exploring and confirming the theoretical results. As a whole we discuss a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance.

Comment: Relates attention head count to a 1/(2H) unidentified-parameter fraction and examines training updates along model-invariant directions.

Topic Match: Directly analyzes attention parameter symmetries and their optimization implications, providing mechanistic explanations for multi-head attention, RoPE, and GQA.

Relevance: 9 Novelty: 7

4. Flow Reasoning Models: Turning Flows Into Efficient Recurrent Reasoners

ArXiv ID: 2606.29150

Primary Topic: Architecture and Training Dynamics

Authors: Alec Helbling, Andrey Bryutkin, Mauro Martino, Duen Horng Chau, Nima Dehmamy, Hendrik Strobelt

Abstract: Structured reasoning requires making and revising interdependent decisions to reach a globally consistent solution. Existing architectures struggle with this: autoregressive models commit sequentially and cannot revise earlier decisions, while masked diffusion models often require careful decoding schemes to coordinate interdependent predictions. We introduce Flow Reasoning Models (FRMs), a novel framework for structured reasoning that adapts continuous flows over discrete structured outputs with a simple recurrent refinement mechanism. By self-conditioning a flow model on its own past outputs, we turn one-shot denoising into iterative solution refinement. This lets FRMs make and revise decisions in parallel, efficiently coordinating interdependent choices across solutions. Yet conventional self-conditioning becomes unreliable at greater recurrent depth due to exposure bias between one-step training predictions and recursively generated inference states. We address this mismatch with Fixed-Point Forcing (FPF), which trains FRMs on states produced by their own inference dynamics while preserving the standard flow-matching objective. FRMs achieve solve rates of $99.5\%$, $100.0\%$, and $99.9\%$ on Sudoku-Extreme, Zebra, and Maze-Unique, respectively. On Sudoku-Extreme, FRMs achieve higher peak accuracy than the evaluated masked-diffusion and specialized reasoning baselines while remaining highly compute-efficient, matching the next-best method's $98.7\%$ peak solve rate with $44\times$ fewer inference FLOPs.

Comment: Trains recurrent flow refinement on its own inference states to address exposure bias at greater recurrent depth.

Topic Match: Recurrent self-conditioning and Fixed-Point Forcing change the computational mechanism and training distribution needed to sustain iterative refinement.

Relevance: 8 Novelty: 8

5. FloydNet: A Learning Paradigm for Global Relational Reasoning

ArXiv ID: 2601.19094

Primary Topic: Architecture and Training Dynamics

Authors: Jingcheng Yu, Mingliang Zeng, Qiwei Ye

Abstract: Learning algorithmic computation often requires explicit relational intermediate states, yet many graph processors maintain their primary states on individual entities. We introduce \fnet and \textbf{Pivotal Attention} (PA), which maintain ordered pair states and update a target relation $(i,k)$ by attending over candidates formed from $(i,j)$ and $(j,k)$ for every pivot $j$. Motivated by the pair-and-pivot structure of Floyd--Warshall, PA learns relation composition and pivot weighting in parallel rather than executing its ordered min-plus recurrence. The \kfnet{k} framework extends this operation to ordered $k$-tuples, with Self-Attention and PA as its $k=1$ and $k=2$ cases at the attention-operation level. Under atomic tuple initialization and invariant readout, we show that \kfnet{k} is no more graph-discriminative than k-FWL; on BREC, each evaluated variant matches the success set of its corresponding WL reference. \fnet further achieves 96.64\% mean accuracy under the reported CLRS-30 protocol and a 99.8\% optimality rate with 10 samples on held-out non-metric TSP instances.

Comment: Introduces pair-state attention that composes relations through pivot nodes and extends attention to ordered tuples.

Topic Match: Pivotal Attention is a new architectural operation with an expressiveness analysis; its graph-focused validation makes the large-model connection narrower.

Relevance: 8 Novelty: 8

6. The Topological Trouble With Transformers

ArXiv ID: 2604.17121

Primary Topic: Architecture and Training Dynamics

Authors: Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne Liu

Abstract: Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.

Comment: Analyzes feedforward transformers' depth limitations for sequential state tracking and distinguishes depth-wise from temporal recurrence.

Topic Match: The central contribution examines architectural constraints and recurrent computation; its perspective and taxonomy support moderate novelty.

Relevance: 9 Novelty: 6

7. Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

ArXiv ID: 2609.00746

Primary Topic: Architecture and Training Dynamics

Authors: Minsik Choi, Geewook Kim, Young Geun Kim

Abstract: Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.

Comment: Connects attention-sink corruption and normalization design to capability loss during multimodal adaptation.

Topic Match: The core analysis concerns attention mechanisms and normalization-dependent behavior under training, with scope limited to vision-language adaptation.

Relevance: 8 Novelty: 7

8. Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs

ArXiv ID: 2609.01117

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Zhaoliang Chen, Jie Fu

Abstract: Chain-of-thought reasoning unfolds in discrete token space: each step is committed as text, errors propagate, and eliciting good traces presupposes traces to imitate. Reasoning instead in a model's continuous representation space - where intermediate states are vectors rather than words - sidesteps these constraints, but leaves open how those latent states should be computed. We approach this along two axes. First, we keep a large language model (LLM) frozen and use it for what it is already good at - modeling and decoding sequences - while a small auxiliary network supplies continuous latent thoughts as input. Second, we produce those latents by recurrence: a tiny recurrent reasoner refines them over many steps, decoupling the depth of computation from the size of the model, so that the latents are a product of iterative processing rather than a single forward pass. We instantiate this as Latent Recurrent Thoughts (LRT): a task-dedicated proposer supplies base latents, a recurrent reasoner refines them through bounded residual corrections, and the frozen LLM decodes the answer. On symbolic reasoning with answer supervision but no reasoning traces (Countdown-4, Sudoku) and on natural-language reasoning (HumanEval, MBPP, StrategyQA), LRT substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.

Comment: A trainable recurrent latent module decouples reasoning depth from the size of a frozen LLM decoder.

Topic Match: Recurrent modular computation is the central architectural mechanism, with reduced reasoning compute providing a secondary efficiency match.

Relevance: 8 Novelty: 7

9. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

ArXiv ID: 2603.18899

Primary Topic: Architecture and Training Dynamics

Authors: Steffen Dereich, Thang Do, Arnulf Jentzen

Abstract: The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Comment: Proves uniform Adam iterate bounds enabling unconditional error analysis for a class of strongly convex stochastic problems.

Topic Match: Adam boundedness and convergence directly concern optimizer dynamics, although the strongly convex assumptions limit applicability to large-model pretraining.

Relevance: 7 Novelty: 8

10. Denoising Diffusion Generative Models Secretly Calculate Attentions

ArXiv ID: 2609.00885

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Farzan Haddadi, Leila Monfared, Ebrahim Rezaii, Mohammadreza Malek-Mohammadi, Pejman Zakalvand, Narges Mokhtari

Abstract: Denoising diffusion models are the dominant architecture for image generation, whereas most natural language generation and modeling are primarily handled by well-known transformer architectures employing attention mechanism. Here, we show that diffusion models also inherently use an attention mechanism very similar to that of transformers. Therefore, attention emerges as a universal machine learning principle, based on a general training objective. We also show similarities in basic functional principle of auto-encoders and attention-based models. These equivalences allows us to interchange these designs based on practical requirements. As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation. Using this approach, a simplified algorithm is proposed for image generation which is based on attention mechanism. Results show that the attention-based implementation achieves comparable performance with significantly less effort and computational resources.

Comment: Proposes an attention-based reformulation of diffusion to reduce training and sampling computation.

Topic Match: The claimed diffusion–attention correspondence is architectural, with a secondary efficiency contribution; assumptions and quantitative savings remain unspecified.

Relevance: 8 Novelty: 6

11. Instella-MoE Technical Report

ArXiv ID: 2609.00791

Primary Topic: Architecture and Training Dynamics

Authors: Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum

Abstract: In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research.

Comment: Introduces Gated MLA and FarSkip-Collective connectivity as architectural mechanisms for efficient large-model pretraining.

Topic Match: Attention and connectivity design provide the concrete methodological match; the abstract does not establish a new MoE routing or balancing mechanism.

Relevance: 7 Novelty: 6

12. RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis

ArXiv ID: 2609.01729

Primary Topic: Architecture and Training Dynamics

Authors: Amirhosein Azarpour

Abstract: Kolmogorov--Arnold Networks (KANs) replace the fixed scalar weights of a standard network with learnable univariate functions on each edge, but existing variants still fix the \emph{basis} that those functions are built from: B-splines, Chebyshev polynomials, wavelets, or Jacobi polynomials, and learn only the combination weights over it. We introduce RecKAN, which instead defines the basis itself by a second order polynomial recurrence, $R_{n+1}(x) = (ax^2+bx+c)R_n(x) + (dx+e)R_{n-1}(x)$, whose five coefficients are learned jointly with the network. We show this recurrence recovers several classical polynomial families including both kinds of Chebyshev polynomials, Fibonacci, Pell, and Jacobsthal polynomials as special cases, and prove that its degree grows linearly in $n$ exactly on the sub-family containing all of them, giving a concrete sense in which the learned basis can move beyond any fixed classical choice. Across multiple benchmark datasets spanning image, text, biomedical time series classification, and time series forecasting, RecKAN outperforms three parameter-matched KAN baselines (Chebyshev, Jacobi, and spline based) on all classification tasks and achieves the lowest MSE on the ETTh1 forecasting benchmark. Additionally, when used as a classifier head with a convolutional backbone, RecKAN achieves higher accuracy than standard MLP heads on Fashion MNIST, CIFAR-10, and SVHN. On a synthetic function fitting benchmark it tracks a sharply oscillatory target that a parameter comparable MLP under fits. We further show that the learned recurrence coefficients are interpretable: on the task requiring the most local structure, training moves the basis away from the linear degree growth regime that contains every classical family we identify, consistent with our theoretical analysis of what that structural shift enables.

Comment: Makes KAN basis functions learnable through a parameterized polynomial recurrence.

Topic Match: Learning the basis itself is a core architectural mechanism, although evidence for its value in large-model training is limited.

Relevance: 7 Novelty: 6

13. From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion

ArXiv ID: 2609.01043

Primary Topic: Architecture and Training Dynamics

Authors: Satoshi Hayakawa

Abstract: Uniform-state discrete diffusion models update all tokens in parallel while keeping every position revisable. Even when the commonly used top-$p$ rule leaves only one candidate at a position, that choice affects only the current reverse step and can be revised at the next sampling step. We ask what changes when selected hypotheses instead become persistent context for later predictions. We therefore propose committed reveal sampling (CRS), a training-free sampler that stores selected argmax tokens and inserts them into subsequent model inputs. Our analysis gives a rationale for selecting later and for keeping selected tokens visible. Under the exact forward process, the Bayes error of selecting a clean token cannot increase as noise decreases, while in a simple latent-mode model, keeping the selected token visible helps later parallel predictions agree on the same sequence-level choice. Empirically, paired experiments on Duo-distilled then separate this persistent effect from single-step top-$p$ restriction and scalar temperature scaling. Under the same finalization rule, CRS without top-$p$ truncation reaches lower generative perplexity (GenPPL) than fixed $p=0.95$ and $p=0.9$ baselines across budgets of 8--64 function evaluations (NFE). At 64 NFE, the comparison at matched unigram entropy also gives lower GenPPL for CRS, yielding a more favorable GenPPL--entropy tradeoff. Base Duo shows the same direction in a descriptive comparison, while other diversity and continuation metrics can rank these operating points differently. These results identify support restriction and persistent context as distinct controls of that tradeoff.

Comment: Uses persistent token commitments to coordinate later parallel predictions in uniform discrete diffusion.

Topic Match: The contribution is a computational mechanism governing diffusion updates, supported by analysis of commitment timing and sequence consistency; its scope is sampling behavior.

Relevance: 7 Novelty: 6

14. A Study of Hidden-State Optimization Order in Predictive Coding Networks

ArXiv ID: 2609.00686

Primary Topic: Architecture and Training Dynamics

Authors: Xueyuan Li, Danilo Vasconcellos Vargas

Abstract: Local learning methods offer an alternative to end-to-end backpropagation, but their unstructured local objectives can produce weak feature learning in deep networks. We study whether the order of hidden-state optimization can address this limitation. We propose a boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk. We instantiate this schedule in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction errors are explicitly exposed during inference. On CIFAR-10, the resulting boundary-first predictive-coding instantiation improves accuracy over standard predictive coding by $9.77\%$ under a standard parametrization and by $5.51\%$ under a $μ$-parametrization. Diagnostic analyses further show more non-trivial early-layer updates, lower initial-to-final CKA, and more diverse layerwise gradients, consistent with stronger feature learning. These results support boundary-first, chunk-based inference as a practical design principle for predictive-coding training and motivate its study in broader local-learning systems.

Comment: Boundary-first hidden-state optimization changes layerwise learning dynamics in predictive-coding networks.

Topic Match: The inference schedule introduces a training mechanism, with evidence currently limited to small image-classification networks.

Relevance: 7 Novelty: 6

15. FLaG: Frequency-Domain Latent-attention Gated Pooling for Token Aggregation

ArXiv ID: 2609.00831

Primary Topic: Architecture and Training Dynamics

Authors: Kewei Li, Rongying Zhang, Xueli Wang, Xiwen Gong, Zhongjian Wang, Qiuchen Zhao, Lan Huang, Ruochi Zhang, Fengfeng Zhou

Abstract: Token aggregation converts token-level representations into fixed-dimensional sample representations, but most pooling methods operate only in the original token space. We introduce Frequency-Domain Latent-attention Gated Pooling (FLaG), a plug-in aggregation module that re-expresses encoder outputs in the Fourier domain before final pooling. FLaG represents the nonredundant rFFT spectrum through concatenated real and imaginary components, summarizes spectral tokens with learnable latent queries, derives a sample-conditioned channel gate, and reconstructs modulated token representations for downstream aggregation. We evaluate the same architecture across ESM2-based antimicrobial peptide (AMP) activity prediction, ResNet18 image classification on CIFAR-10 and CIFAR-100, and three RoBERTa-based language tasks. FLaG achieves the best macro-averaged Spearman correlation coefficient, RMSE, and Recall@50 across four AMP backbone-species settings and the highest top-1 accuracy on CIFAR 10. It also achieves the best mean results on five of seven language metrics, although mean pooling remains strongest on STSBenchmark. AMP-side mechanistic analyses reveal low-frequency prediction sensitivity across most encoder layers, with increased relative high-frequency sensitivity in the final layer, and pronounced peptide-specific positional responses. The residual gate broadly amplifies spectral channels while preserving the low-frequency-dominated energy profile, whereas latent cross-attention exhibits sample- and species-specific spectral allocation. Overall, FLaG provides a transferable frequency-domain aggregation bias across protein, visual, and textual representations, with benefits that depend on the backbone and downstream task. Supplementary materials, source code, and data are available at https://www.healthinformaticslab.org/supp/ and https://github.com/Kewei2023/AMPCliff/tree/FLaG.

Comment: Fourier-domain latent attention and channel gating introduce a reusable token-aggregation mechanism.

Topic Match: The new pooling module is the core architectural contribution, although its evidence concerns downstream encoder tasks and offers limited large-scale training insight.

Relevance: 7 Novelty: 6

16. When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection

ArXiv ID: 2605.26872

Primary Topic: Architecture and Training Dynamics

Authors: Zhengyu Hu, Zheyuan Xiao, Linxin Song, Fengqing Jiang, Yuetai Li, Zhihan Xiong, Yue Liu, Junhao Lin, Yao Su, Lijie Hu, Kaize Ding, Teng Xiao, Radha Poovendran

Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.

Comment: A gradient-derived, forward-only learning-cost proxy selects teacher answers matched to each student.

Topic Match: Student-dependent learning signals connect it to training dynamics; its main contribution is supervision selection for distillation and post-training.

Relevance: 6 Novelty: 7

17. Wave Function Backpropagation with Explicit Temporal-Interval Dynamics

ArXiv ID: 2609.00503

Primary Topic: Architecture and Training Dynamics

Authors: Byunggu Yu, Justin Kim

Abstract: Conventional neural networks learn predominantly through affine transformations followed by nonlinear activations, while elapsed time is often treated as an auxiliary feature or assumed to be uniformly sampled. This paper introduces Wave Function Backpropagation (WFB), a wave-parameterized learning formulation in which neural responses are represented by learnable amplitude, wavenumber, angular frequency, and phase. The formulation associates an observed state with its temporal interval Delta t through the phase of a differentiable spatiotemporal wave. We derive standard WFB gradients and a spatial-curvature correction based on the Laplacian of the wave response. WFB is instantiated in a deliberately feed-forward trajectory predictor to provide a controlled proof of concept; sequence learning is outside the scope of the present evaluation. With motion features, STD-WFB using real intervals reduces average displacement error (ADE) by 20.4% relative to the original FFN baseline. In a new position-only evaluation that removes temporal leakage through precomputed velocity and acceleration, real-interval WFB reduces ADE by 10.4% relative to the original FFN and remains competitive with parameter-matched ReLU controls, obtaining 2.1% lower mean ADE than the matched FFN with explicit Delta t. Shuffled-interval WFB attains the lowest mean ADE, indicating that the present evidence supports the effectiveness of the wave representation but does not attribute the gain to interval alignment. These results establish WFB as a viable structured feed-forward learning formulation and define a clear basis for subsequent architectural studies.

Comment: Wave-parameterized neurons introduce learnable spatiotemporal phases and a spatial-curvature gradient correction.

Topic Match: The neuron parameterization and gradient formulation are architectural contributions, although the small trajectory-prediction demonstration establishes no large-model training benefit.

Relevance: 6 Novelty: 6

18. Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

ArXiv ID: 2510.25060

Primary Topic: Architecture and Training Dynamics

Authors: Jingzhou Liu

Abstract: In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter $α$ varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative $α$ and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime $α$ in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.

Comment: Symmetry-based bifurcation analysis links the leaky-ReLU slope to optimization critical-point stability.

Topic Match: Optimization-landscape stability directly overlaps training dynamics, but the shallow Gaussian-input setting limits the conclusions available for large-model training.

Relevance: 6 Novelty: 6

Efficiency, Compression, and Large-Scale Training (20)

1. Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

ArXiv ID: 2609.00762

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Wentao Ye, Zhanming Shen, Zhiqing Xiao, Yao Ding, Haobo Wang, Gang Chen

Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an $r\times r$ core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched $r^2$ budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.

Comment: Task-informed, Fisher-whitened bases restrict adaptation to an r-by-r trainable core, sharply reducing optimizer state.

Topic Match: The core contribution is a low-rank adaptation mechanism that preserves quality with substantially fewer trainable parameters and less optimizer-state memory.

Relevance: 9 Novelty: 7

2. CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing

ArXiv ID: 2609.01925

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt, Ryan A. Rossi, Linh Ngo Van, Jieyang Chen, Thien Huu Nguyen

Abstract: The attention prefilling phase of long-context LLM inference scales quadratically, making self-attention a severe computational bottleneck. Traditional sparse attention methods mitigate this through fixed patterns or offline profiling, but lack the flexibility to adapt to input-dependent attention structure. Recent dynamic methods address this by routing heads to sparse patterns in real-time, but rely on indirect routing proxies with overhead and budget allocation mechanisms that overlook the post-softmax mass hierarchy. We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm. First, we show that the routing decision can be read directly off the structure of the proxy attention map. We replace the Jensen-Shannon Divergence (JSD) routing with C_struct, a structural proxy that measures mass at Vertical-Slash compatible positions and reproduces JSD's routing decisions while eliminating both the pooled matmul and subsequent KL divergence overhead. Second, we formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts. CRISP navigates this via a sink-aware threshold grounded in the noise floor. Empirically, across InfiniteBench, RULER and LongBench on two model families, CRISP is the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy benchmarks, recovering up to +28.0 pp on retrieval tasks over baselines and achieving up to a 5.30x attention speedup at 512k tokens, driven primarily by our O(n) noise elimination during selection while preserving structural integrity.

Comment: Noise-floor-aware sparse attention avoids accumulating background mass when selecting long-context attention entries.

Topic Match: New sparse-prefill routing and selection mechanisms directly reduce attention computation, with reported attention speedups up to 5.30x.

Relevance: 9 Novelty: 7

3. IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models

ArXiv ID: 2604.00757

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Dong-Jae Lee, Sunghyun Baek, Junmo Kim

Abstract: Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.

Comment: Derives token-pruning scores from attention's implicit weight matrix, accounting for token contribution and redundancy.

Topic Match: The core contribution is a mechanistically motivated pruning method that reduces visual-token computation in large vision-language models.

Relevance: 9 Novelty: 7

4. The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally

ArXiv ID: 2609.01587

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jundong Hu, Shekar Ramachandran

Abstract: Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.

Comment: Finds finer global quantization granularity outperforms selective layer precision at matched budgets in the tested compatible LLMs.

Topic Match: Causal precision-restoration experiments produce an actionable insight about allocating the precision budget for LLM compression.

Relevance: 9 Novelty: 7

5. CacheBridge: Efficient Cross-Model KV Cache Transfer

ArXiv ID: 2609.00891

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin

Abstract: Sharing context between LLMs in a multi-model system requires the receiving model to prefill the shared prefix because KV caches are model-specific. Recent closed-form cross-model KV transfer, hereafter Full-Head Mapping, avoids this replay by fitting a training-free affine mapper from source to target caches. However, its full-head design maps each target KV head from every source KV head in the selected layers, making transfer quality sensitive to architectural differences and causing mapper storage and application cost to grow with layer support. To this end, we introduce CacheBridge, which co-designs architecture-indexed mapper support, attention-aligned calibration, and bounded mapper construction while retaining a closed-form affine interface for online deployment. CacheBridge restricts each target head to a matched source head, weights reconstruction errors by causal attention sensitivity, and uses a fused GPU kernel to construct weighted sufficient statistics without materializing full observation tensors. Across three transfer directions, CacheBridge recovers the two Ministral 3 transfer directions where Full-Head Mapping loses substantial accuracy while preserving 99.83\% mean target retention on Qwen3. On Qwen3 $14\mathrm{B}\to32\mathrm{B}$, it reduces mapper storage by $8\times$, accelerates application by up to $3.0\times$, matches \fullhead with one tenth of the calibration data, and reduces 500-sequence construction from 92.63 to 8.63 seconds ($10.7\times$).

Comment: Reduces cross-model KV-transfer storage and computation through head-matched, attention-weighted affine mappers and fused calibration kernels.

Topic Match: The core contribution directly improves large-model cache reuse, mapper memory, and transfer computation through a new mapping and calibration method.

Relevance: 9 Novelty: 7

6. Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

ArXiv ID: 2608.22322

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Yan Wang

Abstract: Optimizer-state quantization is commonly designed for Adam's dense, parameter-aligned first- and second-moment arrays. This abstraction breaks for memory-efficient optimizers, whose states may be factored, confidence-modulated, or maintained in a projected space, so similar reconstruction error can produce different update error. We formulate optimizer-state quantization as a joint problem over representation, topology, and update semantics. We then introduce Adaptive Log-Space (AL) quantization for non-negative states. AL fits each block's observed nonzero logarithmic interval and reserves a separate code for exact zero, enforcing $q = 0 \Leftrightarrow x = 0$; signed momentum and state precision remain independently selectable. Controlled probes show that adaptive ranges reduce update error and temporal drift, exact-zero reservation preserves dormant states, and state topology constrains useful block granularity. End-to-end language-model training evaluates the resulting policy across dense, factored, confidence, and projected optimizer states. On TinyLlama-1.1B, AL8 with uniform 8-bit momentum reaches 72.90 perplexity versus 73.54 for bitsandbytes 8-bit AdamW, with comparable optimizer-state storage and higher throughput. CAME matches reference-level final perplexity across three seeds when its non-negative states use AL16, while a semantic grouping-and-protection policy closes most of quantized Adafactor's 100K-step late-loss gap. These results make state topology and update semantics first-class design constraints for optimizer quantization.

Comment: Quantizes structured optimizer states using adaptive logarithmic ranges and exact-zero codes to reduce training memory while preserving updates.

Topic Match: Optimizer-state compression is the central mechanism, with direct consequences for training memory, throughput, and optimizer stability.

Relevance: 9 Novelty: 7

7. Stream-CQSA: Exact Out-of-Memory Recovery for Attention

ArXiv ID: 2604.20819

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Yiming Bian, Joshua M. Akey

Abstract: Long-context large language models are limited not only by attention cost but also by out-of-memory (OOM) failures. A selected attention call may not fit in available device memory even when the kernel is optimized. Exact and approximate attention methods reduce memory use, but every fixed implementation still has a device-specific capacity boundary. We introduce Stream-CQSA, an attention-level OOM recovery framework based on CQS decomposition, derived from the theory of cyclic quorum sets (CQS). Stream-CQSA recursively partitions an infeasible attention call into independent subsequence tasks, executes each with a compatible inner kernel, and recomposes the local statistics to recover the full attention output. This recovery is exact relative to the wrapped attention kernel, whether that kernel is exact or approximate. Compared with FlashAttention-2, the major baseline, our native Stream-CQSA kernel improves 16-bit forward-output error relative to a dense float64 reference and matches 16-bit backward-gradient error where FlashAttention-2 fits in the GPU memory. At the longest feasible baseline length, it costs $1.5$--$1.9\times$ the forward runtime and $2.1$--$2.4\times$ the forward--backward runtime. Beyond that sequence length boundary, our method continues to return an output while FlashAttention-2 OOMs. Stream-CQSA is therefore not a faster attention method. Instead, it converts memory-capacity failure into a recoverable execution path by trading extra compute, host-device transfer, and recomposition for completion.

Comment: Recursive attention decomposition recovers from OOM while preserving the wrapped kernel's output semantics and supporting backward computation.

Topic Match: Memory-capacity recovery is central; kernel scheduling and host-device transfers also enable training beyond the baseline attention memory boundary.

Relevance: 9 Novelty: 7

8. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

ArXiv ID: 2606.07710

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Young D. Kwon, Miles Williams, Rui Li, Alexandros Kouris, Stylianos I. Venieris

Abstract: The autoregressive nature of large language models (LLMs) remains a significant bottleneck for inference, particularly in complex agentic workloads. While speculative decoding (SD) accelerates inference, current approaches rely on static drafting paradigms, utilising either autoregressive drafting models for reasoning or diffusion-based parallel drafting models for structured outputs. We empirically find that drafting accuracy fluctuates dramatically within a single sequence, leaving significant performance unrealised by static paradigms and coarse-grained routing. To address this volatility, we introduce WhiFlash, the first cross-paradigm SD method that unifies autoregressive and diffusion-based parallel drafting under a single token-level controller. WhiFlash adopts a fine-grained routing mechanism that employs either a lightweight entropy-based or a learned neural policy, both parametrised to provide a tunable balance between expected token gain and latency. To make high-frequency switching computationally viable, we introduce novel cache-management optimisations, Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. By capitalising on the complementary strengths of fundamentally distinct drafting architectures, WhiFlash achieves significantly higher acceptance lengths, yielding category-specific throughput gains of up to 69.6% over the state-of-the-art autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.

Comment: Token-level switching between autoregressive and diffusion drafts reduces speculative-decoding cost.

Topic Match: The contribution combines a new draft-selection mechanism with cache-management methods that directly improve LLM inference throughput.

Relevance: 8 Novelty: 8

9. LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

ArXiv ID: 2608.19800

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Haonan He, Xinyue Fan

Abstract: Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.

Comment: Aligns LoRA initialization and rank allocation with multi-step gradients without additional GPU memory.

Topic Match: The core contribution modifies low-rank adaptation while preserving its training-memory advantage; multi-step probing is an incremental methodological extension.

Relevance: 9 Novelty: 6

10. CAT-Flow: Curvature-Adaptive sTeps for Flow Matching

ArXiv ID: 2609.01746

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Qinchan Li, Pedro Cisneros-Velarde, Keru Fu, Samuel Antunes Miranda, Sharan Vaswani, Hao Zhang

Abstract: Flow Matching has emerged as a leading framework for generative modeling, powering state-of-the-art systems such as FLUX and Stable Diffusion 3.5. However, the iterative nature of its ODE-based sampling process creates a fundamental efficiency bottleneck: the quality of generated samples is highly sensitive to the choice of step-sizes, and current models typically require 20 to 30 steps for good quality. In this work, we propose two lightweight, training-free algorithms, CAT-OV and CAT-OT that adapt step-sizes at inference time based on a novel connection between Flow Matching sampling and gradient flow. Our algorithms are computed efficiently by not requiring additional neural function evaluations. Specifically, CAT-OT estimates curvature over time via a finite-difference approximation of the time-derivative of the vector field, while CAT-OV approximates curvature over the state space via a gradient of the vector field. Under suitable conditions, both methods have truncation error bounds of constant order. Empirically, CAT-OV and CAT-OT outperform existing step-size heuristics in image quality metrics across four text- to-image Flow Matching models, reducing the number of generation steps required to reach comparable quality by up to 40%.

Comment: Adapts flow-matching sampling step sizes using curvature estimates without additional neural function evaluations.

Topic Match: The core contribution is a sampling algorithm that reduces neural evaluations required for large generative models.

Relevance: 8 Novelty: 7

11. HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

ArXiv ID: 2609.01730

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos, Giuseppe Ateniese, Emanuele RodolÃ

Abstract: Fully homomorphic encryption (FHE) allows a server to run a language model directly on encrypted user prompts, but current approaches remain prohibitively slow. Ciphertexts natively support only addition, multiplication, and rotation, and multiplications may be composed only to a bounded depth before a costly bootstrapping operation is needed to continue. Every nonlinearity must therefore be approximated by an iterative method, and each iteration uses multiplications. A higher iteration count buys precision but exhausts the available depth faster and triggers more bootstraps, which dominate latency. Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance. We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonlinearity iteration counts learnable, enabling them and the model weights to co-adapt during training. HEAT optimizes iterations with respect to the task objective, allowing the model to adapt to approximation errors encountered during inference without architectural changes or retraining from scratch. On encrypted GPT-2 decoding, HEAT reduces iterations by $3.1\times$, bootstraps by $1.6\times$, and end-to-end latency by $1.4\times$, while improving decode agreement over the calibrated baseline.

Comment: Jointly learns nonlinearity approximation budgets and model weights to reduce expensive FHE bootstrapping.

Topic Match: The training method directly reduces encrypted language-model inference cost through adaptive approximation budgets, with a specialized FHE scope.

Relevance: 8 Novelty: 7

12. mzCache: On-Device LLM Memory Management under Multitasking

ArXiv ID: 2609.01338

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hongseung Yu, Minsung Kim, Jongseok Park, Kyunghan Lee

Abstract: On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5$\times$ reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.

Comment: Fine-grained weight and KV-cache buffers overlap CPU memory restoration with GPU inference under memory pressure.

Topic Match: Restoration-oriented buffer management and eviction policies introduce a substantive LLM memory-efficiency mechanism, demonstrated on mobile inference.

Relevance: 8 Novelty: 7

13. MineDraft: A Framework for Batch Parallel Speculative Decoding

ArXiv ID: 2603.18016

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhenwei Tang, Arun Verma, Zijian Zhou, Zhaoxuan Wu, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low

Abstract: Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Comment: Overlaps speculative drafting and verification across two request batches to hide drafting latency.

Topic Match: The batch-parallel execution schedule directly improves LLM inference efficiency through a concrete scheduling mechanism.

Relevance: 8 Novelty: 6

14. KV Cache Offloading for Context-Intensive Tasks

ArXiv ID: 2604.08426

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev, Vyacheslav Zhdanovskiy, Yegor Yershov

Abstract: With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.

Comment: Identifies key-projection and landmark failures in KV-cache offloading and proposes a more accurate alternative strategy.

Topic Match: The proposed offloading alternative directly addresses KV-cache memory efficiency and accuracy, providing a methodological contribution beyond the context-intensive benchmark.

Relevance: 8 Novelty: 6

15. SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

ArXiv ID: 2609.01004

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Shiyu Li, Zi-Yuan Hu, Shijia Huang, Yanyang Li, Yiwu Zhong, Liwei Wang

Abstract: Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.

Comment: Filters redundant high-norm visual tokens before text-guided pruning to reduce multimodal LLM inference computation.

Topic Match: The core contribution is a token-pruning mechanism that directly reduces large-model inference cost, with a narrower focus on visual inputs.

Relevance: 8 Novelty: 6

16. DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models

ArXiv ID: 2606.23181

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jungseob Lee, Seongtae Hong, Seungjun Lee, Jaehyung Seo, Junyoung Son, Sugyeong Eo, Chanjun Park, Hyeongju Park, Hyeonseok Moon, Heuiseok Lim

Abstract: Hybrid reasoning models can answer directly or spend extra tokens on extended thinking. A practical router should choose between these modes for each query, so easy problems avoid unnecessary reasoning and hard problems receive enough budget to finish the answer. Existing routers move in this direction, but they typically require labeled training data or fix thinking budgets up front, ignoring answer-level evidence from the model itself. We introduce DART, a training-free routing framework that samples two cheap no-think drafts, accepts direct answering when the drafts agree, and predicts a thinking budget from draft entropy when they disagree. Across the main comparisons, DART preserves or improves always-thinking accuracy in most settings while reducing thinking-token use. Accuracy improves by up to +9.0 points on Olympiad-level math and by up to +22.5 points on code under execution-based equivalence, while thinking-token use drops by 32-73%. The Stage~1 signal extends across model scales (0.6B--32B), model families, and API-only hosted settings, with no labeled data and no gradient updates required. Our code is available at https://github.com/js-lee-AI/DART.

Comment: Draft agreement and entropy adapt thinking budgets to reduce reasoning-token consumption.

Topic Match: An adaptive compute-allocation rule directly reduces large-model inference cost; its relevance is narrower because it operates at the reasoning-budget level.

Relevance: 7 Novelty: 6

17. Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

ArXiv ID: 2609.00632

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Lei Wang, Jieming Bian, Letian Zhang, Jie Xu

Abstract: Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

Comment: Shared adaptation directions and personalized rank-wise scales support LoRA training under heterogeneous rank budgets.

Topic Match: The method changes low-rank adaptation parameterization and aggregation, with a narrower focus on federated personalization.

Relevance: 7 Novelty: 6

18. Relational Representation Distillation

ArXiv ID: 2407.12073

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Nikos Giakoumoglou, Tania Stathaki

Abstract: Knowledge distillation transfers knowledge from large teacher models to more compact student networks. The standard approach minimizes the Kullback-Leibler (KL) divergence between the probabilistic outputs of the teacher and student, aligning predictions but neglecting the structural relationships encoded within the teacher's internal representations. Recent advances have adopted contrastive learning objectives to address this limitation; however, such instance-discrimination-based methods induce a "class collision problem", in which semantically related samples are inappropriately pushed apart despite belonging to similar classes. To overcome this, we propose Relational Representation Distillation (RRD) that preserves the relative relationships among instances rather than enforcing absolute separation. Our method introduces separate temperature parameters for teacher and student distributions, with a sharper teacher (low $τ_t$) emphasizing primary relationships and a softer student (high $τ_s$) maintaining secondary similarities. This dual-temperature formulation creates an implicit information bottleneck that preserves fine-grained relational structure while avoiding the over-separation characteristic of contrastive losses. We establish theoretical connections showing that InfoNCE emerges as a limiting case of our objective when $τ_t \rightarrow 0$, and empirically demonstrate that this relaxed formulation yields superior relational alignment and generalization across classification and detection tasks.

Comment: Preserves teacher-student relational distributions using separate temperatures to avoid contrastive class collisions during distillation.

Topic Match: The core contribution refines teacher-to-student compression through a new distillation objective, with evidence limited to classification and detection models.

Relevance: 7 Novelty: 6

19. TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

ArXiv ID: 2608.30811

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Daniel Agyei Asante, Yang Li

Abstract: Long-context compression is essential for reducing the cost and latency of large language model inference. However, existing methods can fragment important evidence, require additional training or alignment, and often depend on the target model for effective compression. We introduce TopoCompress, a training-free and model-agnostic framework that compresses long contexts by selecting coherent semantic spans. TopoCompress first scores each span using dense and lexical query relevance together with semantic acceleration. It then constructs a hybrid graph that connects spans based on semantic similarity and sequential adjacency, and propagates the query-guided relevance scores over the graph. Across five long-context tasks-HotpotQA, 2WikiMQA, MuSiQue, Qasper, and MultiFieldQA-en-TopoCompress consistently outperforms strong compression baselines. Notably, TopoCompress achieves performance comparable to the strongest baseline while using a 4x smaller compression budget, and provides a 1.41x smaller compression time over the fastest baseline.

Comment: Compresses prompts through query-guided relevance propagation over graphs connecting semantically related and sequentially adjacent spans.

Topic Match: The central contribution is a context-compression mechanism that reduces LLM input cost; its inference focus makes it peripheral to the training-centered feed.

Relevance: 7 Novelty: 6

20. Control Variate Score Matching for Diffusion Models

ArXiv ID: 2512.20003

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Khaled Kahouli, Romuald Elie, Klaus-Robert Müller, Quentin Berthet, Oliver T. Unke, Arnaud Doucet

Abstract: Sampling from unnormalized probability densities is a pervasive challenge across the computational and physical sciences. Diffusion models provide a powerful generative framework for this task, but their success relies on accurately estimating the score of the perturbed target distribution. Current approaches face a dichotomy between two standard estimation methods: the Denoising Score Identity (DSI) requires data samples and exhibits high variance at low noise levels, whereas the Target Score Identity (TSI) relies on the energy function and suffers from diverging variance at high noise levels. In this work, we reconcile both approaches by introducing the Control Variate Score Identity (CVSI), an unbiased estimator with an analytically optimal, state- and time-dependent control coefficient that theoretically minimizes variance over the entire diffusion process. CVSI serves as a robust plug-in estimator that significantly enhances performance and efficiency in data-free sampler learning and training-free diffusion sampling. These gains scale to complex, high-dimensional energy-based models.

Comment: An analytically optimal control variate reduces diffusion-score variance to improve sampler learning and sampling efficiency.

Topic Match: The closest match is computational efficiency through lower-variance score estimation, with evidence focused on density samplers and an indirect connection to large-model training.

Relevance: 6 Novelty: 7

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains