This is a remedial run for missed papers from 07/31/2026 to 08/02/2026.

Results generated on 09/13/2026.

Personalized Daily ArXiv Papers 2026-08-03

Model	Metric	Usage			Papers
Model	Metric	Prompt	Completion	Total	Total arXiv	Scanned	Relevant
`local agent CLI`	Tokens	not reported	not reported	not reported	1019	1019	67
`local agent CLI`	Cost	not reported	not reported	not reported	1019	1019	67

_{Token counts are not reported for this run. 10 of 19 model calls succeeded, 9,191s of model wall clock.}

Topic Coverage:

Topic	Papers
Large-Scale Training Systems and Efficiency	11
Architecture and Training Dynamics	18
Efficiency, Compression, and Large-Scale Training	38

Table of contents by topic:

Large-Scale Training Systems and Efficiency (11)

Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback Authors: Maria Smirnova, Alexey Kravatskiy
Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks Authors: Shenxi Wu, Haosong Zhang, Xingjian Ma, Shirui Bian, Yichi Zhang, Xi Chen, Wei Lin
DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training Authors: Hanlin Du, Zhiyuan Yan, Yungang Bao, Sa wang
Paris: A Decentralized Trained Open-Weight Diffusion Model Authors: Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy
The Convergence Behavior of Adam under Heavy-Tailed Noise Authors: Yijiang Pang
Heterogeneous Decentralized Diffusion Models Authors: Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy
Reassessing Muon for Matrix Factorization Authors: Ali Parviz, Gal Mishne, Alex Cloninger
Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning Authors: Wenhao Zhang, Yibo Xie, Rui Wang, Jiahua Yang, Lei Jiang, Zibo Yang, Yawei Wang, Jiali Xu, jasperawang, Haoyang Long, Huan Xiong, alantzhao
Communication-Efficient Secure Aggregation in Decentralized Learning Authors: Sayan Biswas, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma, Milos Vujasinovic
When Does Muon Help Agentic Reinforcement Learning? Authors: Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun
Learned Digital Over-the-Air Computing for Federated Edge Learning Authors: Antonio Tarizzo, Mohammad Kazemi, Deniz GÃ¼ndÃ¼z

Architecture and Training Dynamics (18)

Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion Authors: Zeyu Liu, Jinhao Zhang, Yunquan Zhang, Guangming Tan, Xiang Gao, Fangming Liu, Daning Cheng
AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction Authors: Ziqiang Cui, Han Shi, Bowei He, Yu Pan, Peiyang Liu, Shengyin Sun, Yankai Chen, Haoli Bai, Yichun Yin, Xue Liu, Chen Ma
Structured Recurrent Mixers for Massively Parallelized Sequence Generation Authors: Benjamin L. Badger
Mask-Based Priors Are More Persistent than Query-Key Initializations Authors: Mingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth Authors: Kehan Wang
Entropy-Gated Latent Recursion Authors: Soham Bhattacharjee, Dushyant Singh Chauhan, Salem Lahlou, Martin Takac, Nils Lukas
Eigenvalues as a Metric for Memory Dynamics in Sequence Models Authors: Rahel Rickenbach, Jelena Trisovic, Alexandre Didier, Jerome Sieber, Melanie N. Zeilinger
Statistical Mechanics of Learning on Product Wasserstein Manifolds Authors: Srinivasa Rao P Vangmayi P Reddy
One-Sided Quantile Coupling for Flow Matching Authors: Jin-Young Kim, So-Yoon Cho, Hyun-Gyoon Kim
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning Authors: Xujun Che, Yuchen Yuan, Weida Zhao, Chenyang Yu
DiffusionGemma Technical Report Authors: DiffusionGemma Team, Adrien Ali TaÃ¯ga, James Assiene, Daniele Calandriello, Rahma Chaabouni, JoÃ£o Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, Ofir Nabati, JoÃ£o Gabriel Oliveira, Nicolas Perez-Nieves, Nastasia Prutianova, Bobak Shahriari, Jean Tarbouriech, Pavel Tyletski, ÃaÄlar ÃnlÃ¼, Cindy Wu, Glenn Cameron, Jerome Connor, Sertan Girgin, Maarten Grootendorst, Alon Levkovitch, Eliya Nachmani, Omar Sanseviero, Piotr Stanczyk, Quentin Berthet, Andrew Campbell, ClÃ©ment Crepy, Valentin De Bortoli, Arnaud Doucet, Romuald Elie, Alexandre Galashov, Klaus Greff, Alexis Jacq, David Ruhe, Yu-Han Wu, Sebastian Flennerhag, Brendan O'Donoghue, George Scrivener, Shantanu Thakoor
Human-like working memory signatures emerge from intrinsically plastic artificial neurons for robust dynamic vision Authors: Jingli Liu, Huannan Zheng, Bohao Zou, Kezhou Yang
The Fourth Quadrant: A Stylized View of Benign Misfitting Authors: Gireeja Ranade, Anant Sahai
Plasticity of Growing and Elastic Neural Networks in Online Continual Learning Authors: Jeong Min Kong, Richard S. Sutton
Beyond Feature and Structure Alignment: Learning Transferable Propagation Knowledge for Graph Foundation Models Authors: Yi Wang, Jitao Zhao, Di Jin, Dongxiao He
Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens Authors: Yi Luo, Rongzhi Gu, Jixun Yao
SAF-OPD: Stable Advantage Fusion for On-Policy Distillation Authors: Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen
When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer Authors: Yuyang Shen

Efficiency, Compression, and Large-Scale Training (38)

LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs Authors: Yirui Liu, Ruoling Qi, Longwen Wang, Xuaner Wu, Jian Chen, Yuxin Jin, Jiawei Shao, Xuelong Li
A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models Authors: Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang, Pan Zhang, Shi-Ju Ran, Gang Su
Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization Authors: Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction Authors: Changwoo Baek, Seungjun Shin, Kyeongbo Kong
Stem: Rethinking Causal Information Flow in Sparse Attention Authors: Lin Niu, Xin Luo, Linchuan Xie, Yifu Sun, Guanghua Yu, Jianchen Zhu, S Kevin Zhou
A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models Authors: Wenjun Wang, Shuo Cai, Congkai Xie, Mingfa Feng, Yiming Zhang, Zhen Li, Kejing Yang, Ming Li, Jiannong Cao, Hongxia Yang
Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling Authors: Rongman Xu, Yifei Li, Tianzhe Zhao, Yanrui Wu, Bo Li, Hang Yan
Just on Time: Token-Level Early Stopping for Diffusion Language Models Authors: Zakhar Kohut, Severyn Shykula, Mykola Vysotskyi, Serhii Dmytryshyn, Dmytro Khamula, Michal Zakrzewski, Damian Rynczak, Jacek MaÅecki, Taras Rumezhak, Volodymyr Karpiv
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding Authors: Aofan Liu, Jingxiang Meng, Fangxin Liu, Yongbiao Chen
Understanding Rate-Distortion Performance in Distributed Transformer Inference Authors: Anderson de Andrade, Alon Harell, Ivan V. BajiÄ
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization Authors: Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra, Ankur Mali
OnlineCache: Learning Dynamic Caching Policies with Error Correction for Efficient Diffusion Inference Authors: Zhikang Xie, Xichen Ye, Yifan Wu, Haoshen Yu, Li chenan, Peizhu Gong, Weizhong Zhang, Cheng Jin
Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates Authors: Weiyi He, Yuping Lin, Jiliang Tang, Yue Xing
CORA-Diff: Confidence-Oriented Residual Acceptance for Efficient Diffusion Language Model Inference Authors: Yifan Wu, Yufeng Zhang, Kenli Li
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation Authors: Jiaqi Liu, Haidong Kang, Qihui Zhao, Guo Yu, Jingchao Wang
TopoTuner: Topological Finetuning of Large Language Models Authors: Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad, Baha Rababah, Santu Karmaker, Cuneyt G. Akcora, Mubarak Shah
Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers Authors: Zhanglu Yan, Kaiwen Tang, Zixuan Zhu, Zhenyu Bai, Qianhui Liu, Yongxin Zhu, Weng-Fai Wong
BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks Authors: Kazi Ahmed Asif Fuad, Lizhong Chen
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts Authors: Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa, Partha Pratim Pande
Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression Authors: Tiancong Cheng, Ying Zhang, Zhiwen Yu, Yifang Yin, Bin Guo
Damage-Aware Bandit Pruning for Vision and Language Transformers Authors: Salem Ameen, Sunil Vadera
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference Authors: Yifan Dou, Shikan Lian, Shibo Li
Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals Authors: Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou
LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning Authors: Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, Zhijie Deng
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR Authors: Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He
SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits Authors: Kazi Ahmed Asif Fuad, Lizhong Chen
Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere Authors: Jiayang He, Tianling Xu, Diancheng Kang, Huaide Jiang, Junyan Bai, Shaoming Zheng, Xuan Song
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients Authors: Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao, Mayi Xu, Quanqing Xu, Wei Ren, Qiang Yang, Yang Liu
Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning Authors: Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie
Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View Authors: Tong Mao, Jinchao Xu
Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers Authors: Kamil KsiÄ Å¼ek, Piotr SuszyÅski, MichaÅ Jan WÅodarczyk, Jacek Tabor, PrzemysÅaw Biecek
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs Authors: Michael Rottoli, Subhankar Roy, Stefano Paraboschi
Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation Authors: Shun Qian, Bingquan Liu, Chengjie Sun, Zhen Xu, Baoxun Wang
Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalization Authors: Xuefei Wang, Jun Han, Zixuan Wang, Qingkai Zeng, Xiao Wang, Ruijie Wang, Jianxin Li
In-situ Autoguidance: Eliciting Self-Correction in Diffusion Models Authors: Enhao Gu, Haolin Hou
Gram-Space: Structure-Preserving Codebook Compression for Memory-Efficient Neuro-Symbolic AI Authors: Weilun Wang, Wantong Li
AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference Authors: Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michael Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li
ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression Authors: Renjie Liang, Zijian Xu, Jinqian Pan, Chengkun Sun, Zhengkang Fan, Shawn Li, You Qin, Mei Liu, Jie Xu

Large-Scale Training Systems and Efficiency (11)

1. Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

ArXiv ID: 2607.29674

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Maria Smirnova, Alexey Kravatskiy

Abstract: SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget. It outperforms SignSGD in practice, yet it can ascend even on a linear function. Signing the gradient before the Linear Minimization Oracle (LMO), rather than after, does not repair this: we construct a small explicit instance on which sign-before (MuonUSign) and sign-on-both-sides (MuonSign) ascend as well, so no placement of the sign around the oracle descends in general. Error feedback, the standard remedy for a biased compressor, does not rescue SignMuon: when applied to Muon's output, error feedback can fail for every smoothness constant, step size, and momentum. Applied to the gradient, error feedback does work, and EF21-MuonUSign and EF21-MuonSign attain the standard $\mathcal{O}(T^{-1/2})$ rate for the squared gradient norm on smooth nonconvex problems, the latter at one bit in each direction. Experiments then reverse the ordering: across centralized CIFAR-10, federated CIFAR-10, and the nanoGPT speedrun, the strongest compressed method is consistently sign-after-the-LMO, precisely the placement we prove divergent, with the provably convergent variants trailing it. Compressing after the LMO, a heuristic, matters more at these scales than the guarantee does.

Comment: One-bit Muon update compression reveals where error feedback does and does not restore convergence.

Topic Match: The work jointly addresses a large-model optimizer and extreme communication compression for distributed training.

Relevance: 9 Novelty: 8

2. Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

ArXiv ID: 2602.07494

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Shenxi Wu, Haosong Zhang, Xingjian Ma, Shirui Bian, Yichi Zhang, Xi Chen, Wei Lin

Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($Î¼$P) helps explain why many hyperparameters transfer across width. Yet depth scaling is less understood for modern architectures, whose computation graphs contain multiple parallel paths and residual aggregation. To unify various non-recurrent multi-path neural networks such as CNNs, ResNets, and Transformers, we introduce a graph-based notion of effective depth. Under stabilizing initializations and a maximal-update criterion, we show that the optimal learning rate decays with effective depth following a universal -3/2 power law. Here, the maximal-update criterion maximizes the typical one-step representation change at initialization without causing instability, and effective depth is the minimal path length from input to output, counting layers and residual additions. Experiments across diverse architectures confirm the predicted slope and enable reliable zero-shot transfer of learning rates across depths and widths, turning depth scaling into a predictable hyperparameter-transfer problem.

Comment: Effective depth predicts a universal minus-three-halves learning-rate transfer law across architectures.

Topic Match: The scaling law directly informs large-run hyperparameter configuration and is derived from multi-path architectural structure.

Relevance: 9 Novelty: 8

3. DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

ArXiv ID: 2607.22614

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Hanlin Du, Zhiyuan Yan, Yungang Bao, Sa wang

Abstract: RL-based LLM post-training increasingly disaggregates Rollout and Training across separate GPU resources, but static GPU partitioning suffers from severe pipeline bubbles under long-tail rollout latency. We present DynaResize, a runtime GPU reallocation system that dynamically switches GPUs between Rollout and Training to balance stage execution times without changing RL semantics. DynaResize decomposes resizing into fine-grained operations and removes non-startup-critical work from the critical path through communicator reuse, bounded state staging, and hysteresis-based resizing. Experimental results show that DynaResize can improve end-to-end throughput by 66.5% and reduce total execution time by 33% over the optimal static configuration, while hiding 27% of role-switching overhead.

Comment: Dynamically reallocates GPUs between rollout and training stages to remove post-training pipeline bubbles.

Topic Match: Runtime resource reallocation is a large-scale training-systems contribution with substantial throughput gains.

Relevance: 9 Novelty: 7

4. Paris: A Decentralized Trained Open-Weight Diffusion Model

ArXiv ID: 2510.03434

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy

Abstract: We present Paris, the first publicly released diffusion model pre-trained entirely through decentralized computation. Paris demonstrates that high-quality text-to-image generation can be achieved without centrally coordinated infrastructure. Paris is open for research and commercial use. Paris required implementing our Distributed Diffusion Training framework from scratch. The model consists of 8 expert diffusion models (129M-605M parameters each) trained in complete isolation with no gradient, parameter, or intermediate activation synchronization. Rather than requiring synchronized gradient updates across thousands of GPUs, we partition data into semantically coherent clusters where each expert independently optimizes its subset while collectively approximating the full distribution. A lightweight transformer router dynamically selects appropriate experts at inference, achieving generation quality comparable to centrally coordinated baselines. Eliminating synchronization enables training on heterogeneous hardware without specialized interconnects. Empirical validation confirms that Paris's decentralized training maintains generation quality while removing the dedicated GPU cluster requirement for large-scale diffusion models. Paris achieves this using 14$\times$ less training data and 16$\times$ less compute than the prior decentralized baseline.

Comment: Independent data-cluster experts eliminate synchronization during diffusion pretraining on heterogeneous hardware.

Topic Match: The defining contribution is a decentralized training design that removes gradient, parameter, and activation synchronization.

Relevance: 8 Novelty: 8

5. The Convergence Behavior of Adam under Heavy-Tailed Noise

ArXiv ID: 2607.27383

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Yijiang Pang

Abstract: We establish the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise. While several Adam variants are known to achieve optimal iteration complexity in bounded-variance nonsmooth nonconvex optimization, little is understood about their behavior when stochastic gradients admit only a bounded $p$-th central moment for some $p \in (1,2]$, a setting increasingly observed in modern deep learning. To address this gap, we generalize the recent online-to-nonconvex conversion framework to accommodate heavy-tailed martingale-difference noise. Building on this generalized framework, we develop a discounted regret analysis for Adam, without restrictive parameter coupling. Our results show that Adam converges to $(Ï,Îµ)$-stationary points under heavy-tailed noise. However, it exhibits a suboptimal iteration complexity and $p$-dependent convergence, a suboptimality that persists even in the bounded-variance case ($p=2$). Specifically, the $Îµ$-dominant term in the iteration complexity for reaching in-expectation stationarity is $T=\mathrm{O}\left(ÎÏ^{1/2}(G+Ï)^{\frac{5p}{3p-4}}Îµ^{-\left(\frac{5p}{3p-4}+\frac{3}{2}\right)}\right)$ for $p\in(\frac{4}{3},2]$, which simplifies to $T=\mathrm{O}(Îµ^{-13/2})$ when $p=2$. When the domain radius is known and used to control the online-learner output, a standard setup in related literature, the convergence rate improves to match the optimal complexity. In this case, the $Îµ$-dominant iteration complexity is $T=\mathrm{O}\left(ÎÏ^{1/2}(G+Ï)^{\frac{p}{p-1}}Îµ^{-\left(\frac{p}{p-1}+\frac{3}{2}\right)}\right)$ for $p\in(1,2]$, which simplifies to $T=\mathrm{O}(Îµ^{-7/2})$ when $p=2$. These findings provide new theoretical insight into the robustness and limitations of Adam in heavy-tailed regimes.

Comment: Establishes convergence rates and limitations for unmodified Adam under heavy-tailed gradient noise.

Topic Match: It provides foundational optimizer theory for a noise regime observed in modern large-model training.

Relevance: 8 Novelty: 8

6. Heterogeneous Decentralized Diffusion Models

ArXiv ID: 2603.06741

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy

Abstract: Training frontier-scale diffusion models often requires substantial computational resources concentrated in tightly-coupled clusters, limiting participation to well-resourced institutions. While Decentralized Diffusion Models (DDM) enable training multiple experts in isolation, existing approaches require 1176 GPU-days and homogeneous training objectives across all experts. We present an efficient framework that dramatically reduces resource requirements while supporting heterogeneous training objectives. Our approach combines three key contributions: (1) a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching), unified at inference time without any retraining; (2) pretrained checkpoint conversion from ImageNet-DDPM to Flow Matching objectives, accelerating convergence and enabling initialization without objective-specific pretraining; and (3) PixArt-$Î±$'s efficient AdaLN-Single architecture, reducing parameters while maintaining quality. Experiments on LAION-Aesthetics show that, relative to the training scale reported for prior DDM work, our approach reduces the compute by 16$\times$ and data by 14$\times$. Under aligned inference settings, our heterogeneous configuration achieves better FID and higher intra-prompt diversity than the homogeneous baseline. By eliminating synchronization requirements and enabling mixed DDPM/FM objectives, our framework makes decentralized generative model training accessible to contributors with single GPUs requiring only 24--48GB VRAM.

Comment: Trains diffusion experts independently under heterogeneous objectives and unifies them without retraining.

Topic Match: The core contribution is a decentralized training paradigm that removes synchronization and sharply lowers compute requirements.

Relevance: 8 Novelty: 8

7. Reassessing Muon for Matrix Factorization

ArXiv ID: 2607.13246

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Ali Parviz, Gal Mishne, Alex Cloninger

Abstract: Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.

Comment: Isolates Muon's orthogonalized update rule and tests when its reported optimizer advantages persist.

Topic Match: Muon is a large-scale training optimizer, and the controlled study provides insight into its optimization dynamics.

Relevance: 8 Novelty: 6

8. Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning

ArXiv ID: 2608.01418

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Wenhao Zhang, Yibo Xie, Rui Wang, Jiahua Yang, Lei Jiang, Zibo Yang, Yawei Wang, Jiali Xu, jasperawang, Haoyang Long, Huan Xiong, alantzhao

Abstract: Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models. Reusing each rollout batch for additional learner updates amortizes this cost, but later updates become increasingly off-policy as the learner departs from the behavior policy. At a token position, exact off-policy correction must account for both the current action and the probability of reaching its prefix. The cumulative importance ratio provides this correction, but its product form can produce an unwieldy dynamic range. We study Prefix-Normalized Policy Optimization (PNPO), which replaces the cumulative ratio with the geometric mean of likelihood ratios along each causal prefix, preserving causal-prefix dependence at each position while compressing the log-weight scale. In controlled long-context mathematical reasoning experiments, we induce two off-policy regimes by using one or four policy-update epochs per rollout batch. PNPO does not consistently outperform GSPO with one epoch. With four epochs, it attains the highest observed Avg@32 on each benchmark; the unweighted mean of the three independently selected benchmark peaks is 50.24, 3.00 percentage points above GSPO. Under a matched 2,400-update budget, four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to the 49.56 reached after 600 batches with one epoch. These results provide preliminary evidence that PNPO can be advantageous as training moves further off-policy.

Comment: Prefix-normalized importance weights enable extra learner updates per expensive LLM rollout batch.

Topic Match: The method primarily reduces rollout-generation cost through an off-policy training algorithm, while also analyzing policy-lag dynamics.

Relevance: 7 Novelty: 7

9. Communication-Efficient Secure Aggregation in Decentralized Learning

ArXiv ID: 2405.07708

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Sayan Biswas, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma, Milos Vujasinovic

Abstract: Decentralized learning (DL) enables participants to collaboratively train models without a central server, yet it faces significant scalability challenges that demand sparsification to reduce the prohibitive communication costs of peer-to-peer exchange. While secure aggregation effectively mitigates privacy risks in standard settings, it has remained fundamentally incompatible with sparsification in decentralized networks due to the mismatch of indices across local updates, forcing a trade-off between communication efficiency and privacy. This paper introduces CESAR, a novel protocol that resolves this incompatibility by integrating secure aggregation and sparsification to provide provable defense against honest-but-curious and colluding adversaries. By coordinating masks over parameter intersections, CESAR supports node dropouts and robust privacy without central aggregation. Empirical evaluations on models up to 124 million parameters demonstrate that CESAR matches the accuracy of non-private baselines while cutting total data exchange by 66.7 % compared to a standard full-parameter decentralized protocol (D-PSGD). With TopK sparsification on IID data, CESAR even exceeds by 0.3 % the accuracy achieved by D-PSGD with sparsification. Collectively, these results establish CESAR as the first decentralized protocol to achieve both privacy and communication efficiency through secure aggregation in DL.

Comment: Combines decentralized secure aggregation with coordinated update sparsification to cut communication by 66.7%.

Topic Match: It contributes a distributed-training communication protocol whose sparsification materially reduces exchange cost.

Relevance: 7 Novelty: 7

10. When Does Muon Help Agentic Reinforcement Learning?

ArXiv ID: 2607.16169

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun

Abstract: Muon is competitive with AdamW in large-scale pre-training, but its operating regime in reinforcement-learning post-training remains unclear. We map this regime on ALFWorld, a sparse-reward agentic benchmark, using three group-based objectives and Qwen2.5 models from 0.5B to 3B. Under a shared KL and clipping recipe, matched optimizer comparisons and AdamW rate controls trace the usable step-size range. AdamW responds non-monotonically to rate, whereas fan-in Muon remains stable at a more aggressive effective step: at $3 \times 10^{-5}$ it improves late success over an AdamW $10^{-6}$ baseline after correction across rate-metric tests. Its normalized-AUC effect is directionally positive but less uniform; the heuristic-matched lower-rate effect is less consistent, and tuned AdamW nearly matches high-rate Muon at 3B GraphGPO. High-rate Muon applies $3.53 \times$ AdamW's hidden-matrix update RMS; a full-budget RMS-matched control removes the late-success gain. Together, these results identify a recipe-level operating regime in which fan-in Muon supports a more aggressive stable effective step under shared KL and clipping: the margin is largest when optimization headroom remains and contracts near saturation, after AdamW tuning, or under magnitude matching. The scale-matched control ties this spectral effect to Muon's scale convention rather than establishing a universal optimizer ranking. Code is available at https://github.com/x66ccff/verl-muon.

Comment: Identifies how Muon's scale convention permits larger stable effective updates under matched RL training recipes.

Topic Match: The central result characterizes an optimizer's stable operating regime and update-scale dynamics.

Relevance: 7 Novelty: 6

11. Learned Digital Over-the-Air Computing for Federated Edge Learning

ArXiv ID: 2509.16577

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Antonio Tarizzo, Mohammad Kazemi, Deniz GÃ¼ndÃ¼z

Abstract: Over-the-air (OTA) aggregation enables federated edge learning (FEEL) by exploiting the superposition property of the wireless channel to merge communication with computation, eliminating the need to schedule and decode devices individually. Analog OTA schemes transmit uncoded updates but are sensitive to noise, fading, and power misalignment, motivating more robust digital alternatives. However, state-of-the-art (SoTA) digital OTA designs that combine unsourced random access (URA) with compressed sensing struggle in the low signal-to-noise ratio (SNR) regimes common in Internet of Things (IoT) deployments, where symbol recovery and active-device estimation become unreliable. We propose a learned digital OTA framework that jointly trains a URA codebook with an unrolled approximate message passing (AMP)-based decoder. The learned decoder incorporates per-layer damping, residual scaling, temperature-controlled Bayesian denoising, and a lightweight convolutional neural network (CNN) refinement, while the codebook is optimised end-to-end through a factorised parameterisation. At near-perfect-aggregation accuracy, the proposed design extends the viable SNR range by approximately 7\,dB over the SoTA baseline at the same uplink overhead, and generalises across models, activity levels, and heterogeneous data.

Comment: Jointly learns an over-the-air codebook and unrolled decoder to improve federated aggregation at low SNR.

Topic Match: It advances communication for distributed training, though specifically in federated edge networks.

Relevance: 6 Novelty: 7

Architecture and Training Dynamics (18)

1. Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

ArXiv ID: 2608.14664

Primary Topic: Architecture and Training Dynamics

Authors: Zeyu Liu, Jinhao Zhang, Yunquan Zhang, Guangming Tan, Xiang Gao, Fangming Liu, Daning Cheng

Abstract: How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.

Comment: Proves a first-order criterion for when function-preserving residual-depth growth can still reduce loss.

Topic Match: The paper directly analyzes residual depth, function-preserving growth, and activation-gradient training dynamics in large neural models.

Relevance: 9 Novelty: 8

2. AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

ArXiv ID: 2608.00434

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Ziqiang Cui, Han Shi, Bowei He, Yu Pan, Peiyang Liu, Shengyin Sun, Yankai Chen, Haoli Bai, Yichun Yin, Xue Liu, Chen Ma

Abstract: Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference. However, existing training frameworks adopt a rigid, fixed-length prediction horizon, disregarding the highly non-uniform information density of natural language and code. Forcing the auxiliary heads to predict across high-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone's latent representations, the resulting gradients backpropagate and interfere with the model's core capabilities. We propose AdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence. At its core, an entropy-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable-length groups. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone. Across mathematical reasoning, code generation, and general benchmarks on three backbones (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-12B), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.

Comment: Entropy-masked multi-token objectives adapt each token's prediction horizon to semantic boundaries.

Topic Match: The core contribution changes the LLM training objective and gradient interactions, with parallel decoding providing secondary efficiency gains.

Relevance: 9 Novelty: 8

3. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

ArXiv ID: 2605.08696

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Benjamin L. Badger

Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput. Here we introduce the Structured Recurrent Mixer, an architecture that allows for algebraic conversion between a sequence parallel representation at train time and a recurrent representation at inference, notably without the need for specialized kernels or device-specific memory management. We show experimentally that this dual representation allows for greater training efficiency, higher input information capacity, and larger inference throughput and concurrency when compared to other linear complexity models. We postulate that recurrent models are poorly suited to extended sequence length scaling for information-rich inputs typical of language, but are well suited to scaling in the sample (batch) dimension due to their constant memory per sample. We provide Mojo/MAX inference implementations of SRMs exhibiting 12x the throughput and 170x the concurrency of similarly powerful Transformers inferenced on vLLM, increases characteristic of Pytorch implementations resulting in a 30\% increase in compute-constant GSM8k Pass@k. We conclude by demonstrating that SRMs are effective reinforcement learning training candidates.

Comment: Introduces a recurrent mixer with parallel training and an algebraically equivalent recurrent inference form.

Topic Match: The central contribution is a new sequence architecture that changes both training parallelism and inference cost.

Relevance: 9 Novelty: 8

4. Mask-Based Priors Are More Persistent than Query-Key Initializations

ArXiv ID: 2608.00418

Primary Topic: Architecture and Training Dynamics

Authors: Mingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey

Abstract: Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that, despite fitting the observed domain, Transformers often extrapolate according to a simpler minimum-degree interpolator rather than the true target function. These Boolean tasks are not practical applications, but controlled stress tests for understanding Transformer inductive bias. We ask whether this failure mode can be corrected by injecting explicit structural priors into attention. Existing structured-initialization methods alter Transformer inductive bias indirectly, by choosing query and key projections whose similarity scores approximate a desired attention pattern. However, we find that when applied to Boolean extrapolation, these QK-based priors can be rapidly overwritten during training and fail to change the learned extrapolation rule. We propose a simpler alternative: initialize the additive attention mask directly. Unlike standard hard masks used for causality or locality attention, our mask is a finite, learnable attention-logit bias initialized from task-level interaction structure. This separates the structural prior from content-dependent attention scores, allowing it to persist throughout optimization. On Boolean reasoning tasks, mask-based initialization achieves near-perfect extrapolation where vanilla and QK-initialized Transformers remain trapped by the default inductive bias. The same mechanism also improves low-data arithmetic performance and remains competitive on vision and language benchmarks. These results show that attention masks can serve not only as architectural constraints, but as a simple substrate for encoding persistent inductive bias in Transformers.

Comment: Encodes persistent structural priors directly as learnable attention-mask biases.

Topic Match: It introduces an attention mechanism and explains why its inductive bias survives optimization better than QK initialization.

Relevance: 9 Novelty: 8

5. Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth

ArXiv ID: 2608.01075

Primary Topic: Architecture and Training Dynamics

Authors: Kehan Wang

Abstract: Depth-routing residual architectures allow Transformer layers to retrieve earlier representations instead of inheriting only the immediately preceding state. Existing Block Attention Residuals, however, use a single content-dependent depth mixture to construct the inputs to queries, keys, and values. This design couples two functionally different decisions: queries and keys determine where attention matches, whereas values determine what content is retrieved. We therefore ask whether matching and content retrieval should be forced to read from the same depth. We introduce Role-Decoupled Attention Residuals (RD-AttnRes), a minimal extension that shares one depth route between queries and keys while learning an independent value route over the same residual sources. Tying the two routing queries exactly recovers the parent architecture, while decoupling them adds only one model-width vector per layer and introduces no additional token-to-token attention operation. We evaluate RD-AttnRes using a frozen, paired pretraining protocol on FineWeb-Edu with five matched seeds for both 120M- and 343M-parameter models and a 2.0B-token training budget. RD-AttnRes improves validation negative log-likelihood in all 10 matched comparisons. The mean reductions are 0.0301 and 0.0247, corresponding to perplexity reductions of 2.97 percent and 2.43 percent at 120M and 343M parameters, respectively. Early-budget controls indicate that neither the additional parameter count, duplicated routing execution, nor a fixed value route reproduces the improvement. Routing diagnostics further reveal persistent divergence between the query-key and value depth distributions. These results suggest that, within the evaluated training regime, attention matching and content retrieval benefit from distinct reads over the residual hierarchy.

Comment: Separates query-key depth routing from value depth routing within attention residuals.

Topic Match: It proposes and analyzes a minimal architectural change to residual-depth computation in Transformers.

Relevance: 9 Novelty: 7

6. Entropy-Gated Latent Recursion

ArXiv ID: 2606.16620

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Soham Bhattacharjee, Dushyant Singh Chauhan, Salem Lahlou, Martin Takac, Nils Lukas

Abstract: Inference-time scaling has become the dominant lever for improving language-model reasoning, but existing methods derive rollout diversity from a single source: stochastic token-level sampling. We argue that this single-axis sampling space is fundamentally limiting, and identify a second, fully deterministic and complementary axis: the layer span $L$ at which a frozen model's top decoder layers are recursively re-applied at high-uncertainty tokens. Different choices of $L$ produce distinct rollouts that solve different subsets of problems, with no stochasticity. We instantiate this axis through Entropy-Gated Latent Recursion (EGLR), a training-free decoding procedure that re-applies the top-$L$ layers for at most $K_{\max}$ iterations until the next-token distribution converges. Combined with $T$ temperature samples, EGLR turns a single-axis stochastic rollout pool into an $L\times T$ Cartesian sampling space at almost the same per-rollout cost. We characterize this space across $8$ instruction-tuned models and $6$ math reasoning benchmarks, and show that the $L$-axis is genuinely complementary to temperature: on MATH-500 with Qwen2.5-3B-Instruct, the joint $L\times T$ oracle reaches $91.6\%$, $+8.2$ percentage points beyond the temperature-only oracle ($83.4\%$) and $+10.4$ points beyond the layer-only oracle ($81.2\%$), confirming that the two axes capture genuinely complementary problems. The expanded rollout pool provides richer per-prompt candidates for any downstream procedure that consumes rollouts, including self-consistency, best-of-$N$ with verifiers, and group-relative RL training (GRPO), opening a new direction for inference-time scaling that does not rely on stochastic noise.

Comment: Creates a deterministic rollout axis by recursively reapplying selected decoder layers at uncertain tokens.

Topic Match: Layer-span recursion is a new dynamic-computation mechanism that also expands inference-time scaling efficiently.

Relevance: 8 Novelty: 8

7. Eigenvalues as a Metric for Memory Dynamics in Sequence Models

ArXiv ID: 2510.09379

Primary Topic: Architecture and Training Dynamics

Authors: Rahel Rickenbach, Jelena Trisovic, Alexandre Didier, Jerome Sieber, Melanie N. Zeilinger

Abstract: While softmax attention drives state-of-the-art performance in sequence modeling, its quadratic complexity motivates linear alternatives such as state space models (SSMs). Structural differences between the two model classes, however, hinder direct comparisons of their memory dynamics, creating the need for a common metric to analyze, interpret, and improve their information processing capabilities. Inspired by recent advances in SSM performance driven by eigenvalue-guided insights, we leverage the dynamical systems framework to bring attention models into a unified analytical framework with SSMs. This allows us to perform a structured analysis, which investigates the applicability of an eigenvalue-spectrum memory dynamics metric to attention models. To this end, we first conduct an extensive empirical study across diverse attention-based models and SSMs on a range of benchmarks. We show that, for both model classes, eigenvalues influence key aspects of memory and long-range dependency modeling, revealing spectral signatures that align with task requirements. Building on these findings, we show how spectral signatures can motivate architectural modifications, how they can be guided through the training process, and how they can provide information about feature importance. The results thereby enable and emphasize the role of eigenvalue analysis as a principled metric for interpreting, explaining, and ultimately improving the capabilities of sequence models.

Comment: A shared eigenvalue-spectrum metric links memory dynamics in attention and state-space models.

Topic Match: It provides mechanistic analysis connecting sequence-model architecture, memory behavior, and trainable spectral modifications.

Relevance: 8 Novelty: 7

8. Statistical Mechanics of Learning on Product Wasserstein Manifolds

ArXiv ID: 2608.01434

Primary Topic: Architecture and Training Dynamics

Authors: Srinivasa Rao P Vangmayi P Reddy

Abstract: Normally the statistical mechanics of learning treats constraints on weight distributions as restrictions that shrink the space of possible solutions. Therefore, it reduces model capacity. In this paper we would like to take a contrary approach, which, however, is based on the earlier work on distribution-constrained perceptrons. Rather than treating a prescribed weight distribution as a mere restriction, we propose that it defines the intrinsic geometry upon which learning naturally unfolds. We formulate both deep neural networks and variational quantum circuits as gradient flows on a product of Wasserstein manifolds -- one classical Wasserstein space for each layer and one quantum Wasserstein space for the circuit parameters. Within this geometry, the capacity reduction, which was previously associated with distributional constraints, appears as the metric structure of the constraint manifold itself. We develop a hierarchical mean-field description for deep networks, extend the framework to the quantum setting using the quantum Wasserstein distance of order 1, and introduce two such practical algorithms, Hierarchical DisCo-SGD and Quantum DisCo, that follow approximate geodesics on the manifold of the product itself. Experiments on teacher-student problems, standard image classification tasks, and small variational quantum classifiers show that respecting these distributional geometries improves generalization, stabilizes training, and reduces the severity of barren plateaus compared with unconstrained and purely norm-based baselines. This approach firstly reframes structural constraints as geometric priors and suggests a route for incorporating biological, spectral, or hardware-derived distributional information into both learning systems, viz., classical and quantum learning.

Comment: Wasserstein-manifold gradient flows yield geometry-aware optimization for layered networks.

Topic Match: The central contribution is a geometric account of optimization and training stability across network layers.

Relevance: 7 Novelty: 8

9. One-Sided Quantile Coupling for Flow Matching

ArXiv ID: 2608.00978

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Jin-Young Kim, So-Yoon Cho, Hyun-Gyoon Kim

Abstract: Flow Matching trains continuous-time generative models by regressing the velocity field of a probability path between a simple source distribution and a target data distribution. The coupling that pairs source and target samples strongly affects optimization and sample quality, but structured couplings typically rely on mini-batch transport or assignment procedures whose cost grows at least quadratically in batch size. We propose Quantile Coupling Flow Matching (QC-FM), a lightweight one-sided coupling: rather than matching two pre-sampled batches, it samples only the data batch and constructs each paired source directly. Data ranks projected along a small number of random orthogonal directions are mapped to Gaussian quantiles, and the latent code is completed in the orthogonal complement by conditional Gaussian sampling. The construction is one-dimensional per slice, so the coupling requires no pairwise cost matrix and no assignment to solve. We show that, for each drawn frame, this coupling eliminates the irreducible regression variance along every selected slice and makes the ideal flow exactly straight there, while leaving the sampling prior unchanged: generation still starts from the standard Gaussian, and the training source deviates from it only through the copula of the slice codes, whose transport cost we bound. For training, we apply QC to an anchor subset and complete the remaining source slots with exact Gaussian samples, retaining the QC bias while preserving an explicit signal from the Baseline coupling. Across CIFAR-10, CelebA, FFHQ, and ImageNet-64, QC-FM improves over the Baseline under matched training budgets, reducing FID by up to 12.9%, and outperforms OT-CFM on all four datasets. These results suggest that preserving projected rank structure is a simple and scalable way to inject useful geometric bias into FM couplings without solving a mini-batch transport problem.

Comment: One-sided quantile coupling removes mini-batch assignment while reducing flow-matching regression variance.

Topic Match: The coupling chiefly changes generative-model training dynamics, while eliminating quadratic assignment adds an efficiency benefit.

Relevance: 7 Novelty: 8

10. Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

ArXiv ID: 2608.00301

Primary Topic: Architecture and Training Dynamics

Authors: Xujun Che, Yuchen Yuan, Weida Zhao, Chenyang Yu

Abstract: Error-penalized scoring rules ($+1$ for a correct answer, $-Î»$ for a wrong one, $0$ for abstaining) are increasingly prescribed against hallucination: a rational agent facing such a rule answers exactly when its correctness probability exceeds Chow's threshold $t^\ast=Î»/(1+Î»)$. We prove that a KL-anchored gradient learner can do the opposite. When abstention is a discrete action, the reward gradient and the anchor's restoring force are throttled by the same gate-saturation factor and die together: under explicit conditions (among them, blanket answering loses score in expectation and prompts share a bounded readout) the model drifts toward refusing everything, its mean training reward rising to zero like $1/t$ in training time $t$, so the curve reads as improvement while coverage collapses. The advantage estimator compounds the failure: in its sparse-answer regime, group normalization silently replaces every designed penalty with an effective penalty of one, moving the learned threshold from $t^\ast$ to $1/2$. The repair is structural: train a mandatory confidence report with a strictly proper score plus a correctness reward, and abstain only at deployment by thresholding the report. The always-emitted report has no gate to saturate, so no shared factor can kill its reward gradient and its anchor together, and its calibrated optimum is attracting. Simulations confirm every prediction, and experiments on language models at two scales confirm the mechanism live: the rule silences questions the models demonstrably still solve within ten optimizer steps, an ablation isolates the cause, and report-level training raises coverage, accuracy, and calibration together.

Comment: Derives a collapse law showing that abstention saturation simultaneously suppresses reward and KL-anchor gradients.

Topic Match: The central result is a mechanistic account of training collapse and a structural repair.

Relevance: 7 Novelty: 8

11. DiffusionGemma Technical Report

ArXiv ID: 2608.00146

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: DiffusionGemma Team, Adrien Ali TaÃ¯ga, James Assiene, Daniele Calandriello, Rahma Chaabouni, JoÃ£o Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, Ofir Nabati, JoÃ£o Gabriel Oliveira, Nicolas Perez-Nieves, Nastasia Prutianova, Bobak Shahriari, Jean Tarbouriech, Pavel Tyletski, ÃaÄlar ÃnlÃ¼, Cindy Wu, Glenn Cameron, Jerome Connor, Sertan Girgin, Maarten Grootendorst, Alon Levkovitch, Eliya Nachmani, Omar Sanseviero, Piotr Stanczyk, Quentin Berthet, Andrew Campbell, ClÃ©ment Crepy, Valentin De Bortoli, Arnaud Doucet, Romuald Elie, Alexandre Galashov, Klaus Greff, Alexis Jacq, David Ruhe, Yu-Han Wu, Sebastian Flennerhag, Brendan O'Donoghue, George Scrivener, Shantanu Thakoor

Abstract: We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Comment: Discrete diffusion replaces token-by-token decoding with parallel block refinement while retaining optional autoregressive generation.

Topic Match: Architecture is primary because parallel iterative denoising fundamentally changes the model's generation computation.

Relevance: 7 Novelty: 8

12. Human-like working memory signatures emerge from intrinsically plastic artificial neurons for robust dynamic vision

ArXiv ID: 2512.15829

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Jingli Liu, Huannan Zheng, Bohao Zou, Kezhou Yang

Abstract: While the unsustainable energy cost of artificial intelligence necessitates physics-driven computing, its performance superiority over full-precision GPUs remains a challenge. We bridge this gap by repurposing the Joule-heating relaxation dynamics of magnetic tunnel junctions, conventionally suppressed as noise, into neuronal intrinsic plasticity, realizing working memory with human-like features. Traditional AI utilizes energy-intensive digital memory that accumulates historical noise in dynamic environments. Conversely, our Intrinsic Plasticity Network (IPNet) leverages thermodynamic dissipation as a temporal filter. We provide direct system-level evidence that this physics-driven memory yields an 18x error reduction compared to spatiotemporal convolutional models in dynamic vision tasks, reducing memory-energy overhead by >90,000x. In autonomous driving, IPNet reduces prediction errors by 12.4% versus recurrent networks. This establishes a neuromorphic paradigm that shatters efficiency limits and surpasses conventional algorithmic performance.

Comment: Magnetic-relaxation intrinsic plasticity supplies an energy-efficient recurrent memory primitive.

Topic Match: The artificial-neuron dynamics are the core mechanism, accompanied by unusually large hardware memory-efficiency gains.

Relevance: 6 Novelty: 8

13. The Fourth Quadrant: A Stylized View of Benign Misfitting

ArXiv ID: 2608.01032

Primary Topic: Architecture and Training Dynamics

Authors: Gireeja Ranade, Anant Sahai

Abstract: Training error is what we can observe on a training set; test error is the quantity we actually care about. We study linear regression with squared-error in a deterministic $(d+1)$-dimensional single-spike model. Each stylized training vector has the same informative spike coordinate, of amplitude $\sqrtÎ³$ with $Î³>1$. The remaining directions are nuisance, and the nuisance components of distinct training vectors all have equal norm and are mutually orthogonal. The training labels are all $1$. Fresh test points are drawn from $\vec{x}{\rm test} \sim \mathcal{N}(\vec{0},\operatorname{diag}(Î³,1,\ldots,1))$, with the noise-free test labels being the normalized spike coordinate $x empirical training error (despite the descent premise in its name). Finally, we show that the unavoidable nuisance component responsible for the training misfit also controls the predictor's adversarial sensitivity.}[1]/\sqrtÎ³$. We focus on linear predictors in the span of the training vectors, the class naturally reached by zero-initialized linear gradient methods. We exhibit a range of training-set sizes $n$ in which every span predictor that generalizes well must fit the training data \emph{worse} than the zero predictor. We call this regime \emph{benign misfitting}, or the fourth quadrant. The best span predictor begins to generalize when $n\gg d/Î³^2$, while interpolation does not generalize until the later threshold $n\gg d/Î³$. In the window $d/Î³^2 \ll n \ll d/Î³$, useful prediction within the linear span lies beyond interpolation: predictions on the training points overshoot the labels. We show that one-pass stochastic gradient descent (SGD), with a large constant learning rate, reaches small test error throughout this window---matching the best span predictor up to a logarithmic factor. We also verify directly that it indeed has \emph{large

Comment: A stylized SGD regime generalizes only by deliberately overshooting its training labels.

Topic Match: The paper offers a new optimization and generalization-dynamics phenomenon, albeit in a stylized linear model.

Relevance: 6 Novelty: 8

14. Plasticity of Growing and Elastic Neural Networks in Online Continual Learning

ArXiv ID: 2608.01475

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Jeong Min Kong, Richard S. Sutton

Abstract: Neural networks that can grow or both grow and shrink during learning, referred to as growing neural networks and elastic neural networks, respectively, have recently been explored in offline continual learning with a particular focus on catastrophic forgetting. Driven by the observations that 1) online continual learning closely resembles how animals learn; 2) loss of plasticity---the progressive decline in a learning network's ability to learn---is another crucial challenge facing continual learning; and 3) incremental introduction of randomly initialized hidden units was recently shown to help preserve plasticity, in this paper, we study the plasticity of several foundational growing and elastic networks in online continual learning. Our experiments in supervised learning settings show that adaptive growing networks, which incrementally incorporate new, randomly initialized units to the network while keeping all existing connections adaptive, can maintain high prediction accuracy without losing plasticity despite the continuous increase in the dead hidden unit proportion. Furthermore, we demonstrate that adaptive elastic networks, which in addition to progressively adding new hidden units also prune estimated dead hidden units at the beginning of each new task, can achieve excellent accuracy without loss of plasticity while simultaneously maintaining a near-constant, compact size. Our results suggest that growing and elastic networks, which exhibit the ability to adapt its structure to the relevant learning objectives, can be a promising class of algorithms also for preserving high plasticity in online continual learning.

Comment: Adaptive growth and dead-unit pruning preserve plasticity while maintaining a compact network.

Topic Match: The core contribution concerns dynamic network structure and the training dynamics of plasticity loss.

Relevance: 7 Novelty: 6

15. Beyond Feature and Structure Alignment: Learning Transferable Propagation Knowledge for Graph Foundation Models

ArXiv ID: 2607.28980

Primary Topic: Architecture and Training Dynamics

Authors: Yi Wang, Jitao Zhao, Di Jin, Dongxiao He

Abstract: Graph Foundation Models (GFMs) have recently emerged as a promising paradigm for enabling knowledge transfer across diverse domains. Unlike traditional graph learning methods that are typically designed for in-domain settings, GFMs aim to learn transferable knowledge that can generalize to unseen graph domains. However, unlike language or visual data, graphs lack intrinsic and unified representation units, such as tokens in language and patches in vision, making it challenging to identify transferable knowledge units for building graph foundation models. Existing graph foundation models mainly focus on mitigating domain discrepancies through feature alignment and structure alignment, while overlooking the exploration of transferable knowledge units underlying graph data. Moreover, these methods generally rely on fixed propagation mechanisms during message passing, overlooking the heterogeneity in propagation patterns, as different edges may exhibit distinct propagation patterns for different feature dimensions. To address these limitations, we propose a Propagation-aware Graph Foundation Model (ProGFM), which regards the propagation relationships between edges and feature dimensions as transferable knowledge units. Through a propagation relationship prototype bank, ProGFM learns cross-domain transferable propagation knowledge, enabling adaptive information aggregation in unseen graph domains. Extensive experiments across various cross-domain transfer scenarios demonstrate that ProGFM possesses strong cross-domain knowledge transfer capability and exhibits superior generalization performance compared with existing methods.

Comment: Learns edge-by-feature propagation prototypes for adaptive message passing across unseen graph domains.

Topic Match: The core contribution is a new adaptive propagation mechanism, though it is confined to graph foundation models.

Relevance: 6 Novelty: 7

16. Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

ArXiv ID: 2607.29363

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Yi Luo, Rongzhi Gu, Jixun Yao

Abstract: Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.

Comment: Co-designs low-rate continuous tokens and multi-path autoregressive routing to limit long-horizon error accumulation.

Topic Match: Its main contribution is an autoregressive sequence-modeling mechanism for stable computation with shorter token sequences.

Relevance: 6 Novelty: 7

17. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

ArXiv ID: 2607.29209

Primary Topic: Architecture and Training Dynamics

Authors: Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen

Abstract: Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.

Comment: Diagnoses entropy collapse from magnitude and temporal mismatch when fusing RLVR and distillation advantages.

Topic Match: Its strongest fit is mechanistic training-stability analysis, although it operates in post-training.

Relevance: 6 Novelty: 7

18. When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer

ArXiv ID: 2608.01130

Primary Topic: Architecture and Training Dynamics

Authors: Yuyang Shen

Abstract: A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.

Comment: Characterizes when surrogate-gradient updates transfer to downstream decision utility through alignment and curvature.

Topic Match: Its strongest fit is theoretical analysis of optimization dynamics and gradient-transfer failure.

Relevance: 6 Novelty: 7

Efficiency, Compression, and Large-Scale Training (38)

1. LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

ArXiv ID: 2608.11231

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Yirui Liu, Ruoling Qi, Longwen Wang, Xuaner Wu, Jian Chen, Yuxin Jin, Jiawei Shao, Xuelong Li

Abstract: LLM serving is increasingly accelerated by position-independent caching (PIC). Existing PIC methods, however, are built for full-attention models, where a token-indexed KV cache underlies its core operations: matching reusable token chunks, concatenating their KV entries, and selectively recomputing a few tokens to restore cross-chunk context. Hybrid LLMs break these primitives---they replace most attention layers with linear recurrences that expose only a fixed-size state, leaving no token-indexed KV to concatenate or to locally repair. This raises a natural question: can PIC benefit hybrid models, and what would it take? We present LinearKV, a training-free hybrid-PIC framework. Its key insight is a \emph{decoupled initialization}: each linear layer maps its $K$ matched local states to a single initial state, while full-attention layers concatenate their KV as before. LinearKV is therefore compatible with existing PIC methods, reusing their token selection and recomputation as-is. Under this framework, we find that a \emph{single cached state} suffices as the linear layer's initializer. The algebraically principled alternative---composing all $K$ cached states into the exact full-prefix state, as concurrent work HYPIC does---is unnecessary and, on some architectures, even harmful. We compare the two across three hybrid models and three PIC selectors. On the two GDN models the two tie, both recovering most of full quality (up to $92\%$); on the Mamba-2 model, exact composition instead collapses under every selector---under EPIC, for instance, it recovers only $46.6\%$ of full quality, versus $86.8\%$ for a single cached block initializer. A single state initializer is also cheaper, cutting time-to-first-token to $0.46\times$ full prefill versus a further $5$--$17\%$ overhead for exact composition; results hold across LongBench QA and RULER at 8K--32K.

Comment: A single recurrent state enables position-independent cache reuse in hybrid LLMs.

Topic Match: The core contribution is a cheaper cache-reuse mechanism adapted to hybrid attention-recurrence architectures.

Relevance: 9 Novelty: 8

2. A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models

ArXiv ID: 2605.25344

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang, Pan Zhang, Shi-Ju Ran, Gang Su

Abstract: Dense linear maps carry much of the parameter and computational burden of modern neural networks, yet their dense form leaves the organization of learned couplings implicit. Quantum many-body physics organizes exponentially large operators by writing a global Hamiltonian as a sum of local terms, (\hat H=\sum_k\hat h_k). Whether the same structural principle can carry learned neural maps is unknown. We introduce Tensor Mixture (MixT), which represents a dense map as a natively executable sum of overlapping local tensor operators without imposing an explicit matrix-rank constraint. The local-term count (N_T) sets the effective nonlocality and operator complexity, while the number of replaced Transformer blocks (N_B) extends this structural coordinate across network depth. Tests on Qwen3-8B and LLaMA2-7B reveal a broad recoverable regime followed by an abrupt, model-specific boundary that is remarkably stable against changes in (N_T). Accuracy and output-distribution statistics reorganize together across the boundary; in LLaMA2-7B, the same depth separates two scaling regimes of inter-layer geometry drift. The directly executed structure also reduces parameters, arithmetic, storage, and memory. These results establish the local-sum structure as a viable organizing principle for learned linear maps at billion-parameter scale and expose a sharp boundary in their tolerance to structural simplification.

Comment: Overlapping local tensor operators replace dense Transformer maps without an explicit rank constraint.

Topic Match: The local-operator representation primarily compresses parameters and computation while introducing a distinct structural architecture.

Relevance: 9 Novelty: 8

3. Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization

ArXiv ID: 2608.01078

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao

Abstract: We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are typically trained to exhibit chain-of-thought reasoning capabilities, in the PTQ regime, even the latest CAT-Q method based on learning-based differentiable ternarization still leads to performance collapse on challenging mathematics and coding tasks when using conventional calibration schemes that ignore the model's reasoning process. Driven by this finding, we introduce a simple calibration approach, Attend to Your Own Thoughts (AYOT), where reasoning traces and final answers generated by the pre-trained high-precision target LLM on a proper set of calibration samples are used as the context input during the ternarization process, along with the corresponding questions. ScaleQ-1.58 is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties: (1) with only 4M calibration tokens, Qwen3-1.7B ternarized by ScaleQ-1.58 reaches over 90.52% of the performance of the prior best BitNet b1.58 2B4T averaged over 4 mathematics and coding tasks, and our ternary Qwen3-4B shows an absolute gain of 8.97%, while requiring 1,000,000x fewer calibration tokens for quantization; (2) ScaleQ-1.58 generalizes well to both dense and MoE architectures, with performance improving as model scale increases (up to 235B parameters); (3) ScaleQ-1.58 demonstrates strong generalization across tasks of varying difficulty levels, including mathematics, coding and scientific logic reasoning, as well as commonsense reasoning and basic language generation; (4) its performance continues to improve as the number of calibration tokens increases. Notably, AYOT also exhibits strong generalization ability across other quantization bit-widths. Code will be available at https://github.com/IntelChina-AI/BitTern.

Comment: Uses model-generated reasoning traces to calibrate scalable 1.58-bit post-training quantization.

Topic Match: The paper directly advances extreme LLM quantization and demonstrates scaling across dense and MoE models.

Relevance: 9 Novelty: 8

4. RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction

ArXiv ID: 2608.01247

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Changwoo Baek, Seungjun Shin, Kyeongbo Kong

Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complements this selection-based formulation with learned restoration under the same total KV budget. Our key insight is that, although the information lost through eviction is context-specific, the mechanism for generating its compact complement can be shared across contexts. After context prefill, a few restore tokens attend to the full KV cache in a single LoRA-adapted pass, generating a compact, context-conditioned restore cache. The base importance scorer and eviction rule remain unchanged, and the adapters are disabled for all subsequent queries and decoding. RestoreKV is trained through parameter-efficient self-distillation from the frozen full-cache model, optimizing only $0.4\%$ of the parameters and requiring no task-specific tuning. Across four backbones and four long-context benchmarks, RestoreKV substantially reduces compression-induced degradation. On Qwen3-4B, it improves 59 of 60 paired, budget-matched settings across five base eviction methods; at a $5\%$ budget, it raises KVzip from $38.2$ to $73.2$ on RULER-4K. Applied to KVzip+, RestoreKV reaches $86.4$ RULER accuracy at $16\times$ compression on the KVPress Benchmark, while adding less than $0.5\%$ one-time cache-construction overhead in a 32K-context evaluation. Our project page is available at https://paper.pnu-cvsp.com/RestoreKV/

Comment: Learned restore tokens reconstruct a compact complement to evicted KV entries under the same total cache budget.

Topic Match: Efficiency scaling is primary because the method preserves long-context capability under aggressive KV-cache compression.

Relevance: 9 Novelty: 8

5. Stem: Rethinking Causal Information Flow in Sparse Attention

ArXiv ID: 2603.06274

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Lin Niu, Xin Luo, Linchuan Xie, Yifu Sun, Guanghua Yu, Jianchen Zhu, S Kevin Zhou

Abstract: The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-filling phase. In this paper, we rethink the causal attention mechanism from the perspective of information flow. Due to causal constraints, tokens at initial positions participate in the aggregation of every subsequent token. However, existing sparse methods typically apply a uniform top-k selection across all token positions within a layer, ignoring the cumulative dependency of token information inherent in causal architectures. To address this, we propose Stem, a novel, plug-and-play sparsity module aligned with information flow. First, Stem employs the Token Position-Decay strategy, applying position-dependent top-k within each layer to retain initial tokens for recursive dependencies. Second, to preserve information-rich tokens, Stem utilizes the Output-Aware Metric. It prioritizes high-impact tokens based on approximate output magnitude. Extensive evaluations demonstrate that Stem achieves superior accuracy with reduced computation and pre-filling latency.

Comment: Position-dependent top-k selection and output-aware token scoring reduce sparse-attention prefill cost.

Topic Match: Reducing attention computation is central; the causal information-flow mechanism also contributes to attention design.

Relevance: 9 Novelty: 7

6. A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models

ArXiv ID: 2509.22536

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Wenjun Wang, Shuo Cai, Congkai Xie, Mingfa Feng, Yiming Zhang, Zhen Li, Kejing Yang, Ming Li, Jiannong Cao, Hongxia Yang

Abstract: The immense computational cost of training Large Language Models (LLMs) presents a major barrier to innovation. While FP8 training offers a promising solution with significant theoretical efficiency gains, its widespread adoption has been hindered by the lack of a comprehensive, open-source training recipe. To bridge this gap, we introduce an end-to-end FP8 training recipe that seamlessly integrates continual pre-training and supervised fine-tuning. Our methodology employs a fine-grained, hybrid-granularity quantization strategy to maintain numerical fidelity while maximizing computational efficiency. Through extensive experiments, including the continue pre-training of models on a 160B-token corpus, we demonstrate that our recipe is not only remarkably stable but also essentially lossless, achieving performance on par with the BF16 baseline across a suite of reasoning benchmarks. Crucially, this is achieved with substantial efficiency improvements, including up to a 22% reduction in training time, a 14% decrease in peak memory usage, and a 19% increase in throughput. Our results establish FP8 as a practical and robust alternative to BF16, and we will release the accompanying code to further democratize large-scale model training.

Comment: A hybrid-granularity FP8 recipe enables stable, near-lossless training with lower time, memory, and compute costs.

Topic Match: Efficiency scaling is primary because low-precision training directly reduces the cost of large-model continual pretraining and fine-tuning.

Relevance: 9 Novelty: 6

7. Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling

ArXiv ID: 2605.15100

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Rongman Xu, Yifei Li, Tianzhe Zhao, Yanrui Wu, Bo Li, Hang Yan

Abstract: Large Language Models (LLMs) have demonstrated remarkable abilities in reasoning. However, maximizing their potential through inference-time scaling faces challenges in trade-off between sampling budget and reasoning quality. Current strategies remain inefficient as they typically treat sampling width and depth as orthogonal objectives, where width consensus methods risk reinforcing hallucinations, while depth pruning mechanisms prematurely truncate complex yet valid reasoning chains. Therefore, we propose Dual-Dimensional Consistency (DDC), a unified framework that bridges path quality with adaptive termination. By coupling Confidence-Weighted Bayesian protocol with a Trend-Aware Stratified Pruning, our method ensures that computational resources are concentrated on high quality reasoning paths, filtering hallucinations while accelerating consensus. Evaluations across five benchmarks demonstrate that this approach reduces token consumption by over 10 times while maintaining or exceeding the accuracy of strong baselines across various LLMs.

Comment: Jointly adapts reasoning-path width and depth to reduce inference tokens while preserving quality.

Topic Match: Its primary contribution is adaptive allocation and termination of inference-time computation.

Relevance: 8 Novelty: 7

8. Just on Time: Token-Level Early Stopping for Diffusion Language Models

ArXiv ID: 2602.11133

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Zakhar Kohut, Severyn Shykula, Mykola Vysotskyi, Serhii Dmytryshyn, Dmytro Khamula, Michal Zakrzewski, Damian Rynczak, Jacek MaÅecki, Taras Rumezhak, Volodymyr Karpiv

Abstract: Diffusion language models generate text through iterative refinement, a process that is often computationally inefficient because many tokens reach stability long before the final denoising step. We introduce a training-free, token-level early stopping approach that identifies convergence independently at each position. Our method leverages lightweight signals derived from the model's predictions and local context to dynamically determine when individual tokens can be finalized. This yields adaptive per-token freezing without task-specific fine-tuning, substantially reducing the total number of diffusion steps required. Across diverse benchmarks, spanning mathematical reasoning, general question answering, and scientific understanding, our approach achieves substantial efficiency gains while preserving generation quality.

Comment: Freezes diffusion-language-model tokens independently once lightweight convergence signals indicate stability.

Topic Match: Token-level early stopping directly reduces iterative generation cost through dynamic computation.

Relevance: 8 Novelty: 7

9. CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding

ArXiv ID: 2608.00531

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Aofan Liu, Jingxiang Meng, Fangxin Liu, Yongbiao Chen

Abstract: Speculative decoding mitigates the latency of sequential generation in autoregressive Large Language Models (LLMs) by interleaving draft generation with target verification. However, existing parallel drafting backends often suffer from rapid accuracy degradation over long horizons, leading to high rejection rates during verification and suboptimal wall-clock speedups. We observe that drafting errors are not uniformly distributed but typically stem from localized high-uncertainty tokens that destabilize downstream generation trajectories. Motivated by this token error pattern, we propose CURE, a budget-aware dynamic repair tree designed to repair errors at uncertainty focal points without incurring prohibitive tree-verification overheads. Specifically, our method uses predictive confidence margins to dynamically locate candidate error tokens within a block-parallel draft, expands bounded repair paths only at these fragile nodes, and employs a novel repair resynchronization mechanism to realign draft states post-verification. Evaluations on code-generation benchmarks (HumanEval, MBPP, and LiveCodeBench-lite) and mathematical reasoning benchmark (GSM8K) demonstrate that CURE increases the average accepted length by 4.2-7.5% over parallel baselines without repair, translating to an end-to-end speedup of $2.66-3.49\times$ over target-only decoding. Furthermore, we provide a plug-and-play repair module compatible with standard parallel drafting frameworks. We also characterize the trade-off between draft compute and verification efficiency.

Comment: Repairs localized uncertain draft tokens with bounded branches and resynchronizes speculative-decoding states.

Topic Match: The method introduces a new speculative-decoding mechanism that improves accepted length and wall-clock efficiency.

Relevance: 8 Novelty: 7

10. Understanding Rate-Distortion Performance in Distributed Transformer Inference

ArXiv ID: 2601.22002

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Anderson de Andrade, Alon Harell, Ivan V. BajiÄ

Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn, requires compressing its intermediate representations. We study compressibility of transformer's intermediate representations via learned compression through the lens of rate-distortion and the theory of usable information. Our study reveals that, unlike convolutional models, deeper representations in transformers become more difficult to compress. The reason for this behavior is twofold: first, the complexity of representation increases as we move deeper into the transformer, necessitating a higher rate; and second, the higher complexity of representations worsens the generaliztion bound for learned entropy estimates, which further compromises compression performance. Through a combination of experiments and theory, we characterize and analyze the compressibility of transformer representations, derive bounds on the achievable rate of learned codecs applied to these representations, and offer a unified lens for understanding rate-distortion performance in representation coding.

Comment: Rate-distortion theory characterizes learned compression of partitioned Transformer activations.

Topic Match: Intermediate-representation compression directly targets the communication and memory cost of distributed Transformer inference.

Relevance: 8 Novelty: 7

11. Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization

ArXiv ID: 2603.00910

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: MoE Training

Authors: Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra, Ankur Mali

Abstract: Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant. Existing layer-scoring methods provide sensitivity estimates but do not give a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget. We introduce a curvature-aware, MDL-inspired framework built around the layer gain $Î¶k^2=g_k^\top\widetilde Hg_k$. This quantity equals twice the maximal decrease predicted by the regularized layer-restricted quadratic model and incorporates inverse local curvature; it is therefore a local surrogate for reducible risk, not a universal dominance claim over gradient-norm scores. After normalizing the gains into scores $q_k$, we formulate two convex programs: one allocates expert slots under diminishing returns, and the other assigns layer-wise pruning ratios while protecting high-score layers. Both continuous programs have unique globally optimal solutions characterized by one dual variable and computable in $O(K\log(1/\varepsilon))$ time by bisection. We also prove a quadratic transfer-regret bound: when source and target score vectors differ by at most $Î´$, the target surrogate cost of the transferred decision is within $O(Î´^2)$ of the target optimum. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in some settings and competitive, though mixed, pruning performance. The framework therefore replaces an empirical score-to-decision heuristic with a budget-feasible optimization procedure whose guarantees apply to the stated continuous surrogates. Code is available on github repo - }^{-1TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation

Comment: Curvature-derived layer gains allocate expert slots and pruning ratios under a hardware budget.

Topic Match: Budget-aware capacity allocation and pruning are central, with expert-slot allocation providing a direct secondary MoE connection.

Relevance: 8 Novelty: 7

12. OnlineCache: Learning Dynamic Caching Policies with Error Correction for Efficient Diffusion Inference

ArXiv ID: 2607.29398

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Zhikang Xie, Xichen Ye, Yifan Wu, Haoshen Yu, Li chenan, Peizhu Gong, Weizhong Zhang, Cheng Jin

Abstract: Diffusion models have revolutionized generative tasks but incur high latency due to iterative denoising. While cache-based strategies accelerate inference by reusing intermediate features, they largely rely on static, sample-agnostic schedules. We argue that this rigidity overlooks two facts empirically validated in this paper: (i) generation difficulty varies across prompts, requiring adaptive resource allocation--complex inputs demand more computation while simpler ones require less; (ii) error sensitivity fluctuates across timesteps, where static policies may cache high-error steps or waste computation on low-error ones. We therefore propose OnlineCache, a dynamic caching framework that jointly learns when to cache and how to correct approximation errors. We leverage policy gradient to train a lightweight network for adaptive speed-quality trade-offs, and incorporate a learnable corrector to mitigate caching-induced errors. Both modules are jointly optimized under a bilevel optimization framework, with the policy targeting global generation quality and the corrector minimizing local errors. Our method automatically allocates computational resources across both samples and timesteps, improving overall generation quality. Extensive experiments demonstrate clear superiority. On FLUX.1-dev model, OnlineCache achieves nearly 3 speedup while preserving generation fidelity. On DiT and CogVideoX, it similarly delivers competitive acceleration without compromising quality; across all scenarios, it consistently outperforms existing cache-based acceleration baselines.

Comment: Learned per-sample, per-timestep caching jointly corrects approximation errors in diffusion inference.

Topic Match: Adaptive cache reuse is the primary efficiency mechanism, while input-dependent computation supplies a secondary architectural match.

Relevance: 8 Novelty: 7

13. Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates

ArXiv ID: 2607.28959

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Weiyi He, Yuping Lin, Jiliang Tang, Yue Xing

Abstract: Adversarial training is one of the most effective defenses against adversarial attacks, yet the computational cost remains prohibitive at modern scales, especially for large language models (LLMs). While existing mitigation strategies, e.g., latent adversarial training (LAT), have been developed, they still incur a high computational cost. In this work, we comprehensively investigate computation-efficient strategies to speed up LAT from two complementary perspectives: (1) Defense-side optimization: We explore the representation fine-tuning (ReFT) within LAT, and reveal a potential issue if there is a mismatch on which tokens to apply ReFT and the attack. (2) Attack-side optimization: When computing adversarial attacks in each LAT iteration, we extract only the relevant circuits from the LLM to construct a lightweight surrogate model, avoiding the computation in the forward-backward passes through the full model during the attack generation. For both perspectives, we provide theoretical justifications and numerical evidence to illustrate the effectiveness of the proposed strategies. Ultimately, compared to standard LAT with full fine-tuning, our method on average reduces per-step adversarial-training FLOPs by 48.1% while requiring only 0.0118% trainable parameters.

Comment: Low-rank defenses and circuit-extracted attack surrogates nearly halve adversarial-training FLOPs.

Topic Match: The core result materially reduces LLM training computation and trainable parameters through reusable low-rank and surrogate mechanisms.

Relevance: 8 Novelty: 7

14. CORA-Diff: Confidence-Oriented Residual Acceptance for Efficient Diffusion Language Model Inference

ArXiv ID: 2608.11235

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Yifan Wu, Yufeng Zhang, Kenli Li

Abstract: Diffusion language models (DLMs) update many tokens in parallel, yet practical decoders often use a fixed denoising horizon. Many predictions stabilize early, but blockwise decoding continues until all positions are resolved, causing repeated dense forward passes. Existing accelerators often rely on learned filters, modified scores, dependency models, or cache-specific mechanisms. We ask whether native trajectory signals can identify residual positions likely to match the deterministic dense endpoint. We propose CORA-Diff, a training-free method that preserves the original transfer rule and applies confidence-and-persistence gating only to positions that rule leaves unresolved. Accepted tokens remain visible as context, and the block terminates once all positions are resolved. This requires no backbone change, learned acceptance model, or logit modification. Our theory explains why high-confidence, persistent predictions are more likely to match the fixed-horizon dense endpoint, and paired post-intervention trajectories provide direct empirical support. We select one operating point on a separate GSM8K calibration subset and freeze it for all evaluations. Under a matched Learn2PD-style LLaDA protocol, CORA-Diff has the lowest measured runtime in all eight task-length settings. Task scores match or exceed dense decoding in five settings, and the largest observed drop is 1.22 points. Its incremental speedups over EOS-aware dense decoding are 2.70x and 3.32x on GSM8K and HumanEval. It also reaches 13.14x under the fixed-horizon 1024/1024 mechanism-isolation protocol and transfers to Dream without retuning at 3.18x-3.53x. These results show that native confidence and persistence enable reliable residual acceptance, reducing repeated denoising computation while preserving task quality.

Comment: Uses confidence-and-persistence gating to terminate diffusion decoding without learned filters or backbone changes.

Topic Match: The method introduces dynamic computation that substantially reduces repeated denoising passes.

Relevance: 8 Novelty: 7

15. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

ArXiv ID: 2607.09757

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jiaqi Liu, Haidong Kang, Qihui Zhao, Guo Yu, Jingchao Wang

Abstract: Parameter-efficient fine-tuning enables large language models to adapt to downstream tasks with substantially lower computational and storage cost, and Low-Rank Adaptation (LoRA) is among its most widely used techniques. However, vanilla LoRA assigns a uniform rank to all adapted modules, while existing adaptive methods either incur additional optimization overhead or rely on static weights and local gradients that do not capture task-conditioned representation changes. We propose RSRA, a training-free rank allocator that estimates where adaptation capacity is most needed through forward-only representation sensitivity probing on a small calibration set. Specifically, RSRA uses Spectral Effective Rank to allocate capacity across layers, measures module-wise hidden-state displacement under standardized virtual low-rank updates with the Frechet Distance, and combines both signals through hierarchical normalization to produce a task-aware rank configuration before fine-tuning. Across commonsense reasoning and natural language understanding benchmarks with Qwen3-4B and Mistral-7B, RSRA achieves the highest average performance in all three reported model-benchmark settings and a 1.48x-1.93x speedup in allocation time over the fastest competing pre-allocation method. When integrated with DoRA, LoRA-FA, and PiSSA, RSRA improves 15 of the 18 evaluated combinations and increases the average performance of all three PEFT methods.

Comment: Allocates LoRA ranks using forward-only spectral rank and representation-displacement probes.

Topic Match: The core contribution improves parameter-efficient adaptation by assigning low-rank capacity before training.

Relevance: 8 Novelty: 7

16. TopoTuner: Topological Finetuning of Large Language Models

ArXiv ID: 2607.16637

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad, Baha Rababah, Santu Karmaker, Cuneyt G. Akcora, Mubarak Shah

Abstract: Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. \method treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and transfers it to efficiently fine-tune models on out-of-domain datasets, evaluating whether task-specific topological drift generalizes across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while training only 1-2\% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57\% of the projection parameters. Along with minimized updates, TopoTuner reduces training time by 20.4\% relative to full fine-tuning and 5.5\% relative to LoRA on average. TopoTuner opens a new direction for reusable freezing profiles, where fine-tuning behavior learned on one dataset can be shared across multiple tasks.

Comment: Reusable topology-guided freezing profiles reduce attention-projection updates during adaptation.

Topic Match: Selective parameter training and reported training-time reductions make efficient adaptation the central contribution.

Relevance: 8 Novelty: 7

17. Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers

ArXiv ID: 2601.22876

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Zhanglu Yan, Kaiwen Tang, Zixuan Zhu, Zhenyu Bai, Qianhui Liu, Yongxin Zhu, Weng-Fai Wong

Abstract: Spiking neural networks (SNNs) promise energy-efficient inference for large language models (LLMs), yet most reported savings rely on compute-operation counts that overlook data movement. Energy characterization of representative spiking transformers on a commercial 22-nm process shows that accumulation contributes less than 3% of total energy, while spike-triggered inter-core transfers and weight reads dominate the cost. This makes time-to-first-spike (TTFS) encoding a natural choice, as it limits each neuron to at most one spike. However, standard TTFS maps the silent state, an all-zero spike train that transmits no events, to the rarely occurring smallest value, while the most common activations still spike. This raises a simple question: why reserve the only cost-free codeword for a rare value? This choice inverts a basic principle of energy-aware coding, under which the zero-event codeword should represent the most common value, rather than a rare extreme. Thus, we introduce masked time-to-first-spike encoding (M-TTFS), which uses a temporal mask to reassign the silent state to the most common activation value, and a dead-zone extension that trades a controlled amount of information for greater sparsity. Built on M-TTFS with dead-zone radius $k{=}1$, our spiking transformer Matterhorn reaches an overall spike rate of 1.64% on GLUE at an average score of 84.64, exceeding the best prior spiking transformer by 1.42 percentage points while consuming 67% less energy, with consistent gains on spiking LLaMA models from 7B to 70B parameters. Together, these results show that under hardware-faithful accounting, the energy advantage of SNNs is not a given: it is earned by encodings that align spikes with the data distribution.

Comment: Mapping common activations to the silent spike code reduces spike-triggered data movement and energy.

Topic Match: The core contribution is energy-efficient activation sparsity for spiking transformers, supported by a new temporal encoding mechanism.

Relevance: 8 Novelty: 7

18. BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks

ArXiv ID: 2608.01490

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Kazi Ahmed Asif Fuad, Lizhong Chen

Abstract: Binarizing a polynomial Kolmogorov--Arnold Network (KAN) not only changes parameter precision, but also alters the function space available to each layer. When activations are restricted to ${-1,+1}$, all even powers reduce to $1$ and all odd powers reduce to $x$, causing the elementwise polynomial basis to collapse to constant and first-order responses. We refer to this structural failure as Spatial Orthogonality Collapse. Our proposed BiKAN addresses this critical issue by augmenting each binary KAN layer with selected degree-2 Walsh characters. Fixed circular channel rolls generate pairwise parities, and learned binary projections mix them using the same XNOR--popcount operations as the remaining W1A1 paths. This restores explicit pairwise coordinates without learned routing or multiplier-based feature generation. Experiments on CIFAR-10 confirms that removing parity reduces accuracy by $1.23$ points over five paired seeds ($p=0.003$), the gain increases as width decreases, and accuracy improves monotonically as more parity planes are added. At an equal $\sim$11.9M-parameter budget, parity outperforms conventional widening by $3.09$ points ($p<10^{-4}$). At W1A1, BiKAN reaches $99.48\%$, $84.38\%$, and $55.81\%$ on MNIST, CIFAR-10, and CIFAR-100, respectively. Post-route Zynq-7020 FPGA results show that the repair remains hardware-efficient; the convolutional design cuts DSP usage from 164 to 72 and estimated compute-core latency from 401 to 54.8 ms, while the power-of-two-aware dense design achieves zero-DSP inference with a 0.03-point accuracy loss. The BiKAN implementation is available at https://github.com/OSU-STARLAB/BiKAN.

Comment: Restores basis expressivity in binary KANs with hardware-efficient pairwise Walsh features.

Topic Match: It couples extreme quantization with an architectural mechanism that repairs quantization-induced collapse.

Relevance: 7 Novelty: 8

19. ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts

ArXiv ID: 2607.17074

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa, Partha Pratim Pande

Abstract: Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks. These encompass the massive memory bandwidth required to fetch non-contiguous expert weights, the non-deterministic scatter-gather traffic generated by input-dependent token routing, and the tail-latency dependency imposed by synchronous expert output aggregation. To address these challenges, we propose ThAME, a three-dimensional (3D) heterogeneous multi-chiplet architecture for MoE inference. ThAME employs Ferroelectric Field-Effect Transistor (FeFET)-based non-volatile and DRAM-based volatile memory chiplets with a co-designed compute mapping strategy that aligns the distinct computational profiles of attention mechanisms and expert routing. Furthermore, we design a specialized Network-on-Chip communication backbone optimized to mitigate the bottlenecks associated with non-deterministic token routing traffic across the combinatorial space of input-dependent MoE traffic patterns. Experimental results demonstrate that ThAME outperforms state-of-the-art counterparts by up to 15.7x in terms of speedup and improves energy efficiency by up to 9.8x.

Comment: Co-designs 3D memory chiplets and routing-aware interconnects for irregular MoE inference traffic.

Topic Match: The paper materially changes MoE execution cost through a specialized inference accelerator rather than a training method.

Relevance: 7 Novelty: 8

20. Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

ArXiv ID: 2608.00129

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tiancong Cheng, Ying Zhang, Zhiwen Yu, Yifang Yin, Bin Guo

Abstract: Knowledge distillation (KD) is a widely utilized technique for transferring knowledge from a large model (the teacher) to a smaller model (the student). Owing to its flexibility and broad applicability, KD has been extensively applied in the compression of server-side models to meet the Quality of Service (QoS) requirements of client users. Despite significant advancements, the performance of distillation is substantially compromised when a large disparity exists between the capabilities of the server and the requirements of the client. To alleviate this problem, we propose a novel distillation approach, named Progressive$^2$, which operates through the combination of a progressively stronger teacher and a progressively smaller student. On the side of the teacher, rather than involving all layers simultaneously, we progressively select additional layers for distillation following a raw-to-rich semantic progression, establishing a systematic learning curriculum. Furthermore, we design a teacher-side multi-feature fusion adapter for the teacher to improve training stability, which is theoretically supported by the framework of Lipschitz continuity. On the side of the student, rather than directly training a tiny model, we gradually reduce the size of the network to facilitate an iterative co-evolution with the teacher. Progressive$^2$ serves as a flexible framework; the progressive strategy of the teacher can be deployed independently to achieve an optimal balance between accuracy and training efficiency, while the joint integration of the teacher and the student yields further improvements in overall performance.

Comment: Co-evolving teacher curricula and progressively shrinking students improve distillation-based compression.

Topic Match: The central contribution is a knowledge-distillation framework for compressing a large teacher into a substantially smaller student.

Relevance: 8 Novelty: 6

21. Damage-Aware Bandit Pruning for Vision and Language Transformers

ArXiv ID: 2609.05448

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Salem Ameen, Sunil Vadera

Abstract: Structured post-training pruning of transformers requires selecting complete functional units whose suppression causes limited degradation. We formulate structured-unit selection for language and vision transformers as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget. Attention heads and MLP channel groups are temporarily masked on calibration batches. Paired damage is the masked loss minus the base loss on the same batch, reducing batch-to-batch variation. A smooth bounded reward drives either a UCB-style policy or fractional-Beta Thompson Sampling, and the final mask is constructed sequentially by adding one unit at each step. The selected units are functionally zeroed in the original dense checkpoint; therefore, the reported parameter effects represent effective structural suppression rather than physical compression or measured speedup. Experiments on WikiText-2, LAMBADA, and Imagenette cover GPT-2, OPT, Pythia, Qwen2.5, SmolLM2, ViT-B/16, DeiT-Tiny, and Swin-Tiny, with comparisons against random, magnitude, static-saliency, and budgeted-greedy selection. Across five seeds, the bandit methods usually reduce degradation relative to budgeted greedy in the paired language-model comparisons. Of 28 comparisons highlighted in the paper, 23 bootstrap confidence intervals exclude zero and 11 paired tests have p < 0.05; six have q < 0.05 after Benjamini-Hochberg correction across the full family of 116 dataset-wise tests. Matched-evaluation results for ViT-B/16 and Swin-Tiny indicate that their gains are not explained solely by a larger candidate-evaluation budget.

Comment: Bandit allocation selects attention heads and MLP groups for pruning under a fixed evaluation budget.

Topic Match: Budgeted structured-pruning selection directly fits compression, although masking dense checkpoints leaves actual memory savings and execution speedups unverified.

Relevance: 8 Novelty: 6

22. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

ArXiv ID: 2607.25018

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Yifan Dou, Shikan Lian, Shibo Li

Abstract: Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one. Production cascades govern this deferral through a confidence threshold, but LLM confidence scores are miscalibrated, the threshold must be tuned per model pair and per domain, and no setting yields a formal bound on cascade accuracy. We introduce \textbf{Conformal Cascade} (CC), a multi-tier inference framework that uses conformal prediction set size as the deferral rule: accept when the calibrated set collapses to a single answer, defer otherwise. The procedure delivers a distribution-free, finite-sample accuracy guarantee. By a per-tier union bound, the prediction set at the accepting tier covers the correct answer with probability at least $1 - KÎ±$ for any user-specified $Î±$; under a selection-preservation condition (consistent with, but not strictly implied by, our marginal coverage results), the bound tightens to $1 - Î±$. We further characterise expected cascade cost as an explicit function of $Î±$ and the calibration-set acceptance rate. Across 18 multiple-choice benchmarks spanning science, medicine, commonsense, and standardized exams, evaluated on two-tier cascades drawn from four open-weight model families, CC strictly improves over the strongest calibration-tuned heuristic cascade on the majority of family--benchmark pairs, with the largest gains on reasoning-heavy benchmarks where majority vote is unreliable; on easier benchmarks the cascade commits the vast majority of queries to the small model at no accuracy cost. Extension to open-ended generation requires an answer-clustering step that we leave for future work. The method requires no model training and only black-box API access.

Comment: Uses conformal prediction-set size to route queries across model tiers with finite-sample accuracy guarantees.

Topic Match: The core mechanism reduces LLM inference cost through calibrated adaptive model selection.

Relevance: 7 Novelty: 7

23. Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals

ArXiv ID: 2504.14047

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou

Abstract: There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. While breakthroughs like DeepSeek-R1 highlight the power of reinforcement learning for reasoning, the interaction between ITC and reasoning-optimized weights remains poorly understood. This work conducts a comprehensive analysis of inference-time scaling methods for both reasoning and non-reasoning models on challenging reasoning tasks. While prior work suggests that scaling test-time compute can optimally substitute for model parameter scaling, we identify a fundamental limit to this compute-equivalence, the reasoning floor, a performance plateau that non-reasoning models cannot escape, no matter how much inference compute is spent. We demonstrate that general-purpose models fail to match the accuracy of reasoning-optimized models even with an order of magnitude more inference compute, suggesting that internalizing reasoning protocols is a prerequisite for effective test-time scaling. Within reasoning models, we find that the complexity of the scaling method often yields diminishing returns; simple majority voting consistently outperforms sophisticated sequential revision and mixture-of-agents frameworks. Crucially, we identify a Linguistic Signal of Correctness - correct responses are significantly more concise and exhibit a lower density of hedging and thinking markers. We demonstrate that these intrinsic linguistic features can serve as zero-compute proxies for response quality, providing a pathway to more efficient, self-diagnostic reasoning agents.

Comment: Identifies a reasoning floor that limits substitution of inference compute for reasoning-optimized training.

Topic Match: It establishes practical limits and selection signals for configuring inference-time compute.

Relevance: 7 Novelty: 7

24. LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

ArXiv ID: 2603.13319

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, Zhijie Deng

Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising paradigm for parallel token generation, with block-wise variants garnering significant research interest. Despite their potential, existing dLLMs typically suffer from a rigid accuracy-parallelism trade-off: increasing the number of tokens per forward (TPF) via aggressive parallel decoding often leads to performance degradation and increased generation instability. We identify that this limitation stems from the model's inability to navigate high-parallelism regimes where approximation errors and local corruptions accumulate, ultimately undermining the reliability of parallel generation. To address this, we propose LightningRL, a post-training framework designed to directly optimize the speed-quality Pareto frontier of pre-trained dLLMs. Instead of forcing uniform parallelization, our approach leverages reinforcement learning to identify and reinforce high-parallelism trajectories that maintain generation accuracy. Built upon the Group Relative Policy Optimization (GRPO) framework, LightningRL introduces several enhancements tailored for dLLMs: (1) stabilized training via per-reward decoupled normalization; (2) token-level negative log-likelihood (NLL) regularization on correct trajectories to anchor model performance; and (3) a dynamic sampling strategy with TPF-aware filtering to enhance training efficiency. Experimental results across mathematical and coding benchmarks demonstrate that LightningRL consistently advances the Pareto frontier, achieving competitive task accuracy while significantly increasing parallelism, reaching an average TPF of 7.32 (with a peak of 11.10 on the MBPP dataset). Our code is available at https://github.com/SJTU-DENG-Lab/LightningRL.

Comment: RL-trained trajectory selection raises tokens per forward in block-wise diffusion-LLM decoding.

Topic Match: Its main effect is lower-cost parallel generation, supported by training changes that stabilize high-parallelism computation.

Relevance: 7 Novelty: 7

25. GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

ArXiv ID: 2601.09361

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models. Unlike supervised fine-tuning (SFT), RLVR exhibits distinct optimization dynamics and is sensitive to the preservation of pre-trained geometric structures. However, existing parameter-efficient methods face key limitations in this regime. Low-rank adaptation methods, such as PiSSA, are primarily designed for Supervised Fine-Tuning (SFT) and do not account for the distinct optimization dynamics and geometric structures of RLVR. Conversely, directly fine-tuning the unstructured sparse parameter subspace favored by RLVR encounters efficiency bottlenecks on modern hardware. To address these challenges, we propose GeoRA (Geometry-Aware Low-Rank Adaptation), a low-rank adaptation method tailored for RLVR. Specifically, GeoRA exploits the anisotropic and compressible structure of RL update subspace, and extracts its principal directions via Singular Value Decomposition (SVD) to initialize low-rank adapters, while freezing residual components as a structural anchor during training. This design preserves the pre-trained structure and enables efficient dense computation. Experiments on Qwen and Llama models from 1.5B to 32B parameters show that GeoRA consistently outperforms strong low-rank baselines across RLVR settings in mathematics, medicine, and coding, while showing stronger generalization and less forgetting on out-of-domain tasks.

Comment: SVD-aligned low-rank adapters preserve RL update geometry while reducing trainable capacity.

Topic Match: The main contribution is a geometry-aware low-rank adaptation mechanism, with supporting insight into RL training dynamics.

Relevance: 7 Novelty: 7

26. SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits

ArXiv ID: 2608.00859

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Kazi Ahmed Asif Fuad, Lizhong Chen

Abstract: Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions parameterized by multiple basis coefficients. This introduces a source of redundancy that conventional neural-network compression does not directly expose. We present \textbf{SparseKAN}, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision. SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective. The learned importance structure is subsequently hardened under explicit basis and width budgets, recovered in full or low precision, and physically compacted into smaller dense tensors rather than retained as sparse masks. Experiments on MNIST, CIFAR-10, and CIFAR-100 across spline, polynomial, RBF, wavelet, and convolutional KAN variants show that the structural axes compose predictably in cost. We also find strong basis-dependent differences in term importance: coefficient-based selection outperforms matched low-order truncation by up to 15.25 accuracy points in the evaluated Gram-polynomial settings. Eight-bit quantization is broadly robust, whereas 4-bit convolutional KANs require quantization-aware adaptation. Physical compaction removes up to 73.0\% of parameters without accuracy loss on MNIST and reduces large-batch CUDA latency to as little as $0.51\times$ dense execution. On a ZCU104 FPGA, the resulting sparse low-bit models achieve up to $23.63\times$ lower inference latency, demonstrating that SparseKAN converts functional redundancy into measurable software and hardware efficiency. The SparseKAN implementation is available at https://github.com/OSU-STARLAB/SparseKAN.

Comment: Jointly gates KAN basis terms, neurons, and precision before physically compacting the learned structure.

Topic Match: The core contribution is a unified pruning and quantization mechanism with measured hardware savings.

Relevance: 7 Novelty: 7

27. Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere

ArXiv ID: 2608.01271

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jiayang He, Tianling Xu, Diancheng Kang, Huaide Jiang, Junyan Bai, Shaoming Zheng, Xuan Song

Abstract: Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input. These tokens often contain substantial redundancy arising from repeated visual patterns, leading to unnecessary computation in the subsequent language-model processing. Existing token compression methods, including pruning and merging, perform compression online during inference, repeatedly incurring additional computation for each input video and often relying on model-specific designs that limit their generality, we instead rethink this paradigm by shifting the costly compression process offline. We propose \textbf{ONCE}, a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs. Extensive experiments across multiple video understanding benchmarks and against diverse compression baselines demonstrate that our approach achieves a strong accuracy-efficiency trade-off, maintaining competitive performance while achieving the lowest inference latency among compared methods.

Comment: Moves video-token compression offline by learning a reusable global feature codebook.

Topic Match: Its core contribution is a general token-compression mechanism that lowers online model computation.

Relevance: 7 Novelty: 7

28. Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

ArXiv ID: 2607.29071

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Large-Scale Training Systems and Efficiency

Authors: Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao, Mayi Xu, Quanqing Xu, Wei Ren, Qiang Yang, Yang Liu

Abstract: Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

Comment: Nested low-rank client models enable structurally aligned aggregation when federated participants have heterogeneous memory budgets.

Topic Match: Efficiency scaling is primary because SVD-compressed client models make foundation-model adaptation feasible under severe memory constraints.

Relevance: 7 Novelty: 7

29. Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

ArXiv ID: 2608.13596

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie

Abstract: Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

Comment: Activation-guided donor pruning enables capability transfer into a smaller recipient without additional training.

Topic Match: Pruning and compact capability transfer connect the method to compression; the demonstrated benefit is improved quality at the recipient's existing model size.

Relevance: 7 Novelty: 7

30. Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View

ArXiv ID: 2608.01357

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tong Mao, Jinchao Xu

Abstract: Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom. However, practical computation operates under finite precision: parameters must be encoded using a finite number of bits. Therefore, approximation efficiency should be evaluated in terms of computational bit complexity, which is intrinsically connected to the metric entropy of the underlying function class. In this work, we develop a unified approximation framework based on binary encoding and metric entropy. We analyze classical methods (including polynomial approximation, sparse grids, and finite elements) as well as shallow and deep neural networks, and compare their approximation rates for function classes with comparable metric entropy. We observe that, when evaluated in terms of bits, most classical methods are in general suboptimal relative to the intrinsic limits dictated by metric entropy, while neural network methods may exhibit different behaviors. We show that when complexity is measured in bits rather than parameters, no method fundamentally exceeds the approximation order achieved by classical approaches. Our results also indicate that many seeming advantages of neural networks, including dimension-independent rates and superconvergence phenomena, stem from differences in function class complexity rather than intrinsic architectural superiority. In this sense, the traditional curse of dimensionality can be misleading; the fundamental limitation is instead a curse of bit complexity, governed by metric entropy.

Comment: Reframes neural approximation efficiency through finite-precision bit complexity and metric entropy.

Topic Match: The bit-complexity analysis informs how compression and model-size efficiency should be measured.

Relevance: 6 Novelty: 8

31. Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

ArXiv ID: 2608.00264

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Kamil KsiÄ Å¼ek, Piotr SuszyÅski, MichaÅ Jan WÅodarczyk, Jacek Tabor, PrzemysÅaw Biecek

Abstract: Vision foundation models, such as DINOv2, learn highly expressive representations but rely on massive, opaque architectures that demand substantial computational power and memory. To provide an interpretable-guided and efficient solution to this issue, we first propose a spectral analysis and new visualization technique for individual attention heads based on the Laplacian eigenvectors of their attention maps. Building upon recent observations regarding the block structure of Vision Transformers, we perform semantic clustering of attention heads and identify functional redundancies. Leveraging these insights, we introduce SAPER (Soft Attention PrunER), an end-to-end differentiable pruning framework based on the LapSum Soft Top-K approach. Extensive experiments on ImageNet-1K demonstrate that SAPER achieves a highly favorable accuracy-efficiency trade-off, outperforming the competitive RAPTOR baseline in FLOPs reduction while preserving strong classification performance.

Comment: Differentiable soft top-k head pruning removes redundant Vision Transformer attention computation.

Topic Match: Its central contribution is structured attention-head pruning that directly reduces foundation-model FLOPs.

Relevance: 7 Novelty: 6

32. Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs

ArXiv ID: 2605.04215

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Michael Rottoli, Subhankar Roy, Stefano Paraboschi

Abstract: Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead to significant throughput advantages and superior GPU utilization over the traditional autoregressive paradigm. However, this parallelism is constrained by the requirement of a fixed-size response length prior to generation. This architectural limitation imposes a severe trade-off: oversized response length results in computational waste on semantically meaningless padding tokens, while undersized response length causes output truncation requiring costly re-computations that introduce unpredictable latency spikes. To tackle this issue, we propose Predict-then-Diffuse, a simple and model-agnostic framework that enables compute-budgeted inference per input query by first estimating the response length and then using it to run inference with D-LLM. At its core lies an Adaptive Response Length Predictor (AdaRLP), which estimates the optimal response length given an input query. As a measure against under-estimating the response length and re-running inference with a higher value, we introduce a data-driven safety mechanism based on a small increase of the predicted length. As a whole, our framework avoids wasting computation on padding tokens, at the same time preserving output quality. Experimental validation on multiple datasets demonstrates that Predict-then-Diffuse significantly reduces computational costs (FLOP) compared to the default D-LLM inference mechanism, while being robust to skewed data distributions.

Comment: Predicts response length before diffusion decoding to avoid padding compute and costly reruns.

Topic Match: It directly reduces diffusion-LLM inference FLOPs through adaptive compute allocation.

Relevance: 7 Novelty: 6

33. Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation

ArXiv ID: 2601.22546

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Shun Qian, Bingquan Liu, Chengjie Sun, Zhen Xu, Baoxun Wang

Abstract: The recent advancements in Large Language Models (LLMs) have attracted interest in exploring their in-context learning abilities and chain-of-thought capabilities. However, there are few studies investigating the specific traits related to the powerful generation capacity of LLMs. This paper aims to delve into the generation characteristics exhibited by LLMs. Through our investigation, we have discovered that language models tend to capture target-side keywords at the beginning of the generation process. We name this phenomenon the Holographic Characteristic of language models. For the purpose of exploring this characteristic and further improving the inference efficiency of language models, we propose a plugin called HOLO, which leverages the Holographic Characteristic to extract target-side keywords from language models within a limited number of generation steps and complements the sentence with a parallel lexically constrained text generation method. To verify the effectiveness of HOLO, we conduct massive experiments on language models of varying architectures and scales in the short-text generation scenario. The results demonstrate that HOLO achieves comparable performance to the baselines in terms of both automatic and human-like evaluation metrics and highlight the potential of the Holographic Characteristic.

Comment: Early keyword extraction enables parallel lexically constrained text completion.

Topic Match: The decoding mechanism targets inference cost; evaluation is limited to short texts, and the abstract does not quantify a speedup.

Relevance: 7 Novelty: 6

34. Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalization

ArXiv ID: 2608.12389

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Xuefei Wang, Jun Han, Zixuan Wang, Qingkai Zeng, Xiao Wang, Ruijie Wang, Jianxin Li

Abstract: Cross-domain zero- or few-shot personalization aims to generate user-preferred responses in unseen conversational domains from only a handful of target-domain interactions. Existing adaptation methods struggle to calibrate update magnitude under sparse evidence and thus overfit, whereas history-transfer methods often entangle user preferences with source-domain artifacts, yielding unreliable personalization priors and negative transfer. To calibrate adaptation to evidence quality, we propose PAC-Bayes-regularized Meta-LoRA, which uses a meta-learned LoRA initialization as both the adaptation start and prior center, while adjusting update strength according to support-set size and predictive uncertainty. This limits overfitting under sparse or ambiguous evidence while permitting stronger personalization as evidence grows. Controlled adaptation alone does not determine which preferences should transfer across domains or how they should be expressed. We therefore functionally decompose personalization priors into user and domain components, using a human-readable prompt for stable preferences and topology-preserving soft tokens for domain-specific hidden-space conditioning. Experiments across multiple benchmarks and personalization tasks show consistent gains over strong baselines. On HiCUPID, our method reduces cross-domain win-rate degradation by 47.9% relative to the best competing baseline and improves win rate by 110.2% under unseen-user cold start.

Comment: PAC-Bayes-regularized Meta-LoRA calibrates low-rank updates under sparse personalization evidence.

Topic Match: The relevant mechanism is a new low-rank adaptation procedure, though its primary use case is cross-domain personalization.

Relevance: 6 Novelty: 7

35. In-situ Autoguidance: Eliciting Self-Correction in Diffusion Models

ArXiv ID: 2510.17136

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Enhao Gu, Haolin Hou

Abstract: The generation of high-quality, diverse, and prompt-aligned images is a central goal in image-generating diffusion models. The popular classifier-free guidance (CFG) approach improves quality and alignment at the cost of reduced variation, creating an inherent entanglement of these effects. Recent work has successfully disentangled these properties by guiding a model with a separately trained, inferior counterpart; however, this solution introduces the considerable overhead of requiring an auxiliary model. We challenge this prerequisite by introducing In-situ Autoguidance, a method that elicits guidance from the model itself without any auxiliary components. Our approach dynamically generates an inferior prediction on the fly using a stochastic forward pass, reframing guidance as a form of inference-time self-correction. We demonstrate that this zero-cost approach is not only viable but also establishes a powerful new baseline for cost-efficient guidance, proving that the benefits of self-guidance can be achieved without external models.

Comment: A stochastic self-pass replaces the separately trained guidance model in diffusion inference.

Topic Match: The strongest fit is inference efficiency because self-guidance removes the cost and storage of an auxiliary model.

Relevance: 6 Novelty: 7

36. Gram-Space: Structure-Preserving Codebook Compression for Memory-Efficient Neuro-Symbolic AI

ArXiv ID: 2608.01528

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Weilun Wang, Wantong Li

Abstract: Vector symbolic architectures (VSA) are widely used for reasoning in neuro-symbolic (NeSy) AI, yet high-dimensional codebooks often create severe memory bottlenecks that limit scalability and deployment. In this paper, we propose Gram-Space, a compression framework that applies Gram-Schmidt orthogonalization to represent codebook vectors in a compact orthonormal coordinate system. Gram-Space preserves the dot-product structure required by matrix-based VSA operators, which supports numerically equivalent execution of matrix similarity, probability vectorization, and attention score computations. We provide a correctness analysis showing that inner products are preserved under the orthonormal basis representation. Using modern GPU hardware, we benchmark the Gram-Space framework on standard neuro-symbolic reasoning datasets. Experimental evaluations across state-of-the-art VSA models show that Gram-Space reduces model-level GPU memory usage by up to 15.75x and improves inference latency by up to 3.62x. Profiling results further indicate that Gram-Space reduces allocation-heavy overhead in codebook-associated stages and improves hardware utilization for NeSy workloads.

Comment: Orthonormal Gram-space coordinates preserve dot products while shrinking symbolic codebook memory.

Topic Match: The paper's core contribution is a structure-preserving compression method with substantial memory and latency reductions.

Relevance: 6 Novelty: 7

37. AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference

ArXiv ID: 2604.15622

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michael Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li

Abstract: Always-on contextual AI runs language-aligned vision foundation models (VFMs) on edge devices, where the on-device model is the dominant continuous compute cost under strict latency and power limits. Due to an observed low-frequency shift in scene context and its relevant vocabulary, we present AdaDINO, an adaptive framework that makes on-device VFM inference efficient by matching execution to the current scene and task. We build on a known phenomenon, that the accuracy drop of shrinking model sizes depends on the task, and turn it into task-level adaptive execution. AdaDINO integrates neural architecture search (NAS) into a language-aligned VFM backbone distilled from DINOv2, training a single family of subnets for efficient execution during runtime. A multimodal large language model (LLM) on the cloud, invoked at low frequency, refines the candidate class set from scene context, while a learned selector activates the least-cost subnet predicted to retain a target fraction of accuracy. With the backbone and semantic pipeline held fixed, learned selection alone reduces average compute by $37\%$ over the best fixed subnet at equal segmentation accuracy. Across zero-shot classification and open-vocabulary segmentation, AdaDINO establishes a strong accuracy-efficiency frontier, improving over evaluated models of comparable sizes by up to $7.9\%$ in acc@1 on IN1K and $5.2\%$ mIoU on ADE20K, and reducing average FLOPs by up to $74.9\%$ at similar accuracy.

Comment: A trained family of vision subnets dynamically activates the least-cost architecture expected to satisfy an accuracy target.

Topic Match: Efficiency scaling is primary because context-conditioned subnet selection directly reduces average inference computation.

Relevance: 6 Novelty: 7

38. ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression

ArXiv ID: 2608.00345

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Renjie Liang, Zijian Xu, Jinqian Pan, Chengkun Sun, Zhengkang Fan, Shawn Li, You Qin, Mei Liu, Jie Xu

Abstract: A 3D CT scan entering a vision-language model produces a long sequence of visual tokens, often thousands to tens of thousands per volume, and this sequence must be compressed before a language model can consume it. Token compression is well studied in general vision, but little of it targets 3D CT specifically. A common baseline is grid average, which pools regular grid cells and can blend distinct anatomy, lesion, and air into one token. We present \textbf{ORCA} (ORgan-Centroid Aggregation), a token compressor for 3D CT. It merges adjacent tokens with organ guidance and adds a sinusoidal encoding of each region's centroid to preserve spatial layout. This preserves the anatomical information a downstream model needs. ORCA is training-free and plug-and-play, producing an adjustable token set without any model change or text query. We evaluate it across two datasets (CT-RATE and Merlin) and five encoders. The evaluation spans two task types: attribute prediction over five families (size, density, location, texture, and disease) and text generation (visual question answering and report generation). At matched token budgets, ORCA improves consistently over existing compression methods. It shrinks the visual context $64\times$ and its KV-cache $50\times$, and is $31\times$ faster to process each volume. Code released at https://github.com/renjie-liang/ORCA-3DCT.

Comment: Organ-guided adjacent-token merging delivers 64x visual-context and 50x KV-cache compression.

Topic Match: Its core mechanism compresses visual tokens and downstream cache usage, albeit specifically for 3D CT.

Relevance: 6 Novelty: 6

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.

7-8: substantially related, but partly peripheral or focused on a narrower aspect.

5-6: touches the target topics, but the main contribution is elsewhere.

3-4: largely outside the target topics, often application-focused or domain-specific.

1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

9-10: new paradigm, theory, or major methodological breakthrough.

7-8: substantial methodological advance or strong new insight.

5-6: meaningful but incremental extension or refinement.

3-4: minor, narrow, or mostly engineering or domain-specific improvement.

1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains