This is a remedial run for missed papers from 07/03/2026 to 07/05/2026.

Results generated on 09/12/2026.

Personalized Daily ArXiv Papers 2026-07-06

_{Token counts are not reported for this run. 7 of 14 model calls succeeded, 6,209s of model wall clock.}

Topic Coverage:

Table of contents by topic:

MoE Training (1)

1. Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control Authors: Jianjie Fang, Yongyan Xu, Ziyou Wang, Chen Gao, Yuchao Huang, Zhaolu Wang, Rongze Tang, Mingyuan Jia, Baining Zhao, Weichen Zhang, Xin Zhang, Haisheng Su, Yu Shang, Wei Wu, Xinlei Chen, Yong Li

Large-Scale Training Systems and Efficiency (5)

1. On the Convergence of Adam, Revisited Authors: Steven Heilman, Sampad Mohanty

2. PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark Authors: Ali Sultonov

3. FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity Authors: Shuai Li, Qinglin Wang, Ping Luo, Jiahuan Wang, Hongyang Hu, Haotian Mo, Yigui Feng, Ziang Liu, Qisong Xiao, Jie Liu, Tao Sun

4. Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities Authors: Arash Badie-Modiri, Chiara Boldrini, Lorenzo Valerio, János Kertész, Márton Karsai

5. FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filtering Authors: Liyang Yuan, Yibo Yang, Dandan Guo

Architecture and Training Dynamics (17)

1. Separating Representation from Reconstruction Enables Scalable Text Encoders Authors: Megi Dervishi, Mathurin Videau, Yann LeCun

2. Tensor-Train Joint Modeling for Few-Step Discrete Diffusion Authors: Byoungkwon Kim, Minhyuk Sung

3. SHiPPO: Recurrent Memory with Transported Polynomial Projections Authors: Tomoya Mizuguchi, Bum Jun Kim

4. Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning Authors: Chan Li, Nigel Goldenfeld

5. Structure-Specific Representational Priors Causally Control the Grokking Delay Authors: Gunner Levi Howe

6. The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning Authors: Joan Bruna

7. Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks Authors: Yoshihiro Maruyama

8. Tightening the Score Matching Gap for Diffusion Models Authors: Benjamin Dupuis, Tyler Farghly, Maxime Haddouche, Alain Durmus, Umut Simsekli

9. Reflected Schrödinger Bridge Matching Authors: Marcus Häggbom, Viktor Nilsson, Pierre Nyquist, Joakim andén

10. Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models Authors: Aryuemaan Kumar Chowdhury, Praveen Oosa, Vineesha Reddy

11. Transferability Between Understanding and Generation in Unified Multimodal Models Authors: Jiwon Kang, Heeji Yoon, Jaewoo Jung, Jaewon Min, Minkyeong Jeon, Biyeon Hwang, Sangwon Jung, Seungryong Kim

12. ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum Authors: Haiwen Yi, Xinyuan Song

13. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks Authors: Shokhrukh Ibragimov, Arnulf Jentzen

14. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse Authors: Shuang Liang, Tom Jacobs, Guido Montúfar

15. A Unified Framework for In-Context Learning with Causal and Masked Language Models Authors: Chenrui Liu, Chuanlong Xie, Falong Tan, Yicheng Zeng, Lixing Zhu

16. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch Authors: Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

17. Burst Spiking Neural Networks Authors: Jiahong Zhang, Sijun Shen, Man Yao, Han Xu, Mingqiang Huang, Yonghong Tian, Bo Xu, Guoqi Li

Efficiency, Compression, and Large-Scale Training (21)

1. LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression Authors: Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang, Ruiqun Li, Changqun Li

2. Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models Authors: Hamish Ogilvy

3. ELiTeFormer: An Efficient Transformer for FPGAs Authors: Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo

4. HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference Authors: Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou

5. Uncertainty-gated selection for block-sparse attention Authors: Thomas Rossi

6. SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation Authors: Omer Tariq, Syed Muhammad Raza, Jeongbae Son

7. HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion Authors: Dongyeun Lee, Amir Zandieh, Vahab Mirrokni, Junmo Kim, Insu Han

8. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation Authors: Jiaqi Liu, Haidong Kang, Qihui Zhao, Guo Yu, Jingchao Wang

9. RADIO1D: Elastic Representations for Condensed Vision Modeling Authors: Greg Heinrich, Mike Ranzinger, Collin McCarthy, Natan Bagrov, Eugene Khvedchenya, Bryan Catanzaro, Jan Kautz, Andrew Tao, Pavlo Molchanov

10. Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumption Authors: Nidhal Jegham, Boris Gamazaychikov, Sasha Luccioni

11. Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests Authors: Anis Hamadouche, Amir Hussain

12. ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning Authors: Wentao Lu

13. Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic Authors: Anis Hamadouche, Amir Hussain

14. Asymptotic Preservation and Uniform Accuracy of Diffusion and Flow-Matching Samplers Authors: Shiheng Zhang

15. Spectral Rewiring for Exploration, Purification, and Model Merging Authors: Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

16. Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective Authors: Zhenfeng Su, Kang Zhao, Han Bao, Tao Yuan, Zhongzhe Hu, Xianzhi Yu, Wenxuan Wang

17. A Unified Framework for Quantized and Continuous Strong Lottery Tickets Authors: Aakash Kumar, Emanuele Natale

18. Teacher Supervision over Representation Equivalence Classes Authors: Sang Il Han

19. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference Authors: Huajun Bai, Weiwei Lv, Huichuan Zheng, Youyou Lu, Jiwu Shu

20. Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation Authors: Wei Zhang, Lin Tang, Ming Zhao, Yuxuan Wang

21. OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models Authors: Shijie Cao, Qingyu Zhang, Boxi Yu, Yuzhong Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

MoE Training (1)

1. Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

ArXiv ID: 2607.03964

Primary Topic: MoE Training

Also Matches: Architecture and Training Dynamics

Authors: Jianjie Fang, Yongyan Xu, Ziyou Wang, Chen Gao, Yuchao Huang, Zhaolu Wang, Rongze Tang, Mingyuan Jia, Baining Zhao, Weichen Zhang, Xin Zhang, Haisheng Su, Yu Shang, Wei Wu, Xinlei Chen, Yong Li

Abstract: World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video generation world models are still organized around isolated control interfaces, such as camera trajectories, robot actions, or hand-joint signals. This fragmentation is increasingly a scaling bottleneck. The central challenge is not the absence of controllable generators, but the lack of a unified and extensible learning framework that can absorb heterogeneous action supervision while preserving a shared model of world dynamics. In this work, we introduce Worldscape-MoE, a Mixture-of-Experts world model built on Diffusion Transformers for scalable heterogeneous action control. Our key observation is that different controls specify different interfaces to the same underlying world: although their representations differ, they constrain shared physical regularities, scene dynamics, and interaction semantics. Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy that supports continual extension to new action modalities. Experiments across locomotion, robotic manipulation, and egocentric hand control show that heterogeneous supervision improves rather than interferes with individual control capabilities. Worldscape-MoE achieves strong results on WorldArena, improves locomotion and hand-control metrics, exhibits robust out-of-distribution generalization, and demonstrates scaling behavior as additional control data and experts are integrated.

Comment: Combines shared and control-specific experts with progressive MoE tuning for adding heterogeneous action modalities.

Topic Match: Expert sharing, specialization, and progressive expert expansion are substantive MoE design choices despite the world-model setting.

Relevance: 7 Novelty: 7

Large-Scale Training Systems and Efficiency (5)

1. On the Convergence of Adam, Revisited

ArXiv ID: 2607.03519

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Steven Heilman, Sampad Mohanty

Abstract: We show that projected Adam for online optimization with arbitrary moment decay parameters $β_1,β_2\in[0,1)$ can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required $β_1<\sqrt{β_2}$. Similar to their result, we use a three-periodic sequence of linear functions on $[-1,1]$ with slopes $c,-1,-1$, though we use $c$ slightly larger than $2$. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.

Comment: Non-vanishing-regret counterexamples are extended to Adam at all decay settings and several widely used variants.

Topic Match: The paper directly advances understanding of optimizer convergence behavior relevant to large-scale training.

Relevance: 8 Novelty: 7

2. PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

ArXiv ID: 2607.16268

Primary Topic: Large-Scale Training Systems and Efficiency

Also Matches: Architecture and Training Dynamics

Authors: Ali Sultonov

Abstract: Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence. We introduce PsiLogic, an optimizer that augments Adam with a dynamic Active Cancellation Term gated by a dual exponential moving average (EMA) of scale-normalized gradient norms. The resulting chaos detector strengthens damping when gradient statistics are unstable and fades to zero as training stabilizes, providing an implicit warmup without a hand-tuned schedule. We evaluate PsiLogic against Adam, AdamW, and Lion using FairBench -- a reproducible benchmark protocol with per-optimizer learning-rate sweeps, identical initialization per seed, and Welch t-tests. On an NVIDIA H100 80GB reference run (4 arenas, 3 seeds, 2000 steps, bf16 AMP), PsiLogic achieves the best validation metric in three of four arenas: NLP perplexity 7.79 +/- 0.18 vs. 8.17 +/- 0.08 (AdamW, p = 0.049), ViT top-1 accuracy 0.244 +/- 0.006 vs. 0.223 +/- 0.002 (AdamW, p = 0.015), and ResNet top-1 accuracy 0.222 +/- 0.001 vs. 0.172 +/- 0.004 (Adam, p = 0.001). On diffusion, validation MSE is statistically tied with Adam/AdamW (p = 0.49). ResNet accuracy vs. AdamW is a numerical tie without significance at three seeds (p = 0.44). Peak GPU memory is comparable across optimizers; PsiLogic incurs 1.2--1.8x wall-clock overhead on transformer-heavy arenas (implementation-bound). We release an open-source PyTorch implementation, the full FairBench harness, and all raw CSV outputs to support independent verification.

Comment: A gradient-chaos detector gates an active-cancellation term that supplies adaptive damping and implicit warmup.

Topic Match: The paper directly proposes and evaluates a pretraining-relevant optimizer based on observed gradient dynamics.

Relevance: 8 Novelty: 6

3. FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

ArXiv ID: 2607.03763

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Shuai Li, Qinglin Wang, Ping Luo, Jiahuan Wang, Hongyang Hu, Haotian Mo, Yigui Feng, Ziang Liu, Qisong Xiao, Jie Liu, Tao Sun

Abstract: Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federated AdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM fine-tuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released.

Comment: Introduces coordinate-wise trust modulation for AdamW updates under federated data heterogeneity.

Topic Match: Its core contribution is a distributed-training optimizer that changes how adaptive updates are coordinated across clients.

Relevance: 7 Novelty: 7

4. Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities

ArXiv ID: 2607.03171

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Arash Badie-Modiri, Chiara Boldrini, Lorenzo Valerio, János Kertész, Márton Karsai

Abstract: Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no risk of single-point failure. However, the role of structural and temporal inhomogeneities in such fully decentralised settings remains poorly understood. Here, we investigate their effects when model parameters are locally averaged during aggregation. We show that the decentralised federated learning process is governed, both in the early phase and the late, stationary limit, by the same dynamics as a lazy random-walk diffusion process on temporal networks. Based on this mapping, we demonstrate that the typical experimental scenario used in decentralised federated learning leads to unrealistically rapid convergence because of ignoring the temporal and structural inhomogeneities inherent in the communication network. We analyse real-world temporal networks and find that inhomogeneities most often dramatically slow down diffusion, hence the convergence process.

Comment: Maps decentralized federated averaging to lazy random-walk diffusion and quantifies convergence slowdowns from temporal heterogeneity.

Topic Match: The paper analyzes convergence of a decentralized distributed-training protocol under realistic communication structure.

Relevance: 6 Novelty: 8

5. FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filtering

ArXiv ID: 2607.04170

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Liyang Yuan, Yibo Yang, Dandan Guo

Abstract: Federated Learning (FL) enables decentralized training without data sharing, but suffers from statistical heterogeneity across clients, leading to client drift, poor generalization, and sharp minima compared to centralized training. Sharpness-Aware Minimization (SAM) has emerged as a promising approach to improve generalization, yet its application in federated learning still suffers from divergence problems, since perturbations are computed locally and reflect client-specific loss geometries. To better understand this issue, we provide experimental evidence from a new perspective, the frequency domain, for SAM perturbations in federated settings, revealing that inter-client perturbation inconsistencies are predominantly concentrated in the low-frequency spectrum. Motivated by this insight, we propose Federated learning with Frequency-domain Filtering of SAM perturbations (FedFFT). It is a lightweight and plug-and-play method that filters out low-frequency components of SAM perturbations without requiring additional communication, thereby suppressing inconsistent components in client updates while preserving consistent learning signals. Extensive experiments across multiple benchmarks and diverse backbones demonstrate that FedFFT consistently outperforms SAM-based FL methods, particularly under severe non-IID distributions. These results highlight the effectiveness, scalability, and general applicability of our frequency-domain perspective for sharpness-aware federated optimization.

Comment: Filters low-frequency components of local SAM perturbations to reduce client drift without additional communication.

Topic Match: The core contribution is a communication-neutral optimization method for heterogeneous distributed training.

Relevance: 7 Novelty: 6

Architecture and Training Dynamics (17)

1. Separating Representation from Reconstruction Enables Scalable Text Encoders

ArXiv ID: 2607.04011

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Megi Dervishi, Mathurin Videau, Yann LeCun

Abstract: While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

Comment: Separates encoded representations from token reconstruction, enabling high masking ratios and denser gradient collection.

Topic Match: The architectural separation is primary, while its higher throughput and sample efficiency provide an additional efficiency match.

Relevance: 8 Novelty: 8

2. Tensor-Train Joint Modeling for Few-Step Discrete Diffusion

ArXiv ID: 2607.03788

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Byoungkwon Kim, Minhyuk Sung

Abstract: Discrete diffusion promises orders-of-magnitude faster generation than autoregressive (AR) models for sequential discrete data, yet its full potential of few-step generation has remained out of reach due to a fundamental structural limitation. The conditional-independence assumption underlying current discrete diffusion models introduces a systematic parallelization bias that compounds with the number of tokens unmasked per step, becoming severe in the few-step regime that fast generation requires. We address this with the first framework for explicit joint distribution modeling in discrete diffusion via tensor decomposition, which represents the conditional clean distribution as a low-rank tensor with controllable expressivity. The framework supports both Canonical Polyadic (CPD) and Tensor-Train (TTD) decompositions, and we identify a structural bias of TTD toward dependencies between nearby tokens, formalized through Oseledets' theorem relating TT-rank to unfolding-matrix rank, which is well-suited to sequential data such as natural language and line notations for molecular data. To enable efficient generation, we present an iterative marginal inference procedure with specialization for predetermined position schedules. Our framework integrates into pretrained MDMs through lightweight fine-tuning, yielding substantial improvements in few-step generation at a fraction of the cost of training from scratch. Code available at https://github.com/ssamt/tensor-train.

Comment: Low-rank joint token modeling reduces the conditional-independence bias that limits few-step discrete diffusion.

Topic Match: The core architectural change is the denoiser's joint-distribution parameterization; tensor decomposition and efficient marginal inference also address generation cost.

Relevance: 8 Novelty: 8

3. SHiPPO: Recurrent Memory with Transported Polynomial Projections

ArXiv ID: 2607.03055

Primary Topic: Architecture and Training Dynamics

Authors: Tomoya Mizuguchi, Bum Jun Kim

Abstract: HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interaction. We introduce SHiPPO (Sylvester HiPPO), a transported projection-memory prior that lifts HiPPO coefficient memories into a moving channel frame. For any fixed or realized right-transport path, SHiPPO transports the approximation family and channel metric together; conditional on that path, the state is ordinary HiPPO in a tied moving frame and follows Sylvester coefficient dynamics, preserving the left online-memory operator while adding right-action transport. For selective-SSM execution, we derive a restricted group-local realization with controller-compatible right actions, exponential-adjusted updates, exact block-affine scan, and recurrent decoding. We also give a simultaneous-reducibility criterion identifying when right transports collapse to static mixing plus independent scalar or blockwise banks. Controlled diagnostics show that larger current-token write rank improves ordinary prediction error but cannot recover order-sensitive changes to already-written memory; transported-memory variants recover this signal, which disappears when the transport pathway is removed. A finite-field associative-recall diagnostic with interleaved bindings, operations, and queries provides complementary autoregressive evidence while leaving the preferred right-action realization open. Taken together, these results support SHiPPO as a mechanistically grounded transported-memory prior, with evidence focused on memory mechanisms rather than broad sequence-modeling dominance.

Comment: Introduces transported HiPPO state dynamics with token-dependent channel interactions and an exact block-affine scan realization.

Topic Match: Despite its memory framing, the core contribution changes selective-SSM recurrence and execution, supported by a reducibility criterion and controlled mechanistic tests.

Relevance: 8 Novelty: 8

4. Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning

ArXiv ID: 2607.04135

Primary Topic: Architecture and Training Dynamics

Authors: Chan Li, Nigel Goldenfeld

Abstract: The remarkable ability of modern neural networks to generalize improves with increasing network capacity, even when the number of model parameters or effective degrees of freedom exceeds the number of training data points. This phenomenon is all the more surprising given that generalization error diverges when the number of model parameters approaches a critical value from below. Here we use dynamical mean field theory to show that this so-called "double descent" behavior is the outcome of a phase transition in the stochastic field theory describing the training process. We calculate the critical exponents and scaling function of the double descent phase transition, and show that it is marked by a breakdown of the fluctuation-dissipation theorem associated with broken ergodicity. The corresponding response function has the same functional form as the simple London model of the superconducting transition, with the rigidity of the wave function corresponding to the neural network's ability to generalize accurately.

Comment: Models double descent as a training phase transition caused by broken ergodicity and fluctuation-dissipation violation.

Topic Match: The paper directly analyzes optimization dynamics and the emergence of generalization in overparameterized networks.

Relevance: 7 Novelty: 8

5. Structure-Specific Representational Priors Causally Control the Grokking Delay

ArXiv ID: 2607.04333

Primary Topic: Architecture and Training Dynamics

Authors: Gunner Levi Howe

Abstract: Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.

Comment: Causally links grokking delay to task-structured feature formation and identifies weight-norm clamping as an accelerator.

Topic Match: The contribution is a direct mechanistic study of training dynamics and delayed generalization.

Relevance: 7 Novelty: 8

6. The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

ArXiv ID: 2607.03347

Primary Topic: Architecture and Training Dynamics

Authors: Joan Bruna

Abstract: We consider the Multiscale Single-Index Model (MSIM), first introduced in \cite{oymak2021learning}, as a stylized model for hierarchical learning with \emph{scale separation}. Each layer extracts a shared single-index feature at one physical scale and passes it to the next, thus defining a tractable setting in which to study how deep architectures learn multiscale representations. Under non-degeneracy and delocalization assumptions on the link function and planted features respectively, for fixed depth $K$ and local scale $d$, the first Wiener chaos of the target behaves as a perturbed spiked tensor, where the perturbation of order $d^{-1/2}$ comes from the non-linearity -- revealing the MSIM as a natural non-linear analogue of the Tensor PCA model \cite{montanari2014statistical}. While this perturbative picture is sufficient to enable efficient spectral recovery based on Tensor unfolding (as already observed in \cite{oymak2021learning}), it is not precise enough for the analysis of backpropagation gradient-based methods. In this work, we address this limitation by performing a fine-grained analysis of the Wiener chaos using Edgeworth expansions. In the first chaos, this gives a finite-rank hierarchy at scales $d^{-q/2}$. In higher chaoses, balanced flattenings exhibit staircase singular-value plateaus of size $d^{-ρ/2}$ and multiplicity $d^ρ$ under a natural higher-chaos non-cancellation condition. Using this higher-chaos structure, and under an additional slow Hermite-energy tail condition, we first establish shallow-network approximation lower bounds, quantifying the benefit of depth in this model. Next, and most importantly, we prove that online SGD on the correlation objective, where all layers evolve in the same timescale, achieves $1 - o_d(1)$ recovery with $n = \widetilde{O}( d^{K-1})$ samples, recovering the same sample complexity as in the linear counterpart.

Comment: Edgeworth-resolved chaos structure yields an SGD recovery guarantee and quantifies depth's sample-complexity benefit.

Topic Match: The core contribution analyzes how depth and joint-timescale SGD learn hierarchical features.

Relevance: 7 Novelty: 8

7. Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks

ArXiv ID: 2607.03798

Primary Topic: Architecture and Training Dynamics

Authors: Yoshihiro Maruyama

Abstract: Symmetry is everywhere in nature and society. Geometric deep learning builds architectures respecting group symmetries, whereas topological deep learning organizes computation through cells, incidence relations, and local-to-global structure. In this paper, we extend geometric deep learning beyond simple group actions and unify it with topological deep learning. Specifically, we develop order-equivariant neural networks (OENN), which generalize standard graph message passing and sheaf neural networks via the theory of equivariant vector bundles over face posets (or face categories). We (i) characterize all linear order-equivariant maps, (ii) build OENN layers, and (iii) prove universal approximation theorems (UATs) for continuous order-equivariant maps, which are new results even when restricted to sheaf neural networks. We illustrate the framework on graph and sheaf models. Our results can also be seen as extending the known UAT for graph neural networks to a more general setting that subsumes sheaf neural networks as well. In the appendix, we show that OENN can be connected, via the action groupoid Grothendieck construction, to CENN (category-equivariant neural network), which gives the categorical general form of equivariant neural networks, allowing us to leverage categorical symmetry in data and extending geometric deep learning from groups of symmetries to categories of transformations.

Comment: Order-equivariant layers characterize all compatible linear maps and unify graph and sheaf neural architectures.

Topic Match: The paper's core contribution is a new general architectural framework with accompanying universality results.

Relevance: 7 Novelty: 8

8. Tightening the Score Matching Gap for Diffusion Models

ArXiv ID: 2607.04442

Primary Topic: Architecture and Training Dynamics

Authors: Benjamin Dupuis, Tyler Farghly, Maxime Haddouche, Alain Durmus, Umut Simsekli

Abstract: Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the \emph{score matching gap}, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.

Comment: Backward-process contraction yields tighter links between score-matching loss and multiple distributional errors.

Topic Match: It provides mechanistic theory for how a diffusion training surrogate relates to the quality of the trained generator.

Relevance: 7 Novelty: 8

9. Reflected Schrödinger Bridge Matching

ArXiv ID: 2607.03626

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Marcus Häggbom, Viktor Nilsson, Pierre Nyquist, Joakim andén

Abstract: Recent advances in generative modeling have enabled the efficient computation of Schrödinger bridges (SB) in high-dimensional settings by leveraging partially simulation-free training methods inspired by flow matching. However, these have not covered SBs with reflecting dynamics, a useful model choice with built-in guarantees that generated samples stay in the data domain. Existing alternatives for reflected SBs instead rely on more complex training based on forward--backward SDE theory, requiring expensive higher-order derivatives and sampling entire paths during training. In this article, we introduce a partially simulation-free framework that allows reflected SBs to be trained similarly to flow matching, using a new sampling method and regression target. We demonstrate our results by coupling pairs of well-known high-dimensional image datasets. Using reflected dynamics incurs negligible additional wall-clock time during both training and inference while maintaining or slightly improving generative performance.

Comment: A partially simulation-free objective trains reflected Schrödinger bridges without higher-order derivatives or full-path sampling.

Topic Match: The main advance is a new generative-model training formulation with reduced computational requirements.

Relevance: 7 Novelty: 8

10. Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

ArXiv ID: 2608.14604

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Aryuemaan Kumar Chowdhury, Praveen Oosa, Vineesha Reddy

Abstract: Small language models in the ten to one hundred million parameter range are attractive for on device inference, rapid experimentation, and controlled scientific study, yet most of them reuse the standard transformer block without adaptation to the small scale regime. We present Wiola, a decoder only language model whose novelty is concentrated in three drop in components of every layer. First, Spiral Rotary Positional Encoding perturbs the standard rotary frequencies by a slowly growing per dimension factor so that phase trajectories fan outward, improving long range discrimination while adding no parameters. Second, Gated Spiral Attention introduces a per head, content adaptive scalar gate derived from a causal cumulative statistic of the query stream, providing an implicit and differentiable form of soft head selection at negligible cost. Third, the Butterfly feed forward block replaces the conventional expansion layer with a multiplicative interaction and an intra block bypass path, matching the parameter count of a four times gated linear unit block while improving gradient flow in shallow stacks. We formalize each component, derive exact parameter and computation budgets, and prove that the gated attention admits an exact and numerically verified equivalence between full sequence training and cached autoregressive decoding, so that no approximation is introduced at inference time. We also describe a fully reproducible training and evaluation protocol on a standard tiny story corpus. The reference implementation is released as an open source package with weights ready publishing support.

Comment: Adds content-adaptive per-head gating with exact equivalence between full-sequence training and cached decoding.

Topic Match: The gated attention mechanism is the primary contribution, with parameter and computation efficiency as a secondary benefit.

Relevance: 8 Novelty: 6

11. Transferability Between Understanding and Generation in Unified Multimodal Models

ArXiv ID: 2607.04423

Primary Topic: Architecture and Training Dynamics

Authors: Jiwon Kang, Heeji Yoon, Jaewoo Jung, Jaewon Min, Minkyeong Jeon, Biyeon Hwang, Sangwon Jung, Seungryong Kim

Abstract: Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Comment: Shows that fully shared multimodal architectures permit capability transfer between understanding and generation tasks.

Topic Match: Controlled architectural comparisons reveal a mechanistic link between parameter sharing and cross-task transfer.

Relevance: 7 Novelty: 7

12. ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum

ArXiv ID: 2607.04535

Primary Topic: Architecture and Training Dynamics

Authors: Haiwen Yi, Xinyuan Song

Abstract: Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow

Comment: A Stiefel-constrained basis is paired with a learnable positive spectrum for direct singular-value control.

Topic Match: The contribution is a new spectrally controlled layer parameterization rather than a compression procedure.

Relevance: 7 Novelty: 7

13. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

ArXiv ID: 2607.04233

Primary Topic: Architecture and Training Dynamics

Authors: Shokhrukh Ibragimov, Arnulf Jentzen

Abstract: Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems. In practically relevant DNN training problems, one does usually not apply the standard gradient descent (GD) optimization method but instead one employs suitable sophisticated GD optimization methods, which incorporate adaptivity and/or acceleration techniques, such as the famous Adam optimizer. It is a key contribution of this work to provide a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation. Our general unified convergence result applies to a large class of gradient based optimization methods such as the standard GD, the momentum, the Nesterov accelerated gradient (NAG), the RMSprop, the Adam, the Adamax, the Nadam, the Nadamax, the Adan, the AdaBelief, the AMSGrad, and the Yogi optimizers. Our analysis employs the theory of Kurdyka-Łojasiewicz (KL) inequalities to establish convergence to critical points in the training of DNNs. To the best of our knowledge, the generality of our convergence analysis is also just in the special situation of the Adam optimizer a new contribution to the literature on the analysis of AI optimization algorithms.

Comment: Uses Kurdyka–Łojasiewicz inequalities to unify convergence-to-critical-point analysis across adaptive and accelerated neural-network optimizers.

Topic Match: The contribution concerns neural-network optimization dynamics and convergence, with large-scale practical implications left unspecified in the abstract.

Relevance: 7 Novelty: 7

14. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

ArXiv ID: 2607.03613

Primary Topic: Architecture and Training Dynamics

Authors: Shuang Liang, Tom Jacobs, Guido Montúfar

Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the structure of this stationary measure and the predictor it represents. We show that, despite the network being infinitely overparameterized, the learned predictor admits an effectively finite representation: the input weights and biases align along finitely many directions, leading to an effective width collapse. In particular, the solution function is continuous piecewise affine, with affine regions determined by the cells of a finite hyperplane arrangement. The number of learned directions, and hence hyperplanes, is bounded above by $2\mathcal{P}-1$, where $\mathcal{P}$ denotes the number of linear dichotomies realizable on the training inputs. We further establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data. Consequently, the complexity of the learned predictor is governed by the combinatorial geometry of the training data.

Comment: Shows that noisy SGD induces effective finite-width collapse in infinitely wide ReLU networks.

Topic Match: The paper provides mechanistic optimization analysis of how SGD shapes learned network structure.

Relevance: 6 Novelty: 8

15. A Unified Framework for In-Context Learning with Causal and Masked Language Models

ArXiv ID: 2607.04081

Primary Topic: Architecture and Training Dynamics

Authors: Chenrui Liu, Chuanlong Xie, Falong Tan, Yicheng Zeng, Lixing Zhu

Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models. Masked language models instead recover masked tokens from bidirectional context, and their role in ICL remains less understood. We develop a statistical learning framework that represents the context examples by their empirical measure and models prediction as a function of the context and the query. This formulation places autoregressive and masked pretraining objectives within a common excess-risk analysis. Under Wasserstein-type regularity conditions, we relate pretraining with T tasks and N samples per task to k-shot excess risk at inference, obtaining same-order upper bounds for masked and autoregressive objectives. We also study task-distribution shift, where pretraining tasks are sampled from P and inference tasks from Q; the resulting bound contains an additional term controlled by the lifted Wasserstein distance between P and Q. The bounds further imply an order-optimal allocation under a fixed pretraining data budget and refined rates under intrinsic low-dimensional structure. Experiments on controlled function-learning tasks show that the Masked Pair Encoder (MPE) can achieve performance comparable to GPT-2-style causal Transformers, suggesting that ICL behavior is not specific to causal language models.

Comment: Unifies causal and masked pretraining under excess-risk bounds that also prescribe data-budget allocation.

Topic Match: The central insight concerns how alternative pretraining objectives produce in-context learning behavior.

Relevance: 6 Novelty: 7

16. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

ArXiv ID: 2607.03660

Primary Topic: Architecture and Training Dynamics

Authors: Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Comment: Shows that sequential correlations alter effective context length and favor different attention mechanisms.

Topic Match: It provides mechanistic insight into when linear and softmax attention are mismatched to correlated sequence data.

Relevance: 6 Novelty: 7

17. Burst Spiking Neural Networks

ArXiv ID: 2607.11914

Primary Topic: Architecture and Training Dynamics

Authors: Jiahong Zhang, Sijun Shen, Man Yao, Han Xu, Mingqiang Huang, Yonghong Tian, Bo Xu, Guoqi Li

Abstract: A central goal of current Spiking Neural Network (SNN) research is to improve their accuracy toward becoming low-power alternatives to Artificial Neural Networks (ANNs). This work further argues that realizing this ambition requires improving not only accuracy but also robustness, defined as the ability to maintain correct predictions under input perturbations. We identify two key issues in existing SNN methods that undermine robustness. First, binary spiking activations can produce large activation-state changes under small perturbations. Second, the lack of effective weight constraints makes network outputs more sensitive to input variations. To this end, we propose Burst Spiking Neural Networks (BuSNNs), built upon Burst-enhanced Spiking Neurons (BSNs) and a Dynamic Weight Constraint (DWC) mechanism. BSNs incorporate burst firing to provide a graded spiking pattern. This spiking mechanism mitigates perturbation-induced transitions in activation states and thereby enhances robustness. DWC penalizes connection weights based on activation states, effectively reducing weight magnitudes and improving robustness while preserving accuracy. We provide theoretical analyses to support these robustness effects. Experimental results further show that, on smaller-scale benchmarks such as CIFAR-10, BuSNNs outperform both SNN and ANN counterparts in accuracy and robustness. On large-scale ImageNet, BuSNN with the MS ResNet-34 backbone further improves top-1 accuracy and corruption robustness over the corresponding SNN baseline by 3.18% and 2.66%, respectively. Despite using spike-based activations, BuSNNs surpass 4-bit activation-quantized ANN baselines and approach 8-bit ANN baselines on ImageNet. They also preserve SNNs' low-power advantage. This work studies the accuracy-robustness problem in SNNs, advancing their practical viability in robust and energy-efficient applications.

Comment: Introduces burst-firing neurons and activation-dependent weight constraints as new architectural mechanisms for robust, low-power SNNs.

Topic Match: The central contribution changes neuron behavior and weight dynamics rather than the distributed training system.

Relevance: 6 Novelty: 6

Efficiency, Compression, and Large-Scale Training (21)

1. LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

ArXiv ID: 2607.03057

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang, Ruiqun Li, Changqun Li

Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).

Comment: Loss-sensitive rank allocation is combined with propagation-aware correction of accumulated SVD compression error.

Topic Match: This is directly centered on low-rank LLM compression and how rank budgets affect global model error.

Relevance: 9 Novelty: 7

2. Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

ArXiv ID: 2607.02893

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hamish Ogilvy

Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width. We introduce Variable Bit-width Quantization (VBQ), a training-time method in which each contiguous group of 64 weights learns its own resolution from {1,2,4,8} bits via a Gumbel-Softmax relaxation, trained jointly by an alternating optimization that gives the precision logits a clean, task-aligned signal. VBQ discovers a consistent, strongly heterogeneous allocation within individual projection types, not merely across layers, impossible to express with per-layer methods: 69% of groups collapse to 1 bit, the LM head averages 1.09 bits, while the first MLP block keeps ~2.5 bits. This pattern is stable enough to freeze into a fixed recipe and reuse without further search. The recipe yields a "bigger-but-smaller" regime: a 131M model at 1.82 mean bits reaches perplexity 4.2 on TinyStories, beating a 55M FP16 model (PPL 4.4) at 3.8x less storage, and lets a 1.46B model on FineWeb-Edu match a 593M FP16 control at ~3.7x less storage with 2.5x more parameters. As quality-per-byte, VBQ is 3.9-8.4x more efficient than FP16. The recipe maps directly to packed low-bit storage, so it also accelerates inference: with custom fused dequantize-and-multiply kernels, memory-bandwidth-bound autoregressive decode is faster at equal output, and the speedup grows with scale (parity at 131M, 1.9x at 1.0B, 4.7x at 9B on Apple silicon). A distributional analysis (KL divergence and argmax-flip rate) reveals a striking mechanism: deeper layers progressively self-heal the quantization error injected by early layers. The win is a from-scratch, train-time phenomenon; scaling the search economically beyond 1.5B parameters remains open. VBQ reframes precision as a learnable, non-uniform resource and shows that spending a fixed bit budget unevenly beats spending it uniformly.

Comment: Learns per-group weight precision during training through alternating optimization of quantized weights and discrete bit-width choices.

Topic Match: Learned mixed-precision allocation directly changes model storage and inference cost, with a training-time mechanism and explicit limits on economical search at larger scales.

Relevance: 9 Novelty: 7

3. ELiTeFormer: An Efficient Transformer for FPGAs

ArXiv ID: 2607.03652

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo

Abstract: Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.

Comment: Co-designs linear attention and ternary projections with multiplier-free FPGA kernels to reduce model and KV-cache cost.

Topic Match: Compression and hardware-efficient inference are central, with linear attention as a supporting architectural mechanism.

Relevance: 8 Novelty: 7

4. HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

ArXiv ID: 2607.04302

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou

Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16. To our knowledge, HiFA4 is the first Ascend-HIF4-targeted design of this kind evaluated on standard NLP benchmarks. HiFA4 combines two mechanisms. Smooth-QK applies a calibration-static per-channel equivalent rescaling to Q and K after RoPE, transferring quantization difficulty from K to Q without per-tile online reduction at inference. P-Reordering accumulates the softmax normalizer from the same quantized attention weights P_hat used in the PV GEMM, rather than from a higher-precision reconstruction. We show that this inconsistent formulation introduces a coherent output-scaling error, and validate the effect on a Qwen3-8B Layer-0 MMLU trace, where all 3.6M measured attention tiles exhibit net probability-mass loss with median epsilon_bar = -0.064. P-Reordering also allows the normalizer to be fused into the PV Cube GEMM. Across five LLMs, HiFA4 reduces quantization-induced decision drift. On Qwen3-8B, it recovers 37.5% of the accuracy gap introduced by direct HIF4 quantization, narrows the sample-weighted accuracy loss from 1.12 pp to 0.70 pp, reduces BF16-inconsistent MMLU predictions from 16.3% to 8.2%, and cuts MMLU accuracy regressions by 57% (1071 to 465). On Gemma2-9B, mild smoothing keeps HiFA4 within 0.7 pp of BF16 while reducing MMLU regressions by 27%. On LLaMA3.1-8B, Mistral-7B, and Phi-4B, where Smooth-QK is disabled, P-Reordering with the adopted Q-Mean auxiliary still reduces full-set MMLU regressions by 41-52%. A preliminary instruction-scheduling analysis projects a 35.4% critical-path latency reduction relative to BF16 by fusing the softmax normalizer into the PV Cube GEMM; on-hardware validation is left to future work.

Comment: Executes both FlashAttention matrix products in 4-bit arithmetic using quantization-aware rescaling and normalizer reordering.

Topic Match: The central contribution is a new low-precision attention operator that reduces projected latency while controlling accuracy loss.

Relevance: 8 Novelty: 7

5. Uncertainty-gated selection for block-sparse attention

ArXiv ID: 2607.07724

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Thomas Rossi

Abstract: Block-sparse attention scales long-context language models by replacing the O(N^2) softmax with a per-query top-k selection over key blocks. This cutoff is myopic: when the k-th and (k+1)-th blocks are nearly tied in score, the selector commits without spending extra budget, and a dropped block carrying answer evidence is unrecoverable downstream. We propose a value-of-information router that measures, for each query, how decisively the top-k cut was made, and doubles the kept set for the queries where that gap is smallest; the rule is backbone-agnostic and stacks with existing block-scoring methods such as Quest. On LongBench-v2 medium at n=215 (the entire dataset subset), router-on-Quest reaches paired recall 0.75 vs. top-k 0.47 -- +28 pp over the SSA-style baseline (McNemar p<0.01) -- and lands within 2 pp of dense on RULER NIAH multikey at the same context. The lift reproduces on four models from three architectures (Qwen2.5, Mistral-Nemo, Qwen3.6). At 128K, the router preserves 0.81 and 0.89 of dense accuracy on Qwen2.5-7B-1M and Qwen3.6 (vs. SSA-style top-k at 0.09 on the former) while the fused selection-plus-kernel pipeline runs at 0.62x and 0.80x dense wall time.

Comment: An uncertainty-gated router expands block-sparse attention selectively when the top-k cutoff is ambiguous.

Topic Match: The primary result is a dynamic sparse-attention mechanism that reduces long-context inference cost while preserving evidence.

Relevance: 8 Novelty: 7

6. SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

ArXiv ID: 2607.04306

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Omer Tariq, Syed Muhammad Raza, Jeongbae Son

Abstract: Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \textbf{SAD-LoRA} (\textbf{S}pectral \textbf{A}lignment \textbf{D}istillation), which selects this subspace from the data-weighted student-space reference update $\DWT\Sigx^{1/2}$ and maintains it during training via a differentiable principal-angle loss on $\colspan(B)$. We show that the data-weighted distillation error decomposes exactly into subspace misalignment, within-subspace coefficient mismatch, and irreducible rank residual; standard KD can affect the first term only indirectly through output gradients. On controlled synthetic problems with a flat teacher spectrum, SAD-LoRA reduces the subspace-misalignment term from $51\%$ to nearly zero and lifts final subspace alignment from $0.49$ to $1.00$. On RoBERTa-large to RoBERTa-base distillation across six GLUE tasks, SAD-LoRA improves rank efficiency: at $r{=}4$, it matches or beats the strongest included spectral baseline on five of six tasks, and at $r{=}8$ it gives the best result on SST-2 and CoLA. Ablations identify subspace alignment as the load-bearing component, while coefficient matching is auxiliary.

Comment: A principal-angle loss explicitly keeps LoRA updates aligned with a data-weighted teacher-update subspace.

Topic Match: The method directly improves the rank efficiency of low-rank knowledge-distillation adapters.

Relevance: 8 Novelty: 7

7. HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

ArXiv ID: 2607.03012

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Dongyeun Lee, Amir Zandieh, Vahab Mirrokni, Junmo Kim, Insu Han

Abstract: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation. However, their ability to produce long-duration videos is fundamentally constrained by the quadratic complexity of the self-attention mechanism. Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks: substantial clustering overhead and low CTA utilization caused by irregular cluster-induced blocks. We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly. To reduce clustering overhead, we introduce 3D local-window clustering, which exploits the spatio-temporal locality of video tokens to restrict centroid search to fixed local neighborhoods, and implement it with a custom Triton kernel for efficient execution. We further propose a hybrid clustering strategy that performs full clustering only at anchor steps and updates only subset tokens at intermediate steps, leveraging the temporal stability of cluster assignments across denoising steps. To improve CTA utilization, we present hardware-aware cluster merging that minimizes CTA-aligned execution cost through parallel agglomerative merging, improving block density and approximation fidelity by utilizing idle tile capacity. Together, these components reduce clustering overhead, avoid redundant updates, and better align sparse attention with the fixed tile structure of modern GPU kernels. Experiments on Text-to-Video generation show that HVA establishes a new Pareto frontier for training-free sparse attention in video diffusion, reducing end-to-end latency by up to $2.13\times$ while improving fidelity over existing training-free sparse attention baselines.

Comment: Hardware-aware cluster merging and temporally reused local clustering reduce sparse-attention execution cost.

Topic Match: The core contribution is an attention-sparsification algorithm and kernel strategy with measured end-to-end savings; video generation provides the evaluation setting.

Relevance: 8 Novelty: 7

8. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

ArXiv ID: 2607.09757

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jiaqi Liu, Haidong Kang, Qihui Zhao, Guo Yu, Jingchao Wang

Abstract: Parameter-efficient fine-tuning enables large language models to adapt to downstream tasks with substantially lower computational and storage cost, and Low-Rank Adaptation (LoRA) is among its most widely used techniques. However, vanilla LoRA assigns a uniform rank to all adapted modules, while existing adaptive methods either incur additional optimization overhead or rely on static weights and local gradients that do not capture task-conditioned representation changes. We propose RSRA, a training-free rank allocator that estimates where adaptation capacity is most needed through forward-only representation sensitivity probing on a small calibration set. Specifically, RSRA uses Spectral Effective Rank to allocate capacity across layers, measures module-wise hidden-state displacement under standardized virtual low-rank updates with the Frechet Distance, and combines both signals through hierarchical normalization to produce a task-aware rank configuration before fine-tuning. Across commonsense reasoning and natural language understanding benchmarks with Qwen3-4B and Mistral-7B, RSRA achieves the highest average performance in all three reported model-benchmark settings and a 1.48x-1.93x speedup in allocation time over the fastest competing pre-allocation method. When integrated with DoRA, LoRA-FA, and PiSSA, RSRA improves 15 of the 18 evaluated combinations and increases the average performance of all three PEFT methods.

Comment: Allocates LoRA ranks through forward-only, task-conditioned representation-sensitivity probing.

Topic Match: It directly improves low-rank adaptation by assigning limited parameter capacity where it is most useful.

Relevance: 8 Novelty: 6

9. RADIO1D: Elastic Representations for Condensed Vision Modeling

ArXiv ID: 2607.03624

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Greg Heinrich, Mike Ranzinger, Collin McCarthy, Natan Bagrov, Eugene Khvedchenya, Bryan Catanzaro, Jan Kautz, Andrew Tao, Pavlo Molchanov

Abstract: This paper challenges the assumption that vision-language models (VLMs) require fixed patch-based 2D vision features. Analyzing fine-tuned vision encoders, we find that representations become increasingly abstract and less spatially coherent during VLM training. Notably, models trained with image-text alignment (such as SigLIP2) develop a small number of specialized tokens that effectively summarize global image content. Building on this, we introduce RADIO1D, which compresses images into a compact, variable-length 1D token sequence using multi-teacher knowledge distillation and an autoencoder design. The resulting representations exhibit strong hierarchical summarization, enabling accurate scene understanding - even with a single token - and support improved composition-aware image retrieval. In VLMs, RADIO1D provides flexible accuracy-efficiency tradeoffs through adjustable token counts, delivering competitive performance on diverse multimodal benchmarks with lower computational overhead and better accuracy.

Comment: Compresses images into adjustable-length one-dimensional token sequences using multi-teacher distillation and an autoencoder.

Topic Match: Variable-length token compression is central, with a new condensed representation architecture enabling the tradeoff.

Relevance: 7 Novelty: 7

10. Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumption

ArXiv ID: 2607.04553

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Nidhal Jegham, Boris Gamazaychikov, Sasha Luccioni

Abstract: We present a bidirectional framework for estimating the energy consumption of text-to-video (T2V) and text-to-video-audio (T2VA) models from architectural first principles and observable generation parameters such as resolution and duration, requiring no access to weights, model size, or implementation details. Forward, it predicts energy from generation parameters and architectural principles; backward, it recovers architectural scaling behavior from observed inference times, with accuracy serving as a criterion for architectural validity. Building on the established compute-bound nature of video diffusion models, we demonstrate that each model's energy profile obeys theoretically derived scaling laws, decomposing into quadratic and linear terms whose coefficients directly reflect the underlying architectural complexity. Validated across six open-source models spanning 8.3B-27B parameters and three GPU configurations, this decomposition achieves below 3% MAPE across all architectures. This approach offers a standardized, empirically and theoretically grounded framework for sustainability benchmarking across T2V models and architectures.

Comment: Derives architecture-aware scaling laws that predict video-generation energy consumption from observable generation parameters.

Topic Match: Its main result quantifies how architectural scaling determines inference energy cost.

Relevance: 7 Novelty: 7

11. Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

ArXiv ID: 2607.05457

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Anis Hamadouche, Amir Hussain

Abstract: Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable--observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy, but also as actual compressed layer widths for realised reduced networks. Experiments on MNIST and CIFAR-10 compare the proposed balanced realisation against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantisation, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression, while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, giving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results show that balanced reachable-observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures with little or no loss in accuracy.

Comment: Controllability-observability ranks directly determine compressed layer widths through balanced state-order reduction.

Topic Match: The method is a structured compression mechanism that converts measured hidden-state redundancy into smaller networks.

Relevance: 7 Novelty: 7

12. ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

ArXiv ID: 2607.07719

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Wentao Lu

Abstract: Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones. We present ReCoLoRA (Recursive Consolidation of Low-Rank Adapters), a spectrum-aware framework for continual fine-tuning: adapters are initialized from a randomized SVD of the pretrained weight, per-layer effective ranks are selected by an elbow criterion, and the principal subspace is adapted before residual capacity is opened. Before each new task, ReCoLoRA re-decomposes the current effective weight, rather than the original one, into a frozen residual, a slowly updated principal component, and a fresh adapter (recursive consolidation), so every task starts from the model that has already absorbed its predecessors. On a six-task continual GLUE sequence over four 7-8B backbones, ReCoLoRA attains the best final average score on three of the four backbones against rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters; an oracle-routed task-bank variant serves as an upper bound under full task isolation. Code: https://github.com/bhqy666/ReCoLoRA.

Comment: Recursive spectral decomposition consolidates prior LoRA updates before allocating fresh low-rank capacity.

Topic Match: The core mechanism improves parameter-efficient continual fine-tuning through spectrum-aware low-rank adaptation.

Relevance: 7 Novelty: 7

13. Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic

ArXiv ID: 2607.04531

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Anis Hamadouche, Amir Hussain

Abstract: Low-precision neural networks are attractive for resource-constrained hardware, but fixed-point arithmetic introduces failure modes that are often hidden by idealised quantisation models. In particular, two's-complement overflow wrapping can corrupt hidden activations by changing both their magnitude and sign, leading to unstable numerical error propagation and severe accuracy degradation. This paper proposes a Lyapunov-stabilised quantisation framework for low-precision neural networks operating under hardware-style wrapping arithmetic. The hidden-state energy is monitored through a layerwise Lyapunov function, and a monotone projection is applied to enforce bounded and non-increasing state evolution across depth. The method is evaluated on MNIST using a compact patch-based transformer under post-training quantisation and quantisation-aware training with fixed-point bit-widths from 4 to 16 bits. Monte Carlo results show that unconstrained wrapped quantisation-aware training collapses to near-chance accuracy across 6-16 bits, with activation overflow rates exceeding 11%. In contrast, the proposed monotone Lyapunov projection suppresses activation overflow to below 0.012% and restores stable low-precision learning, achieving 86.55% accuracy at 12 bits. These results demonstrate that Lyapunov-based state control can act as a hardware-aware stabilisation mechanism for reliable fixed-point neural inference and training.

Comment: A monotone Lyapunov projection stabilizes fixed-point quantization-aware training under overflow-wrapping arithmetic.

Topic Match: Low-precision compression is the primary topic, augmented by a new mechanism for numerical training stability.

Relevance: 7 Novelty: 7

14. Asymptotic Preservation and Uniform Accuracy of Diffusion and Flow-Matching Samplers

ArXiv ID: 2607.04113

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Shiheng Zhang

Abstract: Diffusion and Gaussian-interpolant flow-matching samplers approach data through a terminal noise floor $\varepsilon$, a singular limit for manifold-supported or rank-deficient data. We study two properties of a complete sampler specification, comprising its update rule, time grid, and terminal rule. Asymptotic preservation (AP) means a stable and consistent zero-noise discretization with a step count bounded independently of $\varepsilon$. Uniform accuracy (UA) of order $p$ means that, at numerical resolution $h$, the endpoint $W_2$ error is $O(h^p)$ with a floor-independent constant. Bounded log-noise stepping fails AP because its step count diverges. Stopping a stable base solver at a positive switching scale $a$ and appending one map fitted to the analytic normal mode restores AP. On smooth compact boundaryless manifolds, the standard map has exact-input error $O(a^2-\varepsilon^2)$ and sharp zero-floor error $Θ(a^2)$. A base solver with a floor-uniform order-$p$ estimate on the resolved interval retains that order when $a=O(h^{p/2})$, provided the terminal transfer factor remains bounded. Along exact trajectories, the posterior-mean identity $D(x(σ),σ)=x(σ)-σx'(σ)$ cancels the linear terminal defect and enables higher-order fitted maps. A three-evaluation Hermite construction is uniformly third order for exact switching-scale input over $0\le\varepsilon\le a$, and a seven-evaluation construction is fourth order at zero. We classify representative diffusion and flow-matching specifications by AP and UA. On EDM and Rectified Flow checkpoints, a paired decomposition separates base-integration from terminal-completion error and predicts held-out same-seed endpoint errors.

Comment: Fitted terminal maps make diffusion sampling asymptotic-preserving with a noise-floor-independent step count.

Topic Match: The main practical contribution is a numerical sampling design that controls computational cost and endpoint accuracy near zero noise.

Relevance: 6 Novelty: 8

15. Spectral Rewiring for Exploration, Purification, and Model Merging

ArXiv ID: 2607.03065

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Abstract: Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.

Comment: Post-hoc spectral filtering extracts a compact reasoning-effective core from dense parameter updates.

Topic Match: The closest foundational fit is spectral compression of learned updates for efficient retention and merging of capabilities.

Relevance: 6 Novelty: 8

16. Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

ArXiv ID: 2607.03784

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhenfeng Su, Kang Zhao, Han Bao, Tao Yuan, Zhongzhe Hu, Xianzhi Yu, Wenxuan Wang

Abstract: While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58$\times$ speedup for DeiT-B while maintaining accuracy and a 1.39$\times$ speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24$\times$ to 5.19$\times$ for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Comment: Uses inter-layer heterogeneity to select removable Transformer layers without recovery-breaking dimension mismatches.

Topic Match: Depth pruning is the central mechanism and directly changes model size and execution cost.

Relevance: 7 Novelty: 6

17. A Unified Framework for Quantized and Continuous Strong Lottery Tickets

ArXiv ID: 2607.03860

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Aakash Kumar, Emanuele Natale

Abstract: The Strong Lottery Ticket Hypothesis (SLTH) asserts that sufficiently overparameterized, randomly initialized neural networks contain sparse subnetworks that, even without any training, can match the performance of a small trained network on a given dataset. A key mathematical tool in the theoretical study of SLTH has been the Random Subset Sum Problem (RSSP). The SLTH has recently been extended to the quantized setting, where the network weights are sampled from a discrete set rather than from a continuous interval. These new results are however far from those in arbitrary-precision setting in several ways. In this work, we provide an analysis of the RSSP in the discrete setting, and use it to derive tight SLTH guarantees in the quantized case. Our analysis obtain tight bounds on the failure probability of finding a strong lottery ticket in the quantized regime, providing an exponential improvement over previous results. Most importantly, it unifies the literature by showing that both approximate representations in the continuous setting and exact representations in quantized settings naturally emerge as limiting cases of our results. This perspective not only sharpens existing bounds but also provides a cohesive framework that simultaneously handles approximation and rounding errors.

Comment: Provides tight unified guarantees for sparse lottery-ticket existence under both quantized and continuous random weights.

Topic Match: The work most directly concerns the theoretical foundations of simultaneous sparsity and quantization.

Relevance: 6 Novelty: 7

18. Teacher Supervision over Representation Equivalence Classes

ArXiv ID: 2607.03572

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Sang Il Han

Abstract: Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.

Comment: A quotient-space formulation identifies invariant or coordinate-aligned objectives for knowledge distillation.

Topic Match: Knowledge distillation is the relevant compression mechanism, with the paper refining what student supervision should preserve.

Relevance: 6 Novelty: 7

19. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

ArXiv ID: 2607.03333

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Huajun Bai, Weiwei Lv, Huichuan Zheng, Youyou Lu, Jiwu Shu

Abstract: LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.

Comment: Self-speculative forking overlaps tool execution with ongoing decoding by using the same model as the predictor.

Topic Match: The closest foundational contribution is a new inference-time computation schedule that reduces agent latency without retraining.

Relevance: 6 Novelty: 7

20. Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation

ArXiv ID: 2607.03522

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Wei Zhang, Lin Tang, Ming Zhao, Yuxuan Wang

Abstract: Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.

Comment: A forward-only controller jointly adjusts LoRA domain sampling and loss weights using competence and transfer signals.

Topic Match: Low-rank adaptation is the primary fit, with an optimization controller governing efficient multi-domain fine-tuning.

Relevance: 6 Novelty: 7

21. OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

ArXiv ID: 2607.03050

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Shijie Cao, Qingyu Zhang, Boxi Yu, Yuzhong Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

Abstract: Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose \textbf{OmniFocus}, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodal-guided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25\% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25\% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38$\times$ prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off.

Comment: Compresses audio and video tokens independently using query-conditioned importance while preserving cross-modal alignment.

Topic Match: Its core mechanism reduces multimodal context and prefill cost, although evaluation is application-specific.

Relevance: 6 Novelty: 6

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains