This is a remedial run for missed papers from 08/27/2026 to 08/27/2026.

Results generated on 09/14/2026.

Personalized Daily ArXiv Papers 2026-08-28

_{Token counts are not reported for this run. 21 of 21 model calls succeeded, 2,539s of model wall clock.}

Topic Coverage:

Table of contents by topic:

MoE Training (1)

1. MM-Spectrum: Multimodal Multi-spectral Molecular Structural Elucidation with a Stable MoE Framework Authors: Hai-tao Yu, Nan Min, Zheng Fang, Hongyu Zhan, Yusen Tan, Yuhan Wang, Jun Xia

Large-Scale Training Systems and Efficiency (3)

1. Tensorion: A Tensor-Aware Generalization of the Muon Optimizer Authors: Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Sergei Kudriashov, Maxim Rakhuba

2. Performance Foundations of Parallel & Distributed Reasoning Language Models Authors: Maciej Besta, Leonard Schmidt, Lara Nonino, Robert Gerstenberger, Pierre Pang, Patrik Okanovic, Ales Kubicek, Tiancheng Chen, Baraq Lipshitz, Torsten Hoefler

3. Beyond Client Averaging: A Client-Independent Second-Order Stationary-Bias Component in Stochastic SCAFFOLD Authors: Yi-Ping Tang, Guan-Ju Peng

Architecture and Training Dynamics (9)

1. Squeezing More from Limited Data with Recursive Transformers Authors: Serdar Gülbahar, Lukas Edman, Alexander Fraser

2. Making Latent Evolution Explicit: Operator-Structured Transitions for World Action Models Authors: Xiaoxiao Lu, Yunlong Dong, Jiahao Shi, Ye Yuan

3. When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging Authors: Shangge Liu, Yuehan Yin, Yinghuan Shi, Lei Wang, Wenbin Li

4. COFM: Consistent Optimal Transport Flow Matching via Partially Input Convex Neural Networks Authors: Fanghui Song, Zhongjian Wang, Jiebao Sun

5. Recurrent Reinforcement Learning with Memoroids Authors: Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, Amanda Prorok

6. Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion Authors: Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat

7. Multivariate Diffusion Transformer with Decoupled Attention for High-Fidelity Mask-Text Collaborative Facial Generation Authors: Yushe Cao, Dianxi Shi, Xing Fu, Xuechao Zou, Haikuo Peng, Xueqi Li, Chun Yu, Junliang Xing

8. Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang, Haichao Zhang, Ran Zhou, Tan Wang, Chunnan Yao, Guoyao Li, Rui Cai, Yihan Cao, Ruijie Jiang, Fedor Borisyuk, Jianqiang Shen, Jingwei Wu, Ramya Korlakai Vinayak

9. Disentangling Optimization Scale from Preference Scale in DPO Authors: Ivan Kruzhilov

Efficiency, Compression, and Large-Scale Training (15)

1. GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets Authors: Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han

2. FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models Authors: Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, Yeseong Kim

3. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization Authors: Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng

4. Auditing Invisible Weight Updates with Reference Traces Authors: Zekai Shang

5. Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs Authors: Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng, Zhuang Ma, Anandharaju Durai Raju, Yao Wang, Xing Huang, Hei Yi Mak, Shadan Golestan, Hoang Le, Yonghan Dong, Wei Guo, Yaoyuan Wang

6. LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics Authors: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

7. Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization Authors: Francesco Pio Monaco, Elia Cunegatti, Flavio Vella, Giovanni Iacca

8. PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference Authors: Junjie Liu, Shengyuan Ye, Xu Chen

9. Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models Authors: Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke, Pradeep Dubey

10. Unifying Detection and Adaptation in Task-Free Continual Learning Authors: Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo

11. ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour Authors: Jianhang Xie, Sicheng Tan, Vishnu Naresh Boddeti, Zhichao Lu

12. Group-Shared Low-Rank Approximation for Mobile-Efficient Pointwise Convolutions in Large-Kernel CNNs Authors: Hao Luo, Yiting Yang, Wenyi Zhao, Man Jiang, Zhijun Lin, Ghulam Mohiuddin, Ting Jiang, Kunming Luo, Zihao Zhang, Qingsen Yan, Guoqing Wang, Wei Dong, Peng Wang

13. AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air Authors: Shiyi Yang, Xiaoxue Yu, Rongpeng Li, Jianhang Zhu, Zhifeng Zhao, Honggang Zhang

14. Adversarial Training Without Input Gradients via Low-Rank Householder Expansions Authors: Tiana C. Johnson, Donsub Rim

15. Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy Authors: Haozhe Hu, Hao Wu, Anhao Zhao, Longwei Ding, Peiran Yin, Yunpu Ma, Xiaoyu Shen

MoE Training (1)

1. MM-Spectrum: Multimodal Multi-spectral Molecular Structural Elucidation with a Stable MoE Framework

ArXiv ID: 2608.27286

Primary Topic: MoE Training

Authors: Hai-tao Yu, Nan Min, Zheng Fang, Hongyu Zhan, Yusen Tan, Yuhan Wang, Jun Xia

Abstract: Inferring molecular structures from multimodal spectroscopic measurements requires integrating complementary yet highly heterogeneous signals. However, the common paradigm of directly concatenating multispectral sequences can exhibit anomalous performance degradation, primarily due to pronounced heterogeneity and the resulting multimodal imbalance across modalities. As a remedy, we propose MM-Spectrum, a sparse Mixture-of-Experts framework tailored for multimodal multispectral spectra-to-structure elucidation. To better match the information characteristics under multispectral imbalance, MM-Spectrum introduces an explicit modality-aware routing mechanism that exposes spectral identity to the router in addition to token content representations. Moreover, it incorporates shared and interaction experts, together with heterogeneous expert capacities, to extract multispectral modality-unique and cross-modal synergistic information while suppressing noise-induced interference. Across full-modality, bimodal, and missing-modality settings on molecular structural elucidation, MM-Spectrum achieves consistent and substantial improvements, supported by ablation studies and interpretability analyses.

Comment: Explicit modality-aware expert routing addresses imbalance between heterogeneous input modalities.

Topic Match: The router and heterogeneous expert-capacity design are substantive MoE contributions, although their validation is specific to molecular spectra.

Relevance: 7 Novelty: 6

Large-Scale Training Systems and Efficiency (3)

1. Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

ArXiv ID: 2606.25975

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Sergei Kudriashov, Maxim Rakhuba

Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optimization dynamics. A notable example is Muon, which performs steepest descent under the spectral norm constraint. We take the next step and introduce Tensorion, a tensor-aware optimizer that extends Muon's constrained optimization perspective from matrices to higher-order tensors. Tensorion is built around a linear minimization oracle (LMO) over a tensor norm ball. The norm is carefully chosen to balance two objectives: tightly bounding the tensor spectral norm, while still keeping the LMO tractable. This LMO becomes computable because it reduces to operations on adaptively selected unfolding matrices. Notably, when restricted to order-2 tensors (i.e., matrices), Tensorion recovers Muon exactly. Experiments on tensor-based computer vision problems suggest that Tensorion can offer improved convergence behavior and more stable gradient updates compared with Adam-based and existing tensor-aware baselines in the evaluated settings.

Comment: Extends Muon's constrained updates to higher-order tensors through tractable linear minimization over adaptively selected unfoldings.

Topic Match: A new structured optimizer directly fits training algorithms; reported validation is on tensor-based vision problems, leaving large-scale pretraining benefits unestablished.

Relevance: 8 Novelty: 8

2. Performance Foundations of Parallel & Distributed Reasoning Language Models

ArXiv ID: 2608.27046

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Maciej Besta, Leonard Schmidt, Lara Nonino, Robert Gerstenberger, Pierre Pang, Patrik Okanovic, Ales Kubicek, Tiancheng Chen, Baraq Lipshitz, Torsten Hoefler

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) and other RL-style post-training paradigms have been used for aligning large language models (LLMs) with reasoning standards. The resulting recent Reasoning Language Models (RLMs) such as DeepSeek-R1, o3, and Kimi k1.5 show that such RL-style post-training ("RL-for-LLMs") can substantially improve chain-of-thought reasoning, long-horizon planning, and self-correction. However, the computational footprint of these systems is massive: state-of-the-art RLM training requires millions of GPU-hours and tightly coupled multi-model pipelines that stress modern hardware far beyond classical supervised LLM training. This makes RLM training as much a parallel and distributed systems problem as an algorithmic one. In this work, to facilitate developing RLMs that are simultaneously high-performance, scalable, and cost-effective, we first systematize the RL-for-LLM paradigm and provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants. Second, we develop a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs, covering both traditional techniques (data, tensor, pipeline, sequence, context, and expert parallelism) as well as novel forms of parallelism and optimization techniques for multi-model RLM training, for example disaggregated placement, stage fusion, hybrid parallelism, and asynchronous execution. We harness the work-depth model of parallel computing to make our taxonomy and its insights rigorous and portable. Finally, we analyze existing RLM frameworks and we distill practical guidelines and outline open research directions for building scalable, fast, and cost-effective RLMs.

Comment: Work-depth modeling analyzes parallelism and scheduling costs in multi-model LLM training.

Topic Match: Distributed-training performance analysis is central, including model placement, stage fusion, and asynchronous execution; the post-training setting does not displace that systems contribution.

Relevance: 9 Novelty: 6

3. Beyond Client Averaging: A Client-Independent Second-Order Stationary-Bias Component in Stochastic SCAFFOLD

ArXiv ID: 2608.26765

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Yi-Ping Tang, Guan-Ju Peng

Abstract: Existing constant-step analysis of stochastic \Scaf{} identifies a leading $O(γ/N)$ stationary mean bias and shows that higher-order bias can persist as the client count increases, but does not identify the first client-independent contribution at coefficient level. For full-participation stochastic \Scaf{} with one-dimensional homogeneous clients, fixed local-step count $H$, and bounded additive gradient noise, we prove, uniformly over $N\ge2$, $$ \begin{aligned} \mathbb{E}{π $$ Hence client averaging suppresses the leading $O(γ/N)$ bias but does not remove the client-independent $O(γ^2)$ component when its coefficient is nonzero. The mechanism is indirect: although the direct control contribution cancels pathwise in the linear global average, the controls still alter within-round local trajectories and their second moments. Fresh gradient noise and persistent control fluctuations therefore generate local second-moment corrections that nonquadratic curvature converts into stationary mean bias. The coefficient vanishes for quadratic objectives. Numerical experiments are consistent with the predicted coefficient, its persistence as client count increases, and the stated joint remainder. The result is restricted to the one-dimensional homogeneous fixed-$H$ setting.}}[x]-x^\star ={}& -\frac{f'''(x^\star)σ^2}{4f''(x^\star)^2}\fracγ{N}\ &- \frac{f'''(x^\star)σ^2}{12f''(x^\star)} \frac{(H-1)(5H-1)}{H}γ^2 +O_H!\left(\frac{γ^2}{N}+γ^3\right). \end{aligned

Comment: A stationary-bias expansion identifies an O(gamma^2) component that additional client averaging cannot eliminate.

Topic Match: Distributed-optimizer convergence is central, with explicit client-count and local-step dependence, although the results cover only scalar homogeneous SCAFFOLD clients.

Relevance: 6 Novelty: 6

Architecture and Training Dynamics (9)

1. Squeezing More from Limited Data with Recursive Transformers

ArXiv ID: 2608.26973

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Serdar Gülbahar, Lukas Edman, Alexander Fraser

Abstract: Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.

Comment: Shared-block recursion decouples pretraining compute from parameter count under fixed data budgets.

Topic Match: Depth-wise parameter sharing and data-limited scaling make architecture_training primary; factorized embeddings also improve parameter efficiency.

Relevance: 8 Novelty: 6

2. Making Latent Evolution Explicit: Operator-Structured Transitions for World Action Models

ArXiv ID: 2608.27259

Primary Topic: Architecture and Training Dynamics

Authors: Xiaoxiao Lu, Yunlong Dong, Jiahao Shi, Ye Yuan

Abstract: World Action Models (WAMs) augment robot policies by predicting how task-relevant scene states may evolve under interaction. Recent WAMs increasingly perform such prediction in latent representation spaces, avoiding full appearance-level generation while preserving control-relevant information. Yet latent transitions are commonly realized with Transformer-based predictors whose inductive structure is centered on token interaction rather than temporal evolution. We study transition realization as an architectural choice distinct from predictive representation and prediction-policy coupling. We introduce the Latent Evolution Operator Network (LEON), which models latent evolution in a learned observable space through context-modulated operator-based propagation and additive forcing. Grounded in the controlled Koopman generator view of evolution, LEON organizes context-dependent transition variation around a shared evolution-operator structure while retaining a complementary path for additive change. Controlled dynamical systems verify the resulting evolution-specific inductive bias and the complementary roles of operator propagation and forcing. Across two WAM formulations that integrate latent prediction into the policy differently, LEON improves closed-loop performance and robustness while remaining effective under full transition replacement. These results establish transition realization as a consequential architectural choice in latent WAMs.

Comment: Introduces latent transitions built from context-modulated evolution operators and additive forcing.

Topic Match: The core contribution is an operator-based transition architecture with mechanistic validation, although its demonstrated scope is robotic world-action models.

Relevance: 7 Novelty: 7

3. When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging

ArXiv ID: 2608.27518

Primary Topic: Architecture and Training Dynamics

Authors: Shangge Liu, Yuehan Yin, Yinghuan Shi, Lei Wang, Wenbin Li

Abstract: Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as \textit{task interference} and reduce it to a common layer-wise Frobenius inner product $\langle ΔW_\ell, J_\ell(x)\rangle_F$. This quantity, in turn, is utilized to expose the role of the optimizer. We theoretically derive an upper bound that isolates the spectral norm $|ΔW_\ell|_2$ as an optimizer-controllable factor of task interference, and a per-mode analysis shows that this bound tracks the dominant part of the empirical interference. Specifically, we then identify the recent Muon optimizer as a mechanism that regulates this factor by construction. Our work reveals that its elegant control on spectral norm tightens the interference bound for both CL and MM, positioning Muon as a principled optimizer-centric approach complementary to existing solutions. Our theoretcal analysis is well validated by experimental results. Replacing the AdamW optimizer with Muon improves accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. For continual learning, Muon also delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark.

Comment: Links optimizer-controlled update spectral norms to cross-task interference, explaining Muon's effect on forgetting and merging.

Topic Match: The core contribution is a mechanistic analysis of optimizer geometry and training interference, with validation focused on continual learning and model merging.

Relevance: 7 Novelty: 7

4. COFM: Consistent Optimal Transport Flow Matching via Partially Input Convex Neural Networks

ArXiv ID: 2511.06042

Primary Topic: Architecture and Training Dynamics

Authors: Fanghui Song, Zhongjian Wang, Jiebao Sun

Abstract: Optimal transport (OT) provides a principled framework for learning mappings between probability distributions, and has found broad applications in generative modeling, inverse problems and scientific computing. Recently, flow matching methods have emerged as an efficient paradigm for learning continuous-time transport dynamics. However, existing OT-based flow matching methods often suffer from either high computational cost due to inner optimization or limited consistency. Moreover, it remains challenging to design neural architectures that can simultaneously guarantee convexity, stability, and efficient transport learning. In this paper, we propose a framework for consistent optimal transport flow matching. Specifically, we parameterize the transport potential using partially input convex neural networks (PICNN), and incorporate a Hamilton-Jacobi residual into the training objective to enforce dynamical consistency of the learned flow. This design enables a unified formulation that supports both one-step transport and multi-step ODE-based sampling, without requiring costly inner optimization. Extensive experiments on benchmark datasets demonstrate that the proposed method achieves competitive performance compared with existing OT-based and flow matching approaches, while maintaining favorable computational efficiency. In particular, under the D=256 benchmark, COFM achieves more than a 2x reduction in L^2-UVP compared with state-of-the-art (SOTA) models, while requiring approximately 9x less computational time. These results suggest that combining convex potential structures with HJ-based dynamical regularization provides an effective framework for scalable and geometrically consistent transport learning.

Comment: Combines partially input-convex potentials with Hamilton-Jacobi regularization to train consistent transport without inner optimization.

Topic Match: Neural parameterization and consistency-enforcing training objectives directly change flow-matching mechanics; applicability to large-model training remains un demonstrated.

Relevance: 7 Novelty: 7

5. Recurrent Reinforcement Learning with Memoroids

ArXiv ID: 2402.09900

Primary Topic: Architecture and Training Dynamics

Authors: Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, Amanda Prorok

Abstract: Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models called Linear Recurrent Models. We discover that the recurrent update of these models resembles a monoid, leading us to reformulate existing models using a novel monoid-based framework that we call memoroids. We revisit the traditional approach to batching in recurrent reinforcement learning, highlighting theoretical and empirical deficiencies. We leverage memoroids to propose a batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in reinforcement learning.

Comment: Reformulates linear recurrent updates as monoids to enable a new sequence-batching method.

Topic Match: The algebraic formulation and batching mechanism concern recurrent sequence computation directly, with validation focused on reinforcement learning.

Relevance: 7 Novelty: 7

6. Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

ArXiv ID: 2608.26879

Primary Topic: Architecture and Training Dynamics

Authors: Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat

Abstract: Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.

Comment: Asymmetric attention preserves the dominant modality while weaker modalities attend to it, preventing fusion-induced pathway degradation.

Topic Match: The paper diagnoses an architectural failure mechanism and changes cross-modal attention to address it; evidence remains concentrated in multimodal fusion benchmarks.

Relevance: 7 Novelty: 6

7. Multivariate Diffusion Transformer with Decoupled Attention for High-Fidelity Mask-Text Collaborative Facial Generation

ArXiv ID: 2511.12631

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Yushe Cao, Dianxi Shi, Xing Fu, Xuechao Zou, Haikuo Peng, Xueqi Li, Chun Yu, Junliang Xing

Abstract: While significant progress has been achieved in multimodal facial generation using semantic masks and textual descriptions, conventional feature fusion approaches often fail to enable effective cross-modal interactions, thereby leading to suboptimal generation outcomes. To address this challenge, we introduce MDiTFace--a customized diffusion transformer framework that employs a unified tokenization strategy to process semantic mask and text inputs, eliminating discrepancies between heterogeneous modality representations. The framework facilitates comprehensive multimodal feature interaction through stacked, newly designed multivariate transformer blocks that process all conditions synchronously. Additionally, we design a novel decoupled attention mechanism by dissociating implicit dependencies between mask tokens and temporal embeddings. This mechanism segregates internal computations into dynamic and static pathways, enabling caching and reuse of features computed in static pathways after initial calculation, thereby reducing additional computational overhead introduced by mask condition by over 94% while maintaining performance. Extensive experiments demonstrate that MDiTFace significantly outperforms other competing methods in terms of both facial fidelity and conditional consistency.

Comment: Decoupled attention separates static and timestep-dependent paths so mask-conditioning features can be cached.

Topic Match: Explicit attention factorization supplies a concrete architectural mechanism with a direct caching benefit, although its demonstrated scope is facial generation.

Relevance: 7 Novelty: 6

8. Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation

ArXiv ID: 2604.02324

Primary Topic: Architecture and Training Dynamics

Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang, Haichao Zhang, Ran Zhou, Tan Wang, Chunnan Yao, Guoyao Li, Rui Cai, Yihan Cao, Ruijie Jiang, Fedor Borisyuk, Jianqiang Shen, Jingwei Wu, Ramya Korlakai Vinayak

Abstract: Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that \emph{token initialization} is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the \emph{Grounded Token Initialization Hypothesis}: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension.

Comment: Diagnoses persistent embedding collapse from mean initialization and introduces semantically grounded initialization for new vocabulary.

Topic Match: Initialization-induced degeneracy connects to training dynamics; the method and evidence remain centered on vocabulary adaptation for generative recommendation.

Relevance: 6 Novelty: 6

9. Disentangling Optimization Scale from Preference Scale in DPO

ArXiv ID: 2608.27032

Primary Topic: Architecture and Training Dynamics

Authors: Ivan Kruzhilov

Abstract: Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $β$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate value, and decreases again for larger $β$. Moreover, standard DPO loss values are not comparable across $β$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $β$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $β>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $β\to0$ endpoint that reduces to a linear preference-margin objective.

Comment: An argmin-equivalent centered-softplus objective separates preference-noise scaling from effective learning-rate scaling.

Topic Match: Loss-scale coupling and optimization dynamics provide a substantive topic connection, but the analysis and reformulation remain specific to DPO post-training.

Relevance: 6 Novelty: 6

Efficiency, Compression, and Large-Scale Training (15)

1. GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets

ArXiv ID: 2605.18475

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han

Abstract: Mixed-precision quantization improves the budget--accuracy trade-off for large language models (LLMs) by allocating more bits to sensitive modules. However, automating this allocation at LLM scale faces a unique combination of constraints: learnable approaches require quantization-aware training, which is infeasible for billion-parameter models; training-free alternatives rely on static proxy metrics that miss cross-module interactions and must be recomputed per target budget; and search-based methods are expensive without guaranteeing exact budget compliance. We propose GAMMA, a quantizer-agnostic framework that learns module-wise precision preferences entirely within a post-training pipeline. GAMMA optimizes a teacher-forced hidden-state reconstruction objective under an augmented Lagrangian constraint, and projects the learned preferences into exact budget-feasible discrete assignments via integer programming. A key property is score reuse: because the learned preferences encode a stable sensitivity ranking rather than budget-specific weights, a single training run serves arbitrary deployment targets by re-solving only the integer program, reducing per-budget adaptation from hours to a few minutes. Across Llama and Qwen models (8B--32B), GAMMA outperforms both fixed-precision baselines (up to +12.99 Avg.) and search-based mixed-precision methods (up to +7.00 Avg.), and can match fixed 3-bit quality at 2.5-bit average precision, enabling deployment at substantially smaller memory footprints.

Comment: Reusable module precision scores and integer programming enable mixed-precision LLM quantization with exact budget compliance.

Topic Match: The core contribution is quantization bit allocation that reduces model memory and reuses learned preferences across budgets.

Relevance: 9 Novelty: 7

2. FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

ArXiv ID: 2608.26676

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, Yeseong Kim

Abstract: Pruning is a practical approach to compress large language models (LLMs), but it can amplify text degeneration, especially repetition loops, even when perplexity and task accuracy remain largely unchanged. In this work, we present a token-level analysis of this failure mode by viewing decoding as a dynamical process that enters and persists in a small set of recurrent contexts. Our analysis decomposes degeneration into loop entry risk and loop persistence, and shows that persistence is controlled by the escape mass assigned to plausible alternatives within the token sampling set. Motivated by these findings, we propose two token-level guidance objectives for post-pruning fine-tuning. FOCUS reweights distillation toward high-confidence teacher regions to suppress leakage, while RePAIR uses onset-centered positive/negative continuation pairs with a margin loss to promote plausible alternatives and prevent early commitment to repetition loops. Experiments on open-ended continuation and instruction-based generation show that both methods consistently reduce repetition and improve generation quality.

Comment: Links pruning-induced repetition to loop-entry risk and escape probability, then designs token-level recovery objectives.

Topic Match: Directly addresses a quality failure caused by LLM compression and derives post-pruning training objectives from its mechanism.

Relevance: 9 Novelty: 7

3. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

ArXiv ID: 2608.27513

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng

Abstract: Softmax attention stores key and value vectors for every preceding token, causing inference memory to grow with sequence length. Recent language models incorporating Gated DeltaNet (GDN) or Kimi Delta Attention (KDA) reduce this cost by replacing the KV cache in most layers with fixed-size recurrent states. However, these recurrent states are commonly stored in FP32 and consume substantial GPU memory; their updates are memory-bandwidth bound and contribute significantly to decoding latency. To our knowledge, we are the first to study post-training quantization of recurrent states in GDN and KDA based language models. We find that uniform quantization provides a poor accuracy--storage trade-off: INT8 and FP8 already degrade accuracy on complex reasoning tasks, while INT4 and NVFP4 reduce it to near zero. We further find that most quantization-error energy is concentrated in a small subset of channels and that the relative decay strength of state channels remains stable across prompts and tasks. Motivated by these findings, DAMP uses both quantization-error energy and decay-based persistence to identify high-risk channels during offline calibration. It stores these channels at higher precision and the remainder in INT8. We evaluate DAMP on Qwen3.6-35B and Kimi-Linear-48B across six benchmarks covering mathematical reasoning, general reasoning, and code generation. At 9.9 bits per state value, DAMP maintains average accuracy close to the FP32 baseline. DAMP reduces recurrent-state storage by 69.1%, accelerates the recurrent-state update kernel by up to 2.01x, and lowers full-model TPOT by up to 10.9%.

Comment: Allocates recurrent-state precision using channel error energy and decay persistence, reducing state storage by 69.1%.

Topic Match: The core quantization mechanism directly reduces recurrent-state memory and decoding cost in large language models.

Relevance: 9 Novelty: 7

4. Auditing Invisible Weight Updates with Reference Traces

ArXiv ID: 2607.09800

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Zekai Shang

Abstract: Direct low-precision write-back can erase nonzero optimizer proposals. We ask what a high-precision reference trace establishes before a low-precision run. The exact target-code event is auditable coordinatewise on a realized target trajectory; pre-run aggregate projection also assumes the reference remains a useful counterfactual. In a controlled two-layer grid, 55/72 cells have measured and predicted post-initialization crossings: times span $384\times$, 52/55 are within 15\%, and 4/72 differ in category. Matched decoder experiments show stochastic rather than nearest write-back recovers most of the loss gap. A prospective analytic-grid E4M3 audit reuses one fp32 trace across three unseen NeoX-style seeds. It passes absolute-accuracy and skill gates (macro RMSE 0.00858) but fails directional specificity. In a target-outcome-blind comparison, a historical template has lower descriptive RMSE (0.00360) than the predeclared source predictor (0.00438); a post-outcome decomposition assigns 99.65\% of variation to common time, while a privileged matched-reference correction reaches 0.00283. Persistent-native Study~1 pairs three seeds across two schedules. Five cells are canonical; a manual sixth lacks canonical process identity, so the registered result remains inconclusive. A retrospective protocol-deviation analysis is negative because the complete constant-mid cohort is disjoint from the recovered cosine-restart cell. Study~2 reports mean full-SR/dead-zone-SR recoveries of 0.9766/0.9777 and a ratio of 1.0012, a policy contrast rather than causal mediation. Simulated-INT3 Study~3 replays six checkpoints and observes a 7.3071-nat (69.71\%) validation-loss reduction in one fixed seed. Exact events and write-back effects are auditable, but aggregate forecasts can reflect shared time rather than source-specific transfer.

Comment: Audits optimizer updates erased by low-precision write-back and measures recovery through stochastic rounding.

Topic Match: Training quantization is central: write-back precision determines whether optimizer updates survive and how learning progresses. Aggregate forecast results remain mixed.

Relevance: 9 Novelty: 6

5. Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs

ArXiv ID: 2608.26581

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng, Zhuang Ma, Anandharaju Durai Raju, Yao Wang, Xing Huang, Hei Yi Mak, Shadan Golestan, Hoang Le, Yonghan Dong, Wei Guo, Yaoyuan Wang

Abstract: Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.

Comment: Adds a quantized residual pathway to reconstruct activations and recover accuracy under ultra-low-bit quantization.

Topic Match: Activation-error reconstruction directly targets the accuracy-cost trade-off of 4-bit computation in large multimodal models.

Relevance: 9 Novelty: 6

6. LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

ArXiv ID: 2608.27395

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

Abstract: Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.

Comment: Combines single-encoder video pretraining with random token dropping, reporting 5.6-20.8x less pretraining compute at matched epochs.

Topic Match: Reducing general-purpose visual pretraining cost is central, with architectural simplification and causal-attention compatibility; the collapse-free objective is inherited from LeJEPA.

Relevance: 8 Novelty: 6

7. Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

ArXiv ID: 2603.16105

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Francesco Pio Monaco, Elia Cunegatti, Flavio Vella, Giovanni Iacca

Abstract: Post-training model compression is essential for enhancing the portability of Large Language Models (LLMs) while preserving their performance. While several compression approaches have been proposed, less emphasis has been placed on selecting the most suitable set of data (the so-called \emph{calibration data}) for finding the compressed model configuration. The choice of calibration data is a critical step in preserving model capabilities both intra- and inter-tasks. In this work, we address the challenge of identifying high-performance calibration sets for both pruning and quantization by analyzing intrinsic data properties rather than model-specific signals. We introduce ZipCal, a model-agnostic data curation strategy that maximizes lexical diversity based on Zipfian power laws. Experiments demonstrate that our method outperforms standard uniform random sampling across various pruning benchmarks. Notably, it also performs on par, in terms of downstream performance, with a state-of-the-art method that relies on model perplexity. The latter becomes prohibitively expensive for large-scale models and datasets, while ZipCal is on average $\sim$240$\times$ faster due to its tractable linear complexity. We make the code and the experiments available at https://github.com/FrancescoMonaco/ZipCal.

Comment: Zipf-based lexical diversity selects compression calibration data without costly model-dependent scoring.

Topic Match: The new calibration-selection method directly reduces preparation cost for LLM pruning and quantization, making compression efficiency its central contribution.

Relevance: 8 Novelty: 6

8. PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

ArXiv ID: 2608.27206

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Junjie Liu, Shengyuan Ye, Xu Chen

Abstract: Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.

Comment: Coordinated pixel compression and dual-attention token pruning reduce computation across both the vision encoder and LLM.

Topic Match: Its core contribution is a token-reduction mechanism that accelerates VLM inference, with visual tasks serving as evaluations of the efficiency method.

Relevance: 8 Novelty: 6

9. Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models

ArXiv ID: 2508.06753

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke, Pradeep Dubey

Abstract: The advent of ultra-low-bit LLM models, approaching the perplexity and task accuracy of their full precision counterparts, is ushering in a new era of LLM inference. While these advances promise models that are cost-effective regarding latency, memory, throughput, and energy consumption, the efficiency of runtimes for deploying ultra-low-bit models remains under-explored. In this work, we take a bottom-up approach: we first implement 2-bit microkernels for modern CPUs, achieving close-to-roofline performance. We integrate these microkernels into LLM inference pipelines and present end-to-end results with 2-bit models, outperforming the state-of-the-art (SOTA) bitnet.cpp runtime by 2.2$\times$, and deliver up to 7$\times$ speedup compared to 16-bit inference. We extend this work to Intel Xe2 GPUs where we implement mixed-precision, 2-bit kernels, and show their performance to be close-to-optimal. We integrated the GPU kernels in the vLLM framework and evaluated end-to-end inference for a range of models and Xe2 GPUs. We obtain up to 6.7$\times$ speedup compared to the 16-bit pipeline, pushing the envelope of LLM inference.

Comment: Near-roofline 2-bit CPU and mixed-precision GPU kernels accelerate ultra-low-bit LLM inference.

Topic Match: Specialized low-bit kernel implementations materially reduce model execution cost, making computational efficiency the primary contribution.

Relevance: 8 Novelty: 6

10. Unifying Detection and Adaptation in Task-Free Continual Learning

ArXiv ID: 2608.27070

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo

Abstract: To mitigate catastrophic forgetting in downstream continual learning (CL) for large language models (LLMs), existing methods typically constrain parameter updates or introduce task-specific adaptation modules. However, these methods often rely on explicit task boundaries during training, limiting their applicability to realistic task-free scenarios. In this paper, we propose a \textbf{Fi}sher-guided \textbf{uni}fied (\textbf{FiUni}) framework for batch-level task detection and parameter-efficient continual adaptation. FiUni is motivated by a key observation about the Fisher information matrix (FIM) of pre-trained models: the orthogonality among the principal subspaces of its Kronecker-Factored Approximate Curvature (K-FAC) approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks. Based on this observation, FiUni constructs FIM-derived frozen subspaces to guide low-rank adaptation (LoRA), while matching the Fisher principal subspace of each incoming batch window with historical subspaces. This enables FiUni to adaptively determine whether to reuse existing knowledge, expand a related subspace, or create a new subspace, dynamically balancing knowledge sharing and task isolation. Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.

Comment: Uses Fisher-derived subspaces to guide LoRA and dynamically reuse, expand, or create adaptation capacity.

Topic Match: The substantive match is a new low-rank adaptation mechanism that reduces trainable parameters, with scope focused on downstream continual learning.

Relevance: 7 Novelty: 7

11. ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

ArXiv ID: 2607.23478

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Jianhang Xie, Sicheng Tan, Vishnu Naresh Boddeti, Zhichao Lu

Abstract: Fully homomorphic encryption (FHE) lets a server run inference on encrypted data with strong privacy guarantees, but running a Transformer under FHE is expensive. Its non-linear operations, such as softmax, normalization, and activation, must be replaced with polynomial approximations that the CKKS scheme supports, and the depth of these approximations dominates inference cost. Existing FHE Transformers use hand-tuned approximation settings, such as iteration count and polynomial degree, applied uniformly across layers, models, and tasks. Hand-tuning is slow and error-prone. Even a single uniform setting has about $10^7$ choices, and manual search cannot exploit layer-wise variation. AutoFHE, the only automated method with multi-objective search, targets ReLU-only CNNs and needs full fine-tuning per candidate, which is too costly for Transformers. Per-layer settings also push the search space to about $10^{85}$ for BERT and ViT and $10^{228}$ for LLaMA3, beyond both manual and fine-tuning-based search. We present ATLAS, a training-free framework that automates this search by treating each layer's approximation setting as a multi-objective optimization over latency and accuracy. The problem is hard: the decision space is large (96 or 256 variables), each configuration takes 70 to 1,000 seconds to evaluate even in cleartext, and 85 to 90 percent of configurations are invalid. ATLAS handles this with a two-stage optimization strategy and a surrogate model, completing the search in about one hour. Compared to an iterative softmax baseline, ATLAS cuts multiplicative depth and end-to-end latency by about 35 percent with little accuracy loss, and works across encoder-only, decoder-only, and vision Transformers, complementing parallel work on packing and matrix multiplication.

Comment: Surrogate-guided search selects layer-specific polynomial approximations to reduce encrypted Transformer inference depth and latency.

Topic Match: Automated approximation directly reduces Transformer computation, although the optimization targets the specialized constraints of homomorphic inference.

Relevance: 7 Novelty: 7

12. Group-Shared Low-Rank Approximation for Mobile-Efficient Pointwise Convolutions in Large-Kernel CNNs

ArXiv ID: 2608.26069

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Hao Luo, Yiting Yang, Wenyi Zhao, Man Jiang, Zhijun Lin, Ghulam Mohiuddin, Ting Jiang, Kunming Luo, Zihao Zhang, Qingsen Yan, Guoqing Wang, Wei Dong, Peng Wang

Abstract: Large-kernel Convolutional Neural Networks (CNNs) deliver remarkable performance in vision tasks by significantly expanding receptive fields, yet their quadratic parameter growth critically impedes storage-efficient edge deployment. While existing efficient architectures adopt parameter-efficient depthwise separable convolution backbones that leverage techniques like low-rank approximation and weight sharing to compress depthwise convolutions, we identify a critical oversight: pointwise convolutions dominate parameter volume (>87% in models like RepLKNet-31B) and constitute the primary deployment bottleneck on resource-constrained edge devices. This results in prohibitive storage costs and severe memory-loading constraints on resource-limited devices (e.g., smartphones with 4-12 GB Random Access Memory (RAM)). To overcome this, we propose Channel Group-Shared (CGS) low-rank approximation, a novel Singular Value Decomposition (SVD)-based parameter-sharing strategy. CGS constructs a structured low-rank paradigm isomorphic to SVD decomposition, comprising shared (high-parameter-cost) down/up-projection matrices across channel groups within a layer and channel-group-specific (low-parameter-cost) scalable diagonal matrices. This group-sharing design achieves significant parameter reduction. Extensive experiments demonstrate that large-kernel CNNs (RepLKNet, ConvNeXt, SLaK) enhanced with CGS strike an empirically favorable balance between competitive performance and substantially reduced storage costs. Crucially, by alleviating storage constraints, reducing memory bandwidth pressure during loading, and minimizing model loading latency, CGS enables the feasible deployment of pre-trained large-kernel CNN models on edge devices, thereby bridging the gap between high-performance vision models and practical edge deployment.

Comment: Shares low-rank projection factors across channel groups to compress parameter-dominant pointwise convolutions.

Topic Match: Structured compression is the core contribution, with a narrower focus on CNN storage, memory bandwidth, and loading costs.

Relevance: 7 Novelty: 6

13. AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air

ArXiv ID: 2507.11515

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Shiyi Yang, Xiaoxue Yu, Rongpeng Li, Jianhang Zhu, Zhifeng Zhao, Honggang Zhang

Abstract: Operating Large Language Models (LLMs) on edge devices is increasingly challenged by limited communication bandwidth and strained computational and memory costs. Thus, cloud-assisted remote fine-tuning becomes indispensable. Nevertheless, existing Low-Rank Adaptation (LoRA) approaches typically employ fixed or heuristic rank configurations, and the subsequent over-the-air transmission of all LoRA parameters could be rather inefficient. To address this limitation, we develop AirLLM, a hierarchical diffusion policy framework for communication-aware LoRA adaptation. Specifically, AirLLM models the rank configuration as a structured action vector that spans all LoRA-inserted projections. To solve the underlying high-dimensional sequential decision-making problem, a Proximal Policy Optimization (PPO) agent generates coarse-grained decisions by jointly observing wireless states and linguistic complexity, which are then refined via Denoising Diffusion Implicit Models (DDIM) to produce high-resolution, task- and channel-adaptive rank vectors. The two modules are optimized alternatively, with the DDIM trained under the Classifier-Free Guidance (CFG) paradigm to maintain alignment with PPO rewards. Experiments under varying signal-to-noise ratios demonstrate that AirLLM consistently enhances fine-tuning performance while significantly reducing transmission costs, highlighting the effectiveness of reinforcement-driven, diffusion-refined rank adaptation for scalable and efficient remote fine-tuning over the air.

Comment: A hierarchical PPO and diffusion controller allocates LoRA ranks to balance fine-tuning quality against parameter-transmission cost.

Topic Match: Adaptive low-rank allocation directly targets fine-tuning efficiency, although the controller is specialized to wireless communication conditions.

Relevance: 7 Novelty: 6

14. Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

ArXiv ID: 2608.26963

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Tiana C. Johnson, Donsub Rim

Abstract: This work concerns adversarial training against the small-norm adversarial examples that arise from the inherent input instability of a trained deep neural network. Examples in this class are small as measured in the relative $\ell^2$-norm, and therefore lie in the neighborhood of the input on which the model acts approximately linearly, the regime in which the perturbation remains imperceptible. We first show that such examples can be computed directly from the trained network parameters, without input gradient iterations, by means of a linearization called the low-rank Householder expansion (LRHE). The expansion describes the composed affine map rather than any individual layer, and the directions it identifies are read from the activation pattern already available in the forward pass. We then propose a simple adversarial training scheme built on this construction. No differentiation with respect to the input is performed at any point: training requires only additional forward evaluations, with weight parameters updated by the standard backward pass, and the inner maximization of the usual min-max formulation is eliminated entirely. That such a regularizer exists is our main finding: the methods that dispense with the inner search all obtain their local geometry by differentiating with respect to the input, and we show this is not necessary. The regularizer costs the equivalent of $2.8$ PGD steps per epoch, an $8.7\times$ reduction relative to 40-step adversarial training on MNIST and below the cost of 3-step training. The resulting models match three-step PGD adversarial training for relative $\ell^2$ budgets $\varepsilon \le 0.02$ and 40-step training for $\varepsilon \le 0.012$, falling away beyond, consistent with the locality of the expansion.

Comment: Forward-pass low-rank linearization eliminates input-gradient search from adversarial training.

Topic Match: Reducing adversarial-training computation is the strongest connection; evidence is limited to local robustness on MNIST, leaving large-model applicability unestablished.

Relevance: 6 Novelty: 7

15. Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

ArXiv ID: 2606.09080

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Haozhe Hu, Hao Wu, Anhao Zhao, Longwei Ding, Peiran Yin, Yunpu Ma, Xiaoyu Shen

Abstract: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns. Despite sharing the same objective, these pruning approaches induce fundamentally different execution behaviors, causing realized speedups to depend heavily on hardware and kernel implementations. Consequently, the practical acceleration benefits of different pruning families remain poorly understood. In this work, we introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical \textbf{M}, \textbf{N}, and \textbf{K} dimensions of general matrix multiplication (GEMM). Leveraging this abstraction, we build a unified benchmarking framework that enables implementation-consistent comparison across the pruning design space and systematically characterizes the acceleration--quality Pareto frontier. Our results on Llama3.1-8B show that static depth pruning remains the strongest Pareto-optimal baseline and stays closest to its theoretical acceleration upper bound in memory-bounded scenarios. During prefill, the frontier transitions from static depth at low quality loss (0\%--4\%), to dynamic depth at moderate loss (5\%--16\%), and finally to static width pruning at higher loss levels (17\%--26\%). These findings establish the first unified view of the practical limits of pruning-based LLM acceleration and provide guidance for future pruning research. Code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim

Comment: GEMM-based analysis connects pruning structure with realized LLM inference acceleration.

Topic Match: Compression efficiency is the subject, but the core contribution is a taxonomy and benchmark, placing it under the evaluation-only exclusion.

Relevance: 6 Novelty: 6

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains