This is a remedial run for missed papers from 08/24/2026 to 08/24/2026.

Results generated on 09/14/2026.

Personalized Daily ArXiv Papers 2026-08-25

_{Token counts are not reported for this run. 22 of 25 model calls succeeded, 7,306s of model wall clock.}

Topic Coverage:

Table of contents by topic:

Large-Scale Training Systems and Efficiency (3)

1. ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork Authors: Tim Beringer, Patrick Diem, Felix Wolf, Arya Mazaheri

2. Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining Authors: Yicheng Mao, Hongru Du

3. What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs Authors: Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai

Architecture and Training Dynamics (12)

1. ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation Authors: Zhongpan Tang

2. The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges Authors: Avinash Raju, Kai Zhang

3. Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization Authors: Qiaozhe Zhang, Jun Sun, Ruijie Zhang, Yingzhuang Liu

4. Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections Authors: Elian Iluk, Gil Ben-Artzi

5. Learning with Local Search MCMC Layers Authors: Germain Vivier-Ardisson, Mathieu Blondel, Axel Parmentier

6. The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions Authors: Alexander Heinlein, Johannes Taraz

7. Population-Scalable Multi-Agent World Modeling Authors: Renjie Zhao, Yuxiang Wu, Mingyu Zhang, Jiaxin Li, Sisi Li, He Li, Yimin Sheng, Tianxi Tan, Zhenkai Zhang, Jiao Liang, Jianyi Zhu, Yong-Lu Li

8. Wait, Wait, Wait... Why Do Reasoning Models Loop? Authors: Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, Dimitris Papailiopoulos

9. Dimensionless Controls of Plasticity Under Alternating Tasks: From Evolutionary Biology to Continual Learning Authors: Owen Skriloff

10. Group-Equivariant Poincaré Convolutional Networks Authors: Aiden Durrant, Rahul Baburajan, Georgios Leontidis

11. DuoGNN: Topology-aware Graph Neural Network with Homophily and Heterophily Interaction-Decoupling Authors: K. Mancini, I. Rekik

12. Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learning Authors: Chongyang Zhao, Dong Gong

Efficiency, Compression, and Large-Scale Training (15)

1. UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference Authors: Lang Zhou, Shuxuan Li, Zhuohao Li, Shi Liu, Wei-Shi Zheng, Zhilin Zhao

2. Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction Authors: Isaac, Li

3. Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode Authors: Tom Poperszky

4. Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs Authors: Yan Zhou, Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak, Marco Fumero, Francesco Locatello, David Alvarez-Melis

5. Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair Authors: Zehao Liu, Chuangchuang Fang, Yang Ren

6. PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression Authors: Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li

7. ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation Authors: Chengjie Lu, Tianchi Deng, Zhengqi He, Chengwen Luo, Xueliang Li

8. Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention Authors: Sergii Kozyrev, Davyd Maiboroda

9. Provably adaptive sampling with uniform and remasking discrete diffusion models Authors: Daniil Dmitriev, Zhihan Huang, Yuting Wei

10. Rethinking Reverse KL as Adaptive Entropy Distillation Authors: Shizhen Li, Zhiyu Shen, Yuyin Lu, Yunhe Pang, Jielin Song, Yanghui Rao, Fu Lee Wang

11. Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap Authors: Sathishkumar Sivashanmugam

12. Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling Authors: Ha Dinh, Xuan Duy Ta, Khoat Than, Khac-Hoai Nam Bui

13. E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models Authors: Taoyu Qian, Qi Wang, Daqian Shi, Yuanhao Jiang, Shang Gao, Hualong Yu

14. Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning Authors: Dongyue Wu, Tao Ma

15. When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving Authors: Wenjun Yu, Shuguang Han, Amelie Chi Zhou

Large-Scale Training Systems and Efficiency (3)

1. ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork

ArXiv ID: 2608.23840

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Tim Beringer, Patrick Diem, Felix Wolf, Arya Mazaheri

Abstract: Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.

Comment: An analytical performance model predicts sharded and multi-cluster training costs from model characteristics and hardware topology.

Topic Match: Modeling sharding, communication bottlenecks, and topology-dependent throughput directly informs large-scale training configuration.

Relevance: 9 Novelty: 6

2. Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

ArXiv ID: 2608.23922

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Yicheng Mao, Hongru Du

Abstract: Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy-based methods address this problem by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for larger-scale training. We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop this formulation using sparse second-order Scheffé response-surface models and construct model-robust $\mathcal{I}$-optimal designs for proxy data-mixing experiments. Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The Scheffé analysis shows that domain value is strongly relational: several domains that are weak under additive effects become favourable through pairwise interactions, especially through combinations with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects. In a simulation study calibrated to observed proxy-training responses, model-robust $\mathcal{I}$-optimal designs recover the relevant mixture ordering after removing about 25\% of the original proxy runs. These results suggest that LLM data mixing should be treated not only as a prediction problem, but also as an experimental-design problem in which the proxy mixtures themselves can be chosen to improve statistical efficiency.

Comment: Uses optimal experimental design to reduce proxy-training runs needed to choose pretraining data mixtures.

Topic Match: The main connection is efficient configuration of large-scale pretraining experiments; the reported 25% proxy-run reduction is simulation-based.

Relevance: 8 Novelty: 7

3. What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

ArXiv ID: 2608.00013

Primary Topic: Large-Scale Training Systems and Efficiency

Authors: Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai

Abstract: Choosing the right large language model (LLM) backbone is the most consequential decision when building a vision-language model (VLM), yet it remains fundamentally unprincipled: compute-based scaling laws fail to generalize across model families, and no framework exists for directly predicting VLM performance before training begins. We propose the Capability-Driven Multimodal Scaling Law, the first cross-family framework that predicts VLM benchmark accuracy from directly observable textual capability. Given a low-dimensional capability score $S$ extracted from LLM textual benchmarks via PCA, we model VLM performance as a function of $S$, with a per-backbone transfer rate and an absorption rate that quantifies data-scaling efficiency. To fit and validate the framework, we train over 150 VLMs on 34 LLMs spanning 7 model families under a strictly controlled recipe. Evaluations on more than 200 textual and 50 multimodal benchmarks show that the law accurately extrapolates transfer rate from models up to 8B parameters to 72B-scale backbones, predicts full VLM training trajectories with high fidelity, and generalizes to entirely held-out model families. Beyond the scaling law, our analysis surfaces actionable insights: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark-gaming behavior; base LLMs outperform instruction-tuned counterparts as VLM backbones due to higher absorption rates and lower data-scaling decay; and different model families occupy distinct positions in the transfer--absorption space. The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision. Code and data are available at https://github.com/wangq-dev/CDMScaling.

Comment: Cross-family scaling law predicting VLM performance from backbone textual capability, with per-backbone transfer and absorption rates fit over 150 trained VLMs.

Topic Match: Scaling-law work that informs how a training run is configured, though the configuration decision is multimodal backbone selection rather than the training system itself.

Relevance: 6 Novelty: 6

Architecture and Training Dynamics (12)

1. ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

ArXiv ID: 2608.23463

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Zhongpan Tang

Abstract: The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative models. We propose ProxyFormer, a general dual-stream architecture built upon proxy tokens. In each layer, fine-grained local features are compressed bottom-up into a small set of proxy states; expensive global interactions are performed only in the compressed proxy space; the globally contextualized proxies are then decompressed and injected top-down back into the local stream. Because the local stream persists across layers, fine-grained information that is not captured by one compression step remains accessible for later refinement, alleviating the irreversible information loss of conventional one-shot compression. We further introduce factorized multi-level compression/decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and a proxy-only KV-cache inference scheme. On a 16GB GPU with batch size 1, a standard decoder-only model can train sequences of only about 20K tokens, whereas ProxyFormer with a compression ratio of 64 extends the trainable sequence length to about 0.7M. A model trained with a 64K window retains 92%-95% retrieval accuracy on a multi-needle retrieval task with 1,048,576 tokens, and a model trained with an 8K window exceeds 94% accuracy when extrapolated to 256K tokens. Preliminary image-generation experiments demonstrate the feasibility of ProxyFormer for both pixel-space and latent-space flow matching.

Comment: A persistent local stream exchanges information through compressed global proxies, changing the attention architecture for long-context training.

Topic Match: The central contribution is a dual-stream architectural mechanism, with compression and proxy-only caching providing additional memory-efficiency benefits.

Relevance: 9 Novelty: 7

2. The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges

ArXiv ID: 2608.23916

Primary Topic: Architecture and Training Dynamics

Authors: Avinash Raju, Kai Zhang

Abstract: Denoising score matching trains diffusion models by regressing onto a conditional score, although generation ultimately requires the marginal score. The two objectives share the same population minimizer, but the conditional target remains random at fixed noisy state and introduces an irreducible excess in the training loss. We isolate this excess and show that, for a general corruption kernel under mild regularity assumptions, it is exactly the trace of the Fisher--Rao metric of the conditional endpoint family, integrated along the diffusion trajectory. This gives an exact conditional-variance decomposition of the denoising objective and identifies the information geometry observed in diffusion latent spaces as an intrinsic component of the training loss. We derive the result from a Schr"odinger bridge variational principle, in which the ideal objective arises as excess path-space relative entropy. For corruption diffusions, the Fisher term is proportional to the rate at which the noisy state loses mutual information about the clean data, separating the loss floor into an information flow determined by the data and a weight determined by the corruption schedule and objective. In the Gaussian case, this yields a closed form for the floor, recovers reparametrization invariance of the continuous-time objective, and relates its high-SNR divergence to the information dimension of the data. Finally, we show that raw losses obtained with different noise ranges or weightings need not rank models consistently because they contain different additive floors, and contrast the second-order geometry seen by training with the third-order conditional statistics entering numerical sampling error.

Comment: Derives corruption-dependent irreducible loss floors that affect interpretation of diffusion-model training losses.

Topic Match: Its Fisher-geometric decomposition explains the training objective and how corruption schedules and loss weighting affect observed losses.

Relevance: 8 Novelty: 7

3. Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization

ArXiv ID: 2510.07758

Primary Topic: Architecture and Training Dynamics

Authors: Qiaozhe Zhang, Jun Sun, Ruijie Zhang, Yingzhuang Liu

Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and generalization is not as strong as expected, and sometimes even contradiction occurs. To address this problem, a key observation in this paper is: what really matters for generalization is the average spread (or unevenness) of the spectrum of loss Hessian $\mathbf{H}$. For this reason, conventional sharpness measures, such as trace sharpness $\operatorname{tr}(\mathbf{H})$, which cares about the average value of the spectrum, or max-eigenvalue sharpness $λ_{\max}(\mathbf{H})$, which concerns the maximum spread of the spectrum, are not sufficient to well predict generalization. To characterize the average spread of the Hessian spectrum, we leverage the notion of Rényi entropy in information theory, which captures the unevenness of a probability vector and can thus be extended to a general non-negative vector, such as the Hessian spectrum at loss minima. Specifically, we propose Rényi sharpness, defined as the negative of the Rényi entropy of loss Hessian $\mathbf{H}$. Extensive experiments demonstrate that Rényi sharpness exhibits strong and consistent correlation with generalization in various scenarios. Moreover, two generalization bounds with respect to Rényi sharpness are established by exploiting its desirable reparametrization invariance property. Finally, as an initial attempt to exploit Rényi sharpness for regularization, Rényi Sharpness Aware Minimization (RSAM) is proposed, where a variant of Rényi sharpness is used as the regularizer. RSAM is competitive with state-of-the-art SAM algorithms and far better than conventional SAM based on max-eigenvalue sharpness.

Comment: Hessian-spectrum entropy links sharpness to generalization and motivates a revised SAM regularizer.

Topic Match: Loss-landscape analysis and optimization are central contributions, although applicability to large-scale pretraining is not established in the abstract.

Relevance: 8 Novelty: 7

4. Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

ArXiv ID: 2608.23794

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Elian Iluk, Gil Ben-Artzi

Abstract: Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.

Comment: Learned sparse channel supports with input-adaptive mixing replace quadratic-cost pointwise projections.

Topic Match: The central contribution is a sparse channel-mixing architecture with measured compute savings; its expert formulation concerns channel supports rather than conventional token dispatch.

Relevance: 8 Novelty: 7

5. Learning with Local Search MCMC Layers

ArXiv ID: 2505.14240

Primary Topic: Architecture and Training Dynamics

Authors: Germain Vivier-Ardisson, Mathieu Blondel, Axel Parmentier

Abstract: Integrating combinatorial optimization layers into neural networks has recently attracted significant research interest. However, many existing approaches lack theoretical guarantees or fail to perform adequately when relying on inexact solvers. This is a critical limitation, as many operations research problems are NP-hard, often necessitating the use of neighborhood-based local search heuristics. In this paper, we introduce a principled approach for learning with such inexact solvers. Inspired by the connection between simulated annealing and Metropolis-Hastings, we transform the problem specific neighborhood systems used in local search heuristics into proposal distributions, implementing MCMC on the set of feasible solutions. This allows us to construct differentiable, stochastic combinatorial layers and associated loss functions. Replacing an exact solver with a local search strongly reduces the computational burden of learning on many applications. We demonstrate our approach on a dynamic vehicle routing problem with time windows, binary vector and k-subset prediction tasks, as well as a multi-dimensional knapsack decision-focused learning problem.

Comment: Constructs differentiable stochastic optimization layers by turning local-search neighborhoods into MCMC proposals.

Topic Match: A new trainable computational layer is the core contribution, with a narrower focus on combinatorial optimization and inexact solvers.

Relevance: 7 Novelty: 7

6. The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions

ArXiv ID: 2602.21910

Primary Topic: Architecture and Training Dynamics

Authors: Alexander Heinlein, Johannes Taraz

Abstract: Operator learning has the potential to strongly impact scientific computing by learning solution operators for differential equations, potentially accelerating multi-query tasks such as design optimization and uncertainty quantification by orders of magnitude. Despite proven universal approximation properties, deep operator networks (DeepONets) often exhibit limited accuracy and generalization in practice, which hinders their adoption. Understanding these limitations is therefore crucial for further advancing the approach. This work analyzes performance limitations of the classical DeepONet architecture. It is shown that the approximation error is dominated by the branch network when the internal dimension is sufficiently large, and that the learned trunk basis can often be replaced by classical basis functions without a significant impact on performance. To investigate this further, a modified DeepONet is constructed in which the trunk network is replaced by the left singular vectors of the training solution matrix. This modification yields several key insights. First, for examples involving the KdV and Burgers equations, a spectral bias in the branch network is observed, with coefficients of dominant, low-frequency modes learned more effectively. Second, through the interplay of the singular-value weighting and the optimizer's neglect of small modes, the branch error is dominated by modes with large and intermediate singular values. Third, using a shared branch network for all mode coefficients, as in the standard architecture, improves generalization of small modes compared to a stacked architecture in which coefficients are computed separately. Finally, detrimental coupling between modes in parameter space is identified, which weakens with increasing network width.

Comment: Branch-trunk error decompositions expose how optimizer weighting and shared-parameter coupling affect the learning of different modes.

Topic Match: The contribution explains architectural and optimization mechanisms, although its evidence is confined to DeepONets rather than large-model pretraining.

Relevance: 7 Novelty: 7

7. Population-Scalable Multi-Agent World Modeling

ArXiv ID: 2608.08600

Primary Topic: Architecture and Training Dynamics

Also Matches: Efficiency, Compression, and Large-Scale Training

Authors: Renjie Zhao, Yuxiang Wu, Mingyu Zhang, Jiaxin Li, Sisi Li, He Li, Yimin Sheng, Tianxi Tan, Zhenkai Zhang, Jiao Liang, Jianyi Zhu, Yong-Lu Li

Abstract: World models have recently achieved impressive progress in visual prediction and interactive generation, but extending them to multi-agent environments introduces a fundamental scalability challenge. Existing methods generally assume a fixed number of agents during training and inference, which ties the model to a pre-determined agent population and limits inference-time scalability. Our key insight is that cross-view consistency should arise from a shared world state whose evolution does not assume a predefined number of agents, while agent-specific observations should be generated by querying this state through a unified rendering interface. Based on this insight, we propose Khora, a scalable multi-agent world model that supports inference-time expansion to arbitrary numbers of agents without retraining. Our framework decouples world-state evolution from visual rendering and introduces a population-agnostic rendering mechanism for incorporating other agent information. This design maintains cross-view consistency through the shared world state rather than through dense interactions among observation streams inside the expensive video generator, enabling approximately linear practical scaling with the number of queried views. Qualitative experiments demonstrate that our approach generalizes to unseen numbers of agents while maintaining visual quality and multi-agent consistency. We further implement a real-time interactive system to demonstrate scalable open-world simulation.

Comment: Separating shared-state evolution from per-agent rendering enables approximately linear scaling with queried views.

Topic Match: The shared-state/rendering separation is a substantive modular architecture with explicit computational-scaling benefits, although its demonstrated scope is multi-agent world simulation.

Relevance: 7 Novelty: 7

8. Wait, Wait, Wait... Why Do Reasoning Models Loop?

ArXiv ID: 2512.12895

Primary Topic: Architecture and Training Dynamics

Authors: Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, Dimitris Papailiopoulos

Abstract: Reasoning models (e.g., DeepSeek-R1) generate long chains of thought to solve harder problems, but they often loop, repeating the same text at low temperatures or with greedy decoding. We study why this happens and what role temperature plays. With open reasoning models, we find that looping is common at low temperature. Larger models tend to loop less, and distilled students loop significantly even when their teachers rarely do. This points to mismatches between the training distribution and the learned model, which we refer to as errors in learning, as a key cause. To understand how such errors cause loops, we introduce a synthetic graph reasoning task and demonstrate two mechanisms. First, risk aversion caused by hardness of learning: when the correct progress-making action is hard to learn but an easy cyclic action is available, the model puts relatively more probability on the cyclic action and gets stuck. Second, even when there is no hardness, Transformers show an inductive bias toward temporally correlated errors, so the same few actions keep being chosen and loops appear. Higher temperature reduces looping by promoting exploration, but it does not fix the errors in learning, so generations remain much longer than necessary at high temperature; in this sense, temperature is a stopgap rather than a holistic solution. We end with a discussion of training-time interventions aimed at directly reducing errors in learning.

Comment: Learning-error analysis links looping to hard-to-learn progress actions and temporally correlated Transformer errors.

Topic Match: Learning hardness and Transformer inductive biases provide a substantive connection to training dynamics, though the main phenomenon is inference-time looping.

Relevance: 7 Novelty: 7

9. Dimensionless Controls of Plasticity Under Alternating Tasks: From Evolutionary Biology to Continual Learning

ArXiv ID: 2608.23889

Primary Topic: Architecture and Training Dynamics

Authors: Owen Skriloff

Abstract: Plasticity under changing environments is central to both evolutionary biology and continual learning. Motivated by recent work on genotype--phenotype maps, we study a minimal deep-learning analogue where a network is trained alternately on two Boolean label sets, and ask which biological controls of plasticity survive the translation to gradient descent. Reinterpreting four proposed biological factors as quantities of training dynamics, we find the system reduces to two dimensionless controls: the task disagreement $r$, the fraction of disagreeing labels, and the reach $ηT$, the product of learning rate and switching period. We derive two bounds on plasticity: $r$ alone fixes an extremal geometric floor on the utopia distance, while $r$ and $ηT$ jointly bound forgetting. Across 9,720 trajectories, an ANOVA confirms that $r$, $η$, and $T$ dominate, while the effect of neutral-set size (emphasized in the biological setting) is negligible. The optimal reach itself follows an approximate inverse power law $ηT^{}\propto r^{-1.18}$, yielding a heuristic that sets the optimal reach $ηT^$ from the task disagreement alone. The analogy that survives is therefore dynamical rather than geometric, and our setting enables a view of plasticity through the lens of other driven systems in physics and engineering.

Comment: Task disagreement and learning-rate times switching-period yield bounds on forgetting and a heuristic for optimal plasticity.

Topic Match: The core is a mechanistic account of training dynamics and forgetting, although evidence is restricted to a toy continual-learning setting.

Relevance: 7 Novelty: 6

10. Group-Equivariant Poincaré Convolutional Networks

ArXiv ID: 2607.00556

Primary Topic: Architecture and Training Dynamics

Authors: Aiden Durrant, Rahul Baburajan, Georgios Leontidis

Abstract: While recent methods like that of the Poincaré ResNet have demonstrated the ability to learning visual representations directly in hyperbolic space, their optimisation remains a challenge, primarily due to the parameter redundancy of learning distinct orientation filters. In addition, hyperbolic learning exhibits distinct computational overheads that limit their wide use, where efforts to improve their efficiency via optimisation have seen good success, there has been limited exploration into structural priors that enable stronger sample efficiency at training. To address this, we propose Equivariant Poincaré ResNets, combining hyperbolic geometry with discrete symmetry groups ($C_4$ and $D_4$). We identify critical roadblocks in applying Euclidean equivariance to hyperbolic space and propose geometrically safe tensor reshaping, left-regular permutations for hyperbolic group convolutions, and joint-orientation Poincaré Midpoint Batch normalisation. Empirical evaluations show that embedding equivariance significantly improves the sample efficiency during training which in-turn accelerates convergence while respecting the boundary constraints of the Poincaré ball and retaining spatial group equivariance.

Comment: Group-equivariant hyperbolic convolutions and joint-orientation normalization reduce parameter redundancy and improve training convergence.

Topic Match: The contribution introduces convolution and normalization mechanisms, with narrower applicability to hyperbolic vision architectures.

Relevance: 7 Novelty: 6

11. DuoGNN: Topology-aware Graph Neural Network with Homophily and Heterophily Interaction-Decoupling

ArXiv ID: 2409.19616

Primary Topic: Architecture and Training Dynamics

Authors: K. Mancini, I. Rekik

Abstract: Graph Neural Networks (GNNs) have proven effective in various medical imaging applications, such as automated disease diagnosis. However, due to the local neighborhood aggregation paradigm in message passing which characterizes these models, they inherently suffer from two fundamental limitations: first, indistinguishable node embeddings due to heterophilic node aggregation (known as over-smoothing), and second, impaired message passing due to aggregation through graph bottlenecks (known as over-squashing). These challenges hinder the model expressiveness and prevent us from using deeper models to capture long-range node dependencies within the graph. Popular solutions in the literature are either too expensive to process large graphs due to high time complexity or do not generalize across all graph topologies. To address these limitations, we propose DuoGNN, a scalable and generalizable architecture which leverages topology to decouple homophilic and heterophilic edges and capture both short-range and long-range interactions. Our three core contributions introduce (i) a topological edge-filtering algorithm which extracts homophilic interactions and enables the model to generalize well for any graph topology, (ii) a heterophilic graph condensation technique which extracts heterophilic interactions and ensures scalability, and (iii) a dual homophilic and heterophilic aggregation pipeline which prevents over-smoothing and over-squashing during the message passing. We benchmark our model on medical and non-medical node classification datasets and compare it with its variants, showing consistent improvements across all tasks. Our DuoGNN code is available at https://github.com/basiralab/DuoGNN.

Comment: Decoupled homophilic and heterophilic aggregation targets over-smoothing and over-squashing in deep graph networks.

Topic Match: Graph aggregation and long-range information flow are the core architectural contributions, though their connection to large-model training is narrower.

Relevance: 7 Novelty: 6

12. Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learning

ArXiv ID: 2412.00776

Primary Topic: Architecture and Training Dynamics

Authors: Chongyang Zhao, Dong Gong

Abstract: Continual learning (CL) learns from a non-stationary data stream without storing or re-training on all seen samples. Meta-continual learning (MCL) casts CL as sequence prediction and meta-learns the continual learner itself as a sequence model, with Transformers as natural choices. However, despite decent performance, a Transformer learner relies on a linearly growing key-value cache to store all past representations, conflicting with CL's objective of not storing all seen samples. Sequence models with a constant-size state, e.g., linear-attention models and state-space models (SSMs), match CL's requirement by definition; however, with past samples compressed and mixed in the bounded state, such learners are harder to meta-learn, and earlier kernel-based ones performed poorly on MCL. We thus aim to obtain effective continual learners with constant-size states and make them work for MCL. By formulating the selective SSM for MCL, we propose MambaCL, with Mamba as the main model in practice. We meta-train the learner with a proposed selectivity regularization, which supervises the implicit associations within the compressed state and applies across constant-state models. Furthermore, we conduct a systematic empirical study of how Mamba and other constant-state models behave across various MCL scenarios, covering effectiveness, generalization, and model designs. The results highlight promising performance and strong generalization, demonstrating the potential of constant-state learners for efficient continual learning and adaptation.

Comment: Introduces selectivity regularization to teach bounded-state sequence models to preserve associations between past samples.

Topic Match: Training selective state-space models is a meaningful architectural connection, but the contribution centers on meta-continual learning rather than foundational large-model training.

Relevance: 6 Novelty: 6

Efficiency, Compression, and Large-Scale Training (15)

1. UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference

ArXiv ID: 2603.18446

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Lang Zhou, Shuxuan Li, Zhuohao Li, Shi Liu, Wei-Shi Zheng, Zhilin Zhao

Abstract: Long-context inference remains challenging for large language models due to attention dilution and out-of-distribution degradation. Context selection mitigates this limitation by attending to a subset of key-value cache entries, yet most methods allocate a fixed context budget throughout decoding despite highly non-uniform token-level contextual demands. To address this issue, we propose Uncertainty-Triggered Adaptive Context Allocation (UT-ACA), an inference-time framework that dynamically adjusts the context window based on token-wise uncertainty. UT-ACA learns an uncertainty detector that combines semantic embeddings with logit-based confidence while accounting for uncertainty accumulation across decoding steps. When insufficient evidence is indicated, UT-ACA selectively rolls back, expands the context window, and regenerates the token with additional support. Experiments show that UT-ACA substantially reduces average context usage while preserving generation quality in long-context settings. Code and dataset are available at https://github.com/Tommy307/UT-ACA.

Comment: Dynamically allocates attended KV context using token uncertainty, selective rollback, and regeneration.

Topic Match: The central mechanism adjusts how much cached context each token uses, directly addressing context and attention efficiency during large-model inference.

Relevance: 9 Novelty: 7

2. Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

ArXiv ID: 2608.23296

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Isaac, Li

Abstract: Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled $2\times2\times2$ comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H$_2$O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.

Comment: Shows how sigmoid attention improves transfer from differentiable training gates to physical KV-cache deletion.

Topic Match: Learned cache eviction directly targets memory cost, while controlled attention comparisons explain the architectural mechanism behind successful hard deletion.

Relevance: 9 Novelty: 7

3. Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

ArXiv ID: 2608.23841

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Tom Poperszky

Abstract: Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of llama.cpp (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold.

Comment: Co-designed inter-layer dependencies enable stage-major execution that reduces weight-bandwidth costs during autoregressive decoding.

Topic Match: The central contribution reduces large-model decoding costs through architectural dependency changes and a matching execution schedule.

Relevance: 9 Novelty: 7

4. Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs

ArXiv ID: 2608.22854

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Yan Zhou, Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak, Marco Fumero, Francesco Locatello, David Alvarez-Melis

Abstract: Practical deployment of large language models (LLMs) requires families of post-trained variants---instruction-tuned, reasoning-tuned, and chat-style models---each at multiple sizes to meet diverse latency and memory budgets. Producing each (variant, size) pair independently is prohibitive, so model families typically span only a handful of coarse-grained sizes per post-trained variant. Boomerang distillation (Kangaslahti et al., 2026) reduces this cost along the size axis for base models. Through model size interpolation, it constructs models of intermediate sizes from a single teacher-student pair without additional training. However, it still treats each post-trained variant as a separate object of optimization. We introduce ADAPT---Amortized Distillation Across Post-Trained LLMs---a framework for amortizing distillation across both axes of a model family: size and post-training variant, producing $L \times K$ models for $L$ interpolated sizes across $K$ post-trained variants with a single distillation run. ADAPT combines two components. First, a two-phase distillation procedure constructs post-trained students through pre-training alignment and supervised fine-tuning distillation, enabling smooth size--performance interpolation on generation and reasoning tasks. Second, weight-delta initialization approximates this construction across post-trained variants by transferring the distillation-induced weight change from the base model to students initialized from different post-trained variants. The resulting continuum of interpolated models also enables adaptive model-size selection at inference time, improving the compute--accuracy trade-off for long-form reasoning tasks.

Comment: Amortizes model-family compression across sizes and post-training variants using a single distillation run.

Topic Match: The core contribution reduces compressed-model construction costs through size interpolation and transferable distillation weight deltas.

Relevance: 9 Novelty: 7

5. Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair

ArXiv ID: 2608.23144

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zehao Liu, Chuangchuang Fang, Yang Ren

Abstract: Low-bit weight quantization saves storage but leaves errors that degrade language-model quality. We introduce Activation-Weighted Seeded Residual Coding (AWSRC), a compact repair codec for an existing quantization backbone. Given a reconstructed weight $W_0$, AWSRC encodes the residual $W-W_0$ using deterministic seed-generated bases. The sidecar stores seed selectors, low-bit coefficients, and scales rather than an explicit codebook. Activation statistics prioritize errors that affect layer outputs. On Qwen2.5-3B-Instruct, adding 0.162 scope-bits/weight to an INT4 RTN backbone closes 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16. Repairing a matched strong low-bit backbone also improves all measured quality metrics. With a matched 49.25 MB sidecar, about 0.8% of the BF16 model-weight payload, AWSRC gives the best perplexity and mean task accuracy among sparse, low-rank, and vector-quantized codecs.

Comment: Activation-weighted, seed-generated residual codes repair low-bit LLM weights using compact side information.

Topic Match: The residual-weight codec directly improves the storage-quality tradeoff of quantized LLMs through a distinct compression mechanism.

Relevance: 9 Novelty: 7

6. PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

ArXiv ID: 2608.23843

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li

Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among existing approaches, low-rank compression is particularly attractive because it represents every token in reduced dimensions. Previous low-rank methods typically derive fixed projection spaces from model weights, construct fixed spaces from calibration activations, or construct a shared basis over a broad cache region. Such representations may not capture detailed but important information. We partition each per-head KV cache into fixed-length logical pages and observe substantial low-rank structure within individual pages. Based on this observation, we propose PuzzleKV, a training- and calibration-free method that treats each completed page as an independent compression unit. PuzzleKV decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages during autoregressive decoding. Experiments across models, context lengths, and benchmarks demonstrate the effectiveness of PuzzleKV under matched storage budgets. At approximately 60% of the original KV cache storage, PuzzleKV achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on LongBench. To achieve a more aggressive compression ratio, PuzzleKV can be further combined with quantization while retaining more than 93% of Full KV performance using only 18.7% of the original storage.

Comment: Page-wise low-rank KV cache decomposition exploiting within-page low-rank structure, training- and calibration-free, composable with quantization.

Topic Match: Core contribution is a new KV-cache memory-compression mechanism, squarely in the efficiency and compression topic.

Relevance: 9 Novelty: 7

7. ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation

ArXiv ID: 2608.23429

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Chengjie Lu, Tianchi Deng, Zhengqi He, Chengwen Luo, Xueliang Li

Abstract: Diffusion Transformers (DiTs) have shown strong performance in high-fidelity image generation, but their sampling process remains computationally intensive due to full model execution at every timestep. While cache-based acceleration has been explored to mitigate inference cost, naive reuse schemes suffer from low accuracy over long intervals, and Taylor-series-based extrapolation methods often face instability caused by Runge oscillations. In this paper, we propose ChebBooster, a training-free extrapolation framework based on Chebyshev polynomial theory that achieves stable and efficient acceleration for DiTs. Specifically, we adopt the Barycentric formulation to evaluate Chebyshev approximants with high numerical stability and minimal overhead, and further decouple the extrapolation into an offline weight precomputation phase and a lightweight online application stage. Extensive experiments across three representative DiT-based models, including DiT-XL/2, PixArt-$Σ$, and FLUX.1-dev, demonstrate that ChebBooster achieves consistent improvements in visual quality and inference efficiency, reaching up to $3.68\times$ latency speedup and $5.12\times$ FLOPs reduction, outperforming existing training-free baselines under diverse generation tasks and resolutions.

Comment: Uses stable Chebyshev extrapolation to skip repeated Diffusion Transformer computation during sampling.

Topic Match: The core mechanism replaces repeated model execution with inexpensive extrapolation, directly reducing large-model inference FLOPs and latency.

Relevance: 9 Novelty: 6

8. Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention

ArXiv ID: 2608.23834

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Sergii Kozyrev, Davyd Maiboroda

Abstract: The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.

Comment: Direct attention over mixed FP8/TQ3 pages compresses live KV without eviction or a dense shadow cache.

Topic Match: Quantized KV storage and normalized merging across format-specific kernels directly address large-model memory costs; demonstrated benefits primarily concern capacity.

Relevance: 9 Novelty: 6

9. Provably adaptive sampling with uniform and remasking discrete diffusion models

ArXiv ID: 2608.23554

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Daniil Dmitriev, Zhihan Huang, Yuting Wei

Abstract: Discrete diffusion models offer a promising alternative to autoregressive generation by enabling parallel updates, but their sampling efficiency can depend strongly on the choice of the forward process and the sampler. For the uniform forward process, existing lower bounds for the standard $τ$-leaping sampler scale linearly with the ambient dimension $d$, raising the question of whether this dependence is intrinsic to the forward process. We answer this question in the negative. We consider a first-order sampler based on the leave-one-out denoiser for uniform and remasking processes whose coordinate updates can be performed in parallel. In both cases, the sampler can correct denoising mistakes during the sampling process, which becomes necessary when many coordinates are updated together. Our main result establishes an adaptive sampling guarantee: up to logarithmic factors, $N = O(\mathrm{DTC}(X_0) / \varepsilon)$ discretization steps suffice to achieve sampling error $O(\varepsilon_{\mathrm{score}}+\varepsilon)$, where $\varepsilon_{\mathrm{score}}$ is the error in score estimation. Thus, the sampling complexity is governed by the intrinsic dependence structure of the target distribution, as measured by its dual total correlation $\mathrm{DTC}(X_0)$, rather than directly by the ambient dimension $d$. Our analysis proceeds through a Bayes-optimal auxiliary sampler that separates discretization error from score-estimation error. We also derive an exact information-theoretic representation of the discretization error in terms of the mutual information between different coordinates of the forward process at different times. This representation applies to general forward processes and, in the uniform and remasking cases, can be controlled by $\mathrm{DTC}(X_0)$. Numerical experiments on structured synthetic distributions illustrate the predicted dimension-adaptive behavior.

Comment: Bounds parallel discrete-diffusion sampling steps using dual total correlation instead of ambient dimension.

Topic Match: Provides a substantive theoretical mechanism for reducing sequential sampling work, although experiments establish synthetic behavior rather than large-model speedups.

Relevance: 7 Novelty: 8

10. Rethinking Reverse KL as Adaptive Entropy Distillation

ArXiv ID: 2608.14685

Primary Topic: Efficiency, Compression, and Large-Scale Training

Also Matches: Architecture and Training Dynamics

Authors: Shizhen Li, Zhiyu Shen, Yuyin Lu, Yunhe Pang, Jielin Song, Yanghui Rao, Fu Lee Wang

Abstract: Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In particular, existing methods mainly combine FKL and RKL, overlooking that RKL itself provides a mechanism for adjusting the student's imitation strength. Motivated by this, we revisit on-policy Reverse Kullback-Leibler (RKL) distillation and decompose its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch. We show theoretically that the token-level optimal student distribution corresponds to a tempered variant of the teacher distribution, where the adaptive weight controls the trade-off between mode-seeking and uncertainty preservation. Guided by this insight, we propose \textbf{Adaptive Entropy Distillation (AED)}, which uses the teacher's entropy to dynamically calibrate token-level imitation strength. Experiments on instruction-following and mathematical reasoning benchmarks demonstrate that AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.

Comment: Teacher-entropy-dependent weighting modifies reverse-KL distillation to train smaller student models.

Topic Match: The primary contribution is a general teacher-student compression objective; its tempered-optimum analysis also explains how entropy weighting changes distillation dynamics.

Relevance: 8 Novelty: 6

11. Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap

ArXiv ID: 2608.23658

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Sathishkumar Sivashanmugam

Abstract: An LLM serving engine sizes its key-value (KV) cache once, at startup, permanently setting aside a reserve for the worst-case prefill activation. During decode-dominant phases that reserve sits idle, yet it cannot be handed to the KV pool because it is exactly the memory a large prefill needs. We ask whether this reserve is reclaimable, and build a mechanism to test it. Our elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill, driven by the scheduler's one-step-ahead view of the next batch. It is pure userspace on the CUDA virtual-memory path: two physical handles mapped into one contiguous virtual range per layer, so the attention kernel is unchanged and no driver patch is required. It decommits in a few milliseconds and recommits in tens of milliseconds, works with CUDA graphs and prefix caching, and never triggers an out-of-memory event. A static commit of the same memory is unsafe, crashing on prefill bursts, which makes the dynamic toggle necessary. Having built the mechanism, we test the premise it rests on and report an honest negative result. It only pays off if a small prefill chunk size badly hurts prefill latency. In a controlled experiment injecting long prompts into a live decode load, that penalty is small (median time-to-first-token differs by about 1% between chunk sizes of 8192 and 32768 tokens), because prefill is compute bound and decode consumes only about one token per sequence per step. Simply lowering max_num_batched_tokens recovers more KV than the controller does, at nearly equal latency. The reserve also dilutes under tensor parallelism, from 16% of KV at TP1 to 2.7% at TP4. We state precisely when reclaiming the reserve could still help, and release the mechanism as a reusable userspace elastic-VMM allocator.

Comment: Scheduler-guided virtual-memory management lends reserved prefill activation memory to the KV cache.

Topic Match: The elastic allocator directly addresses KV-cache memory efficiency, and its negative result establishes when chunked prefill already captures the available benefit.

Relevance: 8 Novelty: 6

12. Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

ArXiv ID: 2608.23048

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Ha Dinh, Xuan Duy Ta, Khoat Than, Khac-Hoai Nam Bui

Abstract: Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible $N$:$M$ patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to $\mathcal{O}({M})$. As a result, RoI requires 1.5-8.75$\times$ fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns, offering a practical path toward efficient LLM deployment.

Comment: Learns N:M semi-structured masks via differentiable subset sampling with a compact-logit parameterization, cutting mask parameters from combinatorial to O(M).

Topic Match: A new sparsity-mask learning mechanism for hardware-friendly pruning, directly in the efficiency and compression topic.

Relevance: 8 Novelty: 6

13. E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models

ArXiv ID: 2608.23253

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Taoyu Qian, Qi Wang, Daqian Shi, Yuanhao Jiang, Shang Gao, Hualong Yu

Abstract: Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Existing pruning methods largely rely on attention scores and directly aggregate outputs across attention heads and network layers, making it difficult to characterize evidential uncertainty and conflict. We propose E2S-Pruner, a progressive two-stage evidence-fusion framework for visual token pruning that requires no auxiliary model, trainable parameters, or fine-tuning. In the first stage, E2S-Pruner treats each attention head as an independent evidence source, estimates its reliability from evidence clarity and inter-head consistency, and represents each visual token using three states: important, unimportant, and uncertain. In the second stage, Dempster--Shafer evidence theory is used to quantify inter-layer conflict and fuse complementary evidence from multiple network layers. We further introduce a spatial novelty constraint that promotes coverage of distinct image regions and prevents the retained tokens from concentrating in a few locally salient areas. On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of the aggregate performance when the average numbers of retained visual tokens are 192, 128, and 64, respectively, while improving throughput by 1.96x and 2.09x under the 128-token and 64-token settings. Experiments on Qwen2-VL-7B further demonstrate cross-model generalization. Code is available at https://github.com/taoyu-qian/E2S-Pruner.git.

Comment: Reliability-weighted evidence fusion across attention heads and layers selects visual tokens to reduce VLM inference cost.

Topic Match: Token pruning is the core contribution, with a distinct selection mechanism and measured throughput improvements.

Relevance: 8 Novelty: 6

14. Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning

ArXiv ID: 2608.22858

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Dongyue Wu, Tao Ma

Abstract: Existing data pruning methods predominantly rely on high-dimensional feature embeddings to measure sample importance. However, these compressed vectors often obscure fine-grained semantic interactions, leading to suboptimal coverage of rare semantic concepts in the pruned subsets. In this paper, we propose Mapping the Concept Landscape (MCL), a novel structural perception framework for transparent data pruning. Instead of abstract embeddings, we represent each image-caption pair as an explicit sample-level graph comprising entities, events, and attributes. By integrating these individual graphs into a comprehensive dataset-level graph, we characterize the global distribution of semantic concepts and quantify their rarity across the entire corpus. Based on this structured perception, we develop a greedy concept-coverage maximization algorithm that iteratively selects samples to maximize the marginal gain of high-value, under-represented concepts. Experimental results on various benchmarks demonstrate that our method not only achieves superior pruning efficiency compared to state-of-the-art methods but also provides a transparent and interpretable audit trail for the selection process.

Comment: Prunes image-caption training data by greedily maximizing rare-concept coverage in a corpus-wide semantic graph.

Topic Match: The core contribution is a training-data selection mechanism that can reduce training workload; large-model compute savings are not quantified in the abstract.

Relevance: 7 Novelty: 6

15. When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving

ArXiv ID: 2605.04450

Primary Topic: Efficiency, Compression, and Large-Scale Training

Authors: Wenjun Yu, Shuguang Han, Amelie Chi Zhou

Abstract: Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficiency but degrades the other. Existing systems optimize them in isolation, overlooking that the optimal EMB-KV allocation ratio can shift by up to 0.35 across workload regimes, leaving 20-30\% latency improvement unrealized. While online reallocation is required to close this gap, naive approaches introduce H2D refill traffic on the critical path, causing P99 SLO violations. To address this, we present RACER, which jointly manages HBM allocation and request routing at runtime through two key components: (1) Adaptive Memory Allocation, a three-layer PPO-based controller (frozen base policy, online residual adapter, and burst-aware recovery controller) that achieves $32\,\mathrm{μs}$ decision latency while staying within 0.024-0.029 of the offline-optimal ratio; and (2) EMB-KV-Aware Scheduling, which routes requests by jointly considering KV residency, embedding locality, and node load to avoid routing inefficiencies under heterogeneous allocations. Evaluations on three production-scale datasets over a 32-node A100 cluster show that RACER reduces P99 latency by 24-38\% over the best static policy and achieves 93.5-99.6\% SLO satisfaction across Steady, Trend, and Burst workloads, significantly outperforming state-of-the-art baselines without sacrificing throughput.

Comment: Joint GPU memory allocation and cache-aware scheduling reduce contention between embedding and KV caches.

Topic Match: The core contribution is adaptive cache allocation and scheduling, with applicability concentrated in generative recommender serving.

Relevance: 7 Novelty: 6

Paper Selection Prompt

System Prompt

You are a helpful paper reading assistant whose job is to read daily posts from ArXiv and identify a few papers that your friend will enjoy reading. Your job is to carefully read the paper titles and abstracts below and find the ones that match the criteria below.

User Prompt

Instructions

Respond in JSONL. Output exactly one JSON object per paper, one per line:

{"ARXIVID":"...","COMMENT":"...","RELEVANCE":0,"NOVELTY":0,"PRIMARY_TOPIC_ID":"...","MATCHED_TOPIC_IDS":[],"TOPIC_MATCH_COMMENT":"...","HOTSPOT_PAPER_TAGS":[],"HOTSPOT_PAPER_COMMENT":"..."}

Rules: - ARXIVID: the arXiv ID. - COMMENT: identify the single strongest matching criterion. Be brief and specific. Do not rely on generic phrases like "language modeling" or "advancement". Do not mention non-matching criteria. - RELEVANCE: integer from 1 to 10. - NOVELTY: integer from 1 to 10. - PRIMARY_TOPIC_ID: exactly one stable topic ID from the allowed topic registry. - MATCHED_TOPIC_IDS: zero or more stable topic IDs from the same allowed set. Include PRIMARY_TOPIC_ID when there are multiple matches. - TOPIC_MATCH_COMMENT: briefly explain why the primary topic is the best fit. - HOTSPOT_PAPER_TAGS: zero or more tags from this exact set only: daily_hot, new_frontier. - HOTSPOT_PAPER_COMMENT: briefly explain why the paper belongs in the daily hotspot paper feed when HOTSPOT_PAPER_TAGS is non-empty; otherwise use an empty string. - Use HOTSPOT_PAPER_TAGS sparingly. Most papers should return []. - daily_hot means the paper feels broadly important to the day and belongs in the daily hotspot paper section even if it is not part of the personalized foundational reading list. - new_frontier means the paper appears to open a genuinely new direction, paradigm, or field, even if the work is still early. - Do not output markdown, code fences, or any extra text.

Scoring Criteria

Relevance and Novelty are independent axes. Score both from 1 to 10.

Relevance Scoring

• 9-10: directly centered on the target foundational topics; highest when the core contribution is clearly within them.
• 7-8: substantially related, but partly peripheral or focused on a narrower aspect.
• 5-6: touches the target topics, but the main contribution is elsewhere.
• 3-4: largely outside the target topics, often application-focused or domain-specific.
• 1-2: unrelated.

Important: Broad frontier relevance, major launch status, or daily buzz is not enough for a high Relevance score here. Those cases belong in the hotspot digest unless the paper strongly matches the specialized paper topics.

Novelty Scoring

• 9-10: new paradigm, theory, or major methodological breakthrough.
• 7-8: substantial methodological advance or strong new insight.
• 5-6: meaningful but incremental extension or refinement.
• 3-4: minor, narrow, or mostly engineering or domain-specific improvement.
• 1-2: little originality; mainly standard application of existing methods.

Topic Registry

Use exactly one PRIMARY_TOPIC_ID chosen from the stable topic IDs below. - moe_training: MoE Training - Mixture-of-Experts training: routing, load balancing, expert granularity and shared experts, training stability, MoE parallelism and communication, and MoE kernels. - training_systems: Large-Scale Training Systems and Efficiency - Large-scale training systems and efficiency: distributed training algorithms, optimizers, and communication. - architecture_training: Architecture and Training Dynamics - Core architectural or computational mechanisms, dynamic computation, and training-stability dynamics. - efficiency_scaling: Efficiency, Compression, and Large-Scale Training - Compression, sparsity, memory or cache efficiency, and large-scale training systems that materially change cost or behavior.

Papers

[PAPER LIST HERE]

Relevant Topics

This is a training-side feed. The centre of gravity is Mixture-of-Experts training and the large-scale training systems it sits on; architecture and efficiency work is kept when it bears on how a large model is trained or what it costs to train.

Keep a paper when its CORE CONTRIBUTION falls in one of the four topics below. Filter it when the topic is merely the setting for a downstream application, a benchmark, or a product result.

1. MoE Training (primary) - Keep: expert routing and gating mechanisms; load balancing, with or without auxiliary losses; expert granularity, fine-grained and shared experts; MoE training stability (loss spikes, router collapse, z-loss and related fixes); expert and hybrid parallelism; all-to-all dispatch/combine and the communication schedules MoE needs; MoE kernels and grouped-GEMM style implementations; expert capacity, token dropping, and upcycling dense checkpoints into MoE. - Filter: papers that merely train on top of a MoE model without a new routing, balancing, stability, parallelism, or kernel contribution; "mixture of experts" in the classical ensemble or recommender sense.

2. Large-Scale Training Systems - Keep: distributed training algorithms (data, tensor, pipeline, sequence, and expert parallelism, FSDP and ZeRO-style sharding); optimizers and preconditioners for large-scale pretraining; communication algorithms, overlap, and collective design; throughput and MFU work that materially changes what a training run costs or whether it converges; scaling-law work that informs how such runs are configured. - Filter: routine infrastructure tuning with no new algorithmic or systems idea.

3. Architecture and Training Dynamics - Keep: new or analysed architectural mechanisms (attention variants, normalization and residual design, state-space and recurrent sequence modelling, dynamic or modular computation); optimisation and training-dynamics analysis that explains why large models train the way they do. - Filter: applying an existing architecture to a new task or benchmark with no mechanistic insight.

4. Efficiency and Compression - Keep: quantization, sparsity, pruning, low-rank adaptation, KV-cache and memory-efficient designs, and other work that materially changes the cost of running or training a large model, especially when the mechanism is new rather than a tuned variant. - Filter: straightforward application of standard efficiency methods, and deployment or serving engineering with no new idea.

Leave these to the hotspot digest unless the core contribution clearly falls in one of the four topics above: - major model or product releases - agent, tooling, and RAG launches - benchmark, leaderboard, or evaluation-only papers - representation-learning theory, interpretability, and feature-formation analysis - memory mechanisms and agent memory systems - world models, exploration, and reinforcement learning - alignment and post-training (RLHF, DPO, GRPO, RFT, instruction tuning) - downstream applications in medical imaging, segmentation, 3D vision, video understanding, information retrieval, summarization, recommendation, machine translation, speech recognition, time series, knowledge graphs, and similar domains