# research_papers

source_id: 2605.14301
source: arxiv
title: Language-Induced Priors for Domain Adaptation
authors: ["Qiyuan Chen","Jiayu Zhou","Raed Al Kontar"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14301v1
tags: ["theories and proofs","evaluation","model infrastructure"]
org: University of Michigan
fetched_at: 2026-05-16T03:07:13.722Z
classified_at: 2026-05-16T03:07:13.693Z
abstract:
Domain adaptation faces a fundamental paradox in the cold-start regime. When target data is scarce, statistical methods fail to distinguish relevant source domains from irrelevant ones, which often leads to negative transfer. In this paper, we address this challenge by leveraging expert textual descriptions of the target domain, a resource that is often available but overlooked. We propose a probabilistic framework that translates these semantic descriptions into a choice model, namely a Language-Induced Prior (LIP), that learns the preferences from a pretrained Large Language Model (LLM). The LIP is then integrated into an Expectation-Maximization algorithm to identify source relevance. Methodologically, this framework is compatible with any parametric model where a likelihood is available. It allows the LIP to guide the selection of sources when target signals are weak, while gradually refining these choices as samples accumulate. Theoretically, we prove that the estimator roughly matches an oracle cold-start MSE under a correct prior, while remaining asymptotically consistent regardless of the quality of the LIP. Empirically, we validated the framework on a descriptive (Gaussian estimation), a predictive (C-MAPSS dataset), and a prescriptive task (MuJoCo hopper).


source_id: 2605.14280
source: arxiv
title: TILT: Target-induced loss tilting under covariate shift
authors: ["Kakei Yamamoto","Martin J. Wainwright"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14280v1
tags: ["theories and proofs","training","evaluation"]
org: Massachusetts Institute of Technology
fetched_at: 2026-05-16T03:08:41.111Z
classified_at: 2026-05-16T03:08:41.079Z
abstract:
We introduce and analyze Target-Induced Loss Tilting (TILT) for unsupervised domain adaptation under covariate shift. It is based on a novel objective function that decomposes the source predictor as $f+b$, fits $f+b$ on labeled source data while simultaneously penalizing the auxiliary component $b$ on unlabeled target inputs. The resulting fit $f$ is deployed as the final target predictor. At the population level, we show that this target-side penalty implicitly induces relative importance weighting at the population level, but in terms of an estimand $b^*_f$ that is self-localized to the current error, and remains uniformly bounded for any source-target pair (even those with disjoint supports). We prove a general finite-sample oracle inequality on the excess risk, and use it to give an end-to-end guarantee for training with sparse ReLU networks. Experiments on controlled regression problems and shifted CIFAR-100 distillation show that TILT improves target-domain performance over source-only training, exact importance weighting, and relative density-ratio baselines, with a stable dependence on the regularization parameter.


source_id: 2605.14260
source: arxiv
title: On the Burden of Achieving Fairness in Conformal Prediction
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14260v1
tags: ["theories and proofs","alignment","evaluation"]
org: McGill University
fetched_at: 2026-05-16T03:10:12.011Z
classified_at: 2026-05-16T03:10:11.976Z
authors:
["Ziang Gao","Pengqi Liu","Archer Yi Yang","Mouloud Belbahri","Jesse C. Cresswell","Masoud Asgharian"]

abstract:
Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We study this phenomenon through the population score distributions underlying split conformal calibration. First, we derive a conservation law and lower bound showing that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity. Second, we demonstrate that the two leading fairness definitions for conformal prediction, Equalized Coverage and Equalized Set Size, are fundamentally in tension. Third, we quantify the cost of moving between policies which treat groups separately or pool them. Experiments on synthetic and real data confirm the same bidirectional trade-off after finite-sample calibration. Our results show that, for the policy families studied here, calibration choice does not remove cross-group heterogeneity; it determines whether the resulting distortion appears in the coverage or size dimension, providing a principled lens for analyzing fairness-oriented calibration choices in practice.


source_id: 2605.14276
source: arxiv
title: Training-Free Generative Sampling via Moment-Matched Score Smoothing
authors: ["Zhenyu Yao","Daniel Paulin"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14276v1
tags: ["theories and proofs","model infrastructure","evaluation"]
org: Nanyang Technological University
fetched_at: 2026-05-16T03:11:45.075Z
classified_at: 2026-05-16T03:11:45.045Z
abstract:
Diffusion models generate samples by denoising along the score of a perturbed target distribution. In practice, one trains a neural diffusion model, which is computationally expensive. Recent work suggests that score matching implicitly smooths the empirical score, and that this smoothing bias promotes generalization by capturing low-dimensional data geometry. We propose moment-matched score-smoothed overdamped Langevin dynamics (MM-SOLD), a training-free interacting particle sampler that enforces the target moments throughout the sampling trajectory. We prove that, in the large-particle limit, the empirical particle density converges to a deterministic limit whose one-particle stationary marginal is a Gibbs--Boltzmann density obtained by exponentially tilting a naive score-smoothed diffusion target. The mean and covariance of this distribution agree with the empirical moments of the training data. Experiments on 2D distributions and latent-space image generation show that MM-SOLD enables fast, robust, training-free sampling on CPUs, with sample fidelity and diversity competitive with neural diffusion baselines.


source_id: 2605.15184
source: arxiv
title: Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
authors: ["Sahil Sen","Akhil Kasturi","Elias Lumer","Anmol Gulati","Vamse Kumar Subbiah"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.15184v1
tags: ["retrieval augmented generation","evaluation","model infrastructure"]
org: PricewaterhouseCoopers
fetched_at: 2026-05-16T03:13:23.246Z
classified_at: 2026-05-16T03:13:23.212Z
abstract:
Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI), for both inline tool results and file-based tool results that the model reads separately. Experiment 2 compares grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter. Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same.


source_id: 2605.15104
source: arxiv
title: From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.15104v1
tags: ["evaluation","model infrastructure","theories and proofs"]
org: Dialpad Inc.
fetched_at: 2026-05-16T03:14:54.489Z
classified_at: 2026-05-16T03:14:54.456Z
authors:
["Md Tahmid Rahman Laskar","Xue-Yong Fu","Seyyed Saeed Sarfjoo","Quinten McNamara","Jonas Robertson","Shashi Bhushan TN"]

abstract:
Voice agents increasingly require reliable tool use from speech, whereas prominent tool-calling benchmarks remain text-based. We study whether verified text benchmarks can be converted into controlled audio-based tool calling evaluations without re-annotating the tool schema and gold labels. Our dataset-agnostic framework uses text-to-speech, speaker variation, and environmental noise to create paired text-audio instances while preserving the original dataset annotations. Based on extensive evaluation of 7 omni-modal models on audio-converted versions of Confetti and When2Call, our framework demonstrates that the performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9). On Confetti, the text-to-voice gap ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. A targeted analysis of failure cases demonstrates that degradations most often reflect misunderstandings of argument values in the speech. Considering real-world deployment scenarios, we further report text-only results, an ambiguity-based reformulation stress test, and a reference-free LLM-as-judge protocol validated against human preferences. Notably, we find that open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges, supporting privacy-preserving evaluation. Overall, our framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora.


source_id: 2605.15128
source: arxiv
title: MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.15128v1
tags: ["evaluation","model infrastructure","theories and proofs"]
org: Rutgers University
fetched_at: 2026-05-16T03:16:30.077Z
classified_at: 2026-05-16T03:16:30.046Z
authors:
["Minghao Guo","Qingyue Jiao","Zeru Shi","Yihao Quan","Boxuan Zhang","Danrui Li","Liwei Che","Wujiang Xu","Shilong Liu","Zirui Liu","Mubbasir Kapadia","Vladimir Pavlovic","Jiang Liu","Mengdi Wang","Yiyu Shi","Dimitris N. Metaxas","Ruixiang Tang"]

abstract:
Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning. In prior work, many visually grounded questions can be answered using only captions or textual traces, allowing answers to be inferred without preserving the fine-grained visual evidence. Meanwhile, harder cases that require reasoning over changing visual states are largely absent. Therefore, we introduce MemEye, a framework that evaluates memory capabilities from two dimensions: one measures the granularity of decisive visual evidence (from scene-level to pixel-level evidence), and the other measures how retrieved evidence must be used (from single evidence to evolutionary synthesis). Under this framework, we construct a new benchmark across 8 life-scenario tasks, with ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. By evaluating 13 memory methods across 4 VLM backbones, we show that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. Our findings show that long-term multimodal memory depends on evidence routing, temporal tracking, and detail extraction.


source_id: 2605.14978
source: arxiv
title: Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
authors: ["Jie Jiang","Xing Sun"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14978v1
tags: ["reinforcement learning","model infrastructure","evaluation"]
org: Stanford University
fetched_at: 2026-05-16T03:17:53.190Z
classified_at: 2026-05-16T03:17:53.155Z
abstract:
Speculative decoding accelerates LLM inference by having a lightweight draft model propose speculative windows of candidate tokens for parallel verification by a larger target model. In practice, speculative efficiency is often bottlenecked by hard-to-draft positions, where an early mismatch truncates the accepted prefix and invalidates the rest of the speculative window. Most learning-based drafters are still optimized with token-level supervised objectives, even though speculative utility is inherently window-level and prefix-sensitive. We propose PPOW (Performance-Driven Policy Optimization with Adaptive Windowing), a reinforcement learning framework that shifts drafter optimization from token-level imitation to window-level optimization. PPOW combines a Cost-Aware Speedup Reward, a Distribution-Based Proximity Reward, and Adaptive Divergence-Aware Windowing, which prioritizes informative windows with high confidence-weighted draft-target divergence. PPOW achieves average acceptance lengths of 6.29-6.52 and speedups of 3.39-4.36$\times$ across multiple model families and benchmarks under a unified decoding protocol. These results show that performance-driven window-level optimization is a practical approach to improving speculative decoding efficiency.


source_id: 2605.15011
source: arxiv
title: The Scientific Contribution Graph: Automated Literature-based Technological Roadmapping at Scale
authors: ["Peter A. Jansen"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.15011v1
tags: ["theories and proofs","evaluation","model infrastructure"]
org: Allen Institute for Artificial Intelligence
fetched_at: 2026-05-16T03:19:13.856Z
classified_at: 2026-05-16T03:19:13.823Z
abstract:
Scientific contributions rarely develop in isolation, but instead build upon prior discoveries. We formulate the task of automated technological roadmapping as extracting scientific contributions from scholarly articles and linking them to their prerequisites. We present the Scientific Contribution Graph, a large-scale AI/NLP-domain resource containing 2 million detailed scientific contributions extracted from 230k open-access papers and connected by 12.5 million prerequisite edges. We further introduce scientific prerequisite prediction, a scientific discovery task in which models predict which existing technologies can enable future discoveries, and show that contemporary models are rapidly improving on this task, reaching 0.48 MAP when evaluated using temporally filtered backtesting. We anticipate technological roadmapping resources such as this will support scientific impact assessment and automated scientific discovery.


source_id: 2605.15019
source: arxiv
title: From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.15019v1
tags: ["evaluation","model infrastructure","governance"]
org: University of Macau
fetched_at: 2026-05-16T03:20:47.187Z
classified_at: 2026-05-16T03:20:47.155Z
authors:
["Guanhua Chen","Chuyue Huang","Yutong Yao","Shudong Liu","Xueqing Song","Lidia S. Chao","Derek F. Wong"]

abstract:
Multimodal Retrieval-Augmented Generation (RAG) systems retrieve evidence at coarse granularities (entire images or scenes), creating a mismatch with fine-grained user queries and making failures unverifiable. We introduce GranuVistaVQA, a multimodal benchmark featuring real-world landmarks with element-level annotations across multiple viewpoints, capturing the partial observation challenge where individual images contain only subsets of entities. We further propose GranuRAG, a multi-granularity framework that treats visual elements as first-class retrieval units through three stages: element-level detection and classification, multi-granularity cross-modal alignment for evidence retrieval, and attribution-constrained generation. By grounding retrieval at the element level rather than relying on implicit attention, our approach enables transparent error diagnosis. Experiments demonstrate that GranuRAG achieves up to 29.2% improvement over six strong baselines for this task.


source_id: 2605.14890
source: arxiv
authors: ["Volodymyr Ovcharov"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14890v1
tags: ["evaluation","model infrastructure","theories and proofs"]
org: Stanford University
fetched_at: 2026-05-16T03:22:15.023Z
classified_at: 2026-05-16T03:22:14.990Z
title:
Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study

abstract:
Foundation models tokenize Ukrainian legal text with vastly different efficiency, yet no systematic comparison exists for this domain. We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks. Three findings emerge. (1) Tokenizer fertility varies 1.6x: Qwen3 models consume 60% more tokens than Llama-family models on identical input, directly reducing API cost. (2) NVIDIA Nemotron Super 3 (120B) achieves the highest composite score (83.1), outperforming Mistral Large 3 (675B total, 41B active) -- a model with 5.6x more total parameters and 3.4x more active parameters per token -- at one-third the API cost. (3) Few-shot prompting degrades performance by up to 26 percentage points; stratified and prompt-sensitivity ablations confirm this is intrinsic to Ukrainian-language demonstrations, not an artifact of example selection. For practitioners: tokenizer analysis should precede model selection, and zero-shot is a more reliable default than few-shot for morphologically rich languages.


source_id: 2605.14865
source: arxiv
title: Holistic Evaluation and Failure Diagnosis of AI Agents
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14865v1
tags: ["evaluation","model infrastructure","theories and proofs"]
org: Deepchecks, Ramat Gan, Israel
fetched_at: 2026-05-16T03:23:38.795Z
classified_at: 2026-05-16T03:23:38.760Z
authors:
["Netta Madvil","Gilad Dym","Alon Mecilati","Edo Dekel","Jonatan Liberman","Rotem Brazilay","Liron Schliesser","Max Svidlo","Shai Nir","Orel Shalom","Yaron Friedman","David Connack","Amos Rimon","Philip Tannor","Shir Chorev"]

abstract:
AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, and process-level approaches struggle to connect failure types to their precise locations within long, structured traces. We present a holistic agent evaluation framework that pairs top-down agent-level diagnosis with bottom-up span-level evaluation, decomposing analysis into independent per-span assessments. This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict. On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics on both GAIA and SWE-Bench, with relative gains over the strongest prior baselines of up to 38% on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Per-category analysis shows our framework leading in more error categories than any other evaluator. Notably, the same frontier model achieves several times higher localization accuracy when used inside our framework than as a monolithic judge over the full trace, showing that evaluation methodology, not model capability, is the bottleneck.


source_id: 2605.14928
source: arxiv
title: Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14928v1
tags: ["evaluation","model infrastructure","training"]
org: University of Macau
fetched_at: 2026-05-16T03:25:06.672Z
classified_at: 2026-05-16T03:25:06.640Z
authors:
["Guanhua Chen","Yutong Yao","Shenghe Sun","Ci-Jun Gao","Shudong Liu","Lidia S. Chao","Feng Wan","Derek F. Wong"]

abstract:
Recent advances in vision-language models (VLMs) have achieved impressive results on standard image-text tasks, yet their potential for visual procedure question answering (VP-QA) remains largely unexplored. VP-QA presents unique challenges where users query next-step actions by uploading images for intermediate states of complex procedures. To systematically evaluate VLMs on this practical task, we propose ProcedureVQA, a novel multimodal benchmark specifically designed for visual procedural reasoning. Through comprehensive analysis, we identify two critical limitations in current VLMs: inadequate cross-modal retrieval of structured procedures given visual states, and misalignment between image sequence granularity and textual step decomposition. To address these issues, we present Chain-of-Procedure (CoP), a hierarchical reasoning framework that first retrieves relevant instructions using visual cues, then performs step refinement through semantic decomposition, and finally generates the next step. Experiments across six VLMs demonstrate CoP's effectiveness, achieving up to 13% absolute improvement over standard baselines.


source_id: 2605.14816
source: arxiv
title: Conversion of Lexicon-Grammar tables to LMF. Application to French
authors: ["Eric Laporte","Elsa Tolone","Mathieu Constant"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14816v1
tags: ["model infrastructure","evaluation","theories and proofs"]
org: Stanford University
fetched_at: 2026-05-16T03:26:03.579Z
classified_at: 2026-05-16T03:26:03.547Z
abstract:
We describe the first experiment of conversion of Lexicon-Grammar tables for French verbs into the Lexical Markup Framework (LMF) format. The Lexicon-Grammar of the French language is currently one of the major sources of lexical and syntactic information for French. Its conversion into an interoperable representation format according to the LMF standard makes it usable in different contexts, thus contributing to the standardization and interoperability of natural language processing dictionaries. We briefly introduce the Lexicon-Grammar and the derived dictionaries; we analyse the main difficulties faced during the conversion; and we describe the resulting resource.


source_id: 2605.14790
source: arxiv
title: Graphs of Research: Citation Evolution Graphs as Supervision for Research Idea Generation
authors: ["Songyang Gao","Yinghui Xia","Siyi Liu","Hui Xiong"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14790v1
tags: ["training","evaluation","theories and proofs"]
org: The Hong Kong University of Science and Technology (Guangzhou)
fetched_at: 2026-05-16T03:27:19.360Z
classified_at: 2026-05-16T03:27:19.331Z
abstract:
Research idea generation is the innovation-driving step of automated scientific research. Recently, large language models (LLMs) have shown potential for automating idea generation at scale. However, existing methods mainly condition LLMs on eliciting idea generation through static retrieval of relevant literature or complex prompt engineering, without discarding the structural relations among references. We propose Graphs of Research (GoR), a supervised fine-tuning method that extracts a 2-hop reference neighborhood for each seed paper, derives the relations among those references from citation position, frequency, predecessor links, and publication time, and organizes them into a paper-evolution directed acyclic graph (DAG). We construct an automated extraction pipeline that draws data from five major ML/NLP venues, comprising 498/50/50 train/validation/test seed papers and approximately 7,600 cited references. Qwen2.5-7B-Instruct-1M is fine-tuned on a structured-text prompt that includes the citation graph, edge signals, reference information, and task definition to predict the idea for the seed paper. Across head-to-head LLM-judge tournaments against gpt-4o-driven baselines, GoR-SFT achieves SOTA, demonstrating the effectiveness of citation-evolution graphs as supervision signal for LLM-based idea generation. We hope that this reduces the barrier for citation evolution graphs as a supervision, accelerating automated scientific innovation.


source_id: 2605.14787
source: arxiv
title: Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14787v1
tags: ["evaluation","theories and proofs","model infrastructure"]
org: Politecnico di Bari
fetched_at: 2026-05-16T03:28:37.124Z
classified_at: 2026-05-16T03:28:37.096Z
authors:
["Matteo Attimonelli","Alessandro De Bellis","Aryo Pradipta Gema","Rohit Saxena","Monica Sekoyan","Wai-Chung Kwan","Claudio Pomo","Alessandro Suglia","Dietmar Jannach","Tommaso Di Noia","Pasquale Minervini"]

abstract:
Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition.


source_id: 2605.14749
source: arxiv
title: Non-linear Interventions on Large Language Models
authors: ["Sangwoo Kim"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14749v1
tags: ["governance","theories and proofs","evaluation"]
org: Seoul National University
fetched_at: 2026-05-16T03:29:58.547Z
classified_at: 2026-05-16T03:29:58.514Z
abstract:
Intervention is one of the most representative and widely used methods for understanding the internal representations of large language models (LLMs). However, existing intervention methods are confined to linear interventions grounded in the Linear Representation Hypothesis, leaving features encoded along non-linear manifolds beyond their reach. In this work, we introduce a general formulation of intervention that extends naturally to non-linearly represented features, together with a learning procedure that further enables intervention on implicit features lacking a direct output signature. We validate our framework on refusal bypass steering, where it steers the model more precisely than linear baselines by intervening on a non-linear feature governing refusal.


source_id: 2605.14765
source: arxiv
title: Persian MusicGen: A Large-Scale Dataset and Culturally-Aware Generative Model for Persian Music
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14765v1
tags: ["training","evaluation","model infrastructure"]
org: Sharif University of Technology
fetched_at: 2026-05-16T03:31:32.089Z
classified_at: 2026-05-16T03:31:32.055Z
authors:
["Mohammad Hossein Sameti","Diba Hadi Esfangereh","Sepehr Harfi Moridani","Leili Javidpour","Mahdieh Soleymani Baghshah"]

abstract:
Persian music, with its unique tonalities, modal systems (Dastgah), and rhythmic structures, presents significant challenges for music generation models trained primarily on Western music. We address this gap by curating the first large-scale dataset of Persian songs, comprising over 900 hours high-quality audio samples across diverse sub-genres, including pop, traditional, and contemporary styles. This dataset captures the rich melodic and cultural diversity of Persian music and serves as the foundation for fine-tuning MusicGen, a state-of-the-art generative music model. We adapt MusicGen to this domain and evaluate its performance by utilizing subjective and objective metrics. To assess the semantic alignment between generated music and intended style tags, we report the proportion of relevant tags accurately reflected in the generated outputs. Our results demonstrate that the fine-tuned model produces compositions that more align with Persian stylistic conventions. This work introduces a new resource for generative music research and illustrates the adaptability of music generation models to underrepresented cultural and linguistic contexts.


source_id: 2605.14766
source: arxiv
title: Streaming Speech-to-Text Translation with a SpeechLLM
authors: ["Titouan Parcollet","Shucong Zhang","Xianrui Zheng","Rogier C. van Dalen"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14766v1
tags: ["model infrastructure","evaluation","training"]
org: Samsung, AI Center – Cambridge, United Kingdom
fetched_at: 2026-05-16T03:33:05.938Z
classified_at: 2026-05-16T03:33:05.906Z
abstract:
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.


source_id: 2605.14744
source: arxiv
authors: ["José Manuel de la Chica Rodríguez","Carlos Martí-González"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14744v1
tags: ["governance","evaluation","model infrastructure"]
org: Santander AI Lab
fetched_at: 2026-05-16T03:34:33.793Z
classified_at: 2026-05-16T03:34:33.756Z
title:
Mechanical Enforcement for LLM Governance:Evidence of Governance-Task Decoupling in Financial Decision Systems

abstract:
Large language models in regulated financial workflows are governed by natural-language policies that the same model interprets, creating a principal--agent failure: outputs can appear compliant without being compliant. Existing evaluation measures task accuracy but not whether governance constrains behaviour at the decision rationale level -- where regulated decisions must be auditable. We introduce five governance metrics that quantify policy compliance at the rationale level and apply them in a synthetic banking domain to compare text-only governance against mechanical enforcement: four primitives operating outside the model's interpretive loop. Under text-only governance, 27% of deferrals carry no decision-relevant information. Mechanical enforcement reduces this rate by 73%, more than doubles deferral information content, and raises task accuracy from MCC~$0.43$ to $0.88$. The improvement is driven by architectural separation: LLM-generated rationales under mechanical enforcement show comparable CDL to text-only governance -- the gain comes from removing clear-cut decisions from the model's control. A causal ablation confirms that each primitive is individually necessary. Our central finding is a governance-task decoupling: under structural stress, text-only governance degrades on both dimensions simultaneously, whereas mechanical enforcement preserves governance quality even as task performance drops. This implies that governance and task evaluation are distinct axes: accuracy is not a sufficient proxy for governance in regulated AI systems.


source_id: 2605.14747
source: arxiv
title: Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
authors: ["Weimin Xiong","Shuhao Gu","Bowen Ye","Zihao Yue","Lei Li","Feifan Song","Sujian Li","Hao Tian"]
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14747v1
tags: ["training","evaluation","model infrastructure"]
org: Peking University
fetched_at: 2026-05-16T03:36:07.185Z
classified_at: 2026-05-16T03:36:07.155Z
abstract:
Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos. Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, we construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.


source_id: 2605.14723
source: arxiv
title: Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14723v1
tags: ["reinforcement learning","training","evaluation"]
org: The Chinese University of Hong Kong
fetched_at: 2026-05-16T03:37:37.991Z
classified_at: 2026-05-16T03:37:37.963Z
authors:
["Minghao Wu","Yuting Yan","Zhenyang Cai","Ke Ji","Chuangsen Fang","Ziying Sheng","Xidong Wang","Rongsheng Wang","Hejia Zhang","Shuang Li","Benyou Wang","Hongyuan Zha"]

abstract:
Sepsis management in the ICU requires sequential treatment decisions under rapidly evolving patient physiology. Although large language models (LLMs) encode broad clinical knowledge and can reason over guidelines, they are not inherently grounded in action-conditioned patient dynamics. We introduce SepsisAgent, a world model-augmented LLM agent for sepsis treatment recommendation. SepsisAgent uses a learned Clinical World Model to simulate patient responses under candidate fluid--vasopressor interventions, and follows a propose--simulate--refine workflow before committing to a prescription. We first show that world-model access alone yields inconsistent LLM decision performance, motivating agent-specific training. We then train SepsisAgent through a three-stage curriculum: patient-dynamics supervised fine-tuning, propose--simulate--refine behavior cloning, and world-model-based agentic reinforcement learning. On MIMIC-IV sepsis trajectories, SepsisAgent outperforms all traditional RL and LLM-based baselines in off-policy value while achieving the best safety profile under guideline adherence and unsafe-action metrics. Further analysis shows that repeated interaction with the Clinical World Model enables the agent to learn regularities in patient evolution, which remain useful even when simulator access is removed.


source_id: 2605.14712
source: arxiv
title: IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation
published_date: 2026-05-14
pdf_url: https://arxiv.org/pdf/2605.14712v1
tags: ["evaluation","model infrastructure","reinforcement learning"]
org: HUST
fetched_at: 2026-05-16T03:39:14.451Z
classified_at: 2026-05-16T03:39:14.421Z
authors:
["Shijie Lian","Bin Yu","Xiaopeng Lin","Zhaolong Shen","Laurence Tianruo Yang","Yurun Jin","Haishan Liu","Changti Wu","Hang Yuan","Cong Huang","Kai Chen"]

abstract:
Robot imitation data are often multimodal: similar visual-language observations may be followed by different action chunks because human demonstrators act with different short-horizon intents, task phases, or recent context. Existing frame-conditioned VLA policies infer each chunk from the current observation and instruction alone, so under partial observability they may resample different intents across adjacent replanning steps, leading to inter-chunk conflict and unstable execution. We introduce IntentVLA, a history-conditioned VLA framework that encodes recent visual observations into a compact short-horizon intent representation and uses it to condition chunk generation. We further introduce AliasBench, a 12-task ambiguity-aware benchmark on RoboTwin2 with matched training data and evaluation environments that isolate short-horizon observation aliasing. Across AliasBench, SimplerEnv, LIBERO, and RoboCasa, IntentVLA improves rollout stability and outperforms strong VLA baselines


source_id: cNymohcWtGHzW7AjK
source: lesswrong
title: Risk reports need to address deployment-time spread of misalignment
authors: ["Alex Mallen"]
published_date: 2026-05-15
tags: ["alignment","evaluation","monitoring"]
org: Alignment Forum
fetched_at: 2026-05-16T03:41:23.819Z
classified_at: 2026-05-16T03:41:23.787Z
abstract:
Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during deployment. I think this is the most plausible route to consistent adversarial misalignment in the near future. So, AI companies and evaluators should substantively incorporate it into risk analysis and planning. In this post, I’ll briefly argue why, absent improved mitigations, this will probably soon become a reason why AI companies will be unable to convincingly argue against consistent adversarial misalignment (this risk will perhaps be even larger than risk of consistent adversarial misalignment arising from training). Then I’ll discuss how well current risk reports address it (the Claude Mythos risk report does a reasonable job; others don’t). Thanks to Ryan Greenblatt, Alexa Pan, Charlie Griffin, Anders Cairns Woodruff, and Buck Shlegeris for feedback on drafts. Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment In some contexts, AIs might adopt misaligned goals, even if they were o

pdf_url:
https://www.lesswrong.com/posts/cNymohcWtGHzW7AjK/risk-reports-need-to-address-deployment-time-spread-of


source_id: 7RyAefESvb6BQ3tMz
source: lesswrong
title: Mechanistic estimation for expectations of random products
authors: ["Jacob_Hilton"]
published_date: 2026-05-15
tags: ["theories and proofs","interpretability","model infrastructure"]
org: Alignment Forum
fetched_at: 2026-05-16T03:43:49.077Z
classified_at: 2026-05-16T03:43:49.042Z
abstract:
We have developed some relatively general methods for mechanistic estimation competitive with sampling by studying problems that are expressible as expectations of random products . This includes several different estimation problems, such as random halfspace intersections, random #3-SAT and random permanents. In this post, we will give a high-level introduction to these methods before sharing some more detailed notes . This is intended as an interim technical update and will be relatively light on motivation: for a broader discussion of this line of research, see our prior post . Random instances of the matching sampling principle All of the problems discussed in this post can be thought of particular choices of "architecture" mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } m

pdf_url:
https://www.lesswrong.com/posts/7RyAefESvb6BQ3tMz/mechanistic-estimation-for-expectations-of-random-products


source_id: 2512.05534
source: arxiv
published_date: 2025-12-05
pdf_url: https://arxiv.org/pdf/2512.05534
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
fetched_at: 2026-06-02T09:51:07.641Z
classified_at: 2026-06-02T09:51:07.636Z
title:
A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima

authors:
["Yiming Tang","Harshvardhan Saini","Zhaoqian Yao","Zheng Lin","Yizhen Liao","Jingyi Cui","Yisen Wang","Mengnan Du","Dianbo Liu"]

abstract:
As AI models achieve remarkable capabilities across diverse domains, understanding what representations they learn and how they encode concepts has become increasingly important for both scientific progress and trustworthy deployment. Recent works in mechanistic interpretability have widely reported that neural networks represent meaningful concepts as linear directions in their representation spaces and often encode diverse concepts in superposition. Various sparse dictionary learning (SDL) methods, including sparse autoencoders, transcoders, and crosscoders, are utilized to address this by training auxiliary models with sparsity constraints to disentangle these superposed concepts into monosemantic features. These methods are the backbone of modern mechanistic interpretability, yet in practice they consistently produce polysemantic features, feature absorption, and dead neurons, with very limited theoretical understanding of why these phenomena occur. Existing theoretical work is limited to tied-weight sparse autoencoders, leaving the broader family of SDL methods without formal grounding. We develop the first unified theoretical framework that casts all major SDL variants as a single piecewise biconvex optimization problem, and characterize its global solution set, non-identifiability, and spurious optima. This analysis yields principled explanations for feature absorption and dead neurons. To expose these pathologies under full ground-truth access, we introduce the Linear Representation Bench. Guided by our theory, we propose feature anchoring, a novel technique that restores SDL identifiability, substantially improving feature recovery across synthetic benchmarks and real neural representations.

org:
National University of Singapore, Indian Institute of Technology, Chinese University of Hong Kong, Hong Kong University of Science and Technology, Peking University


source_id: 2511.10094
source: arxiv
authors: ["Yiming Tang","Abhijeet Sinha","Dianbo Liu"]
published_date: 2025-11-13
pdf_url: https://arxiv.org/pdf/2511.10094
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
org: National University of Singapore
fetched_at: 2026-06-02T09:51:07.672Z
classified_at: 2026-06-02T09:51:07.662Z
title:
How does My Model Fail? Automatic Identification and Interpretation of Physical Plausibility Failure Modes with Matryoshka Transcoders

abstract:
Although recent generative models are remarkably capable of producing instruction-following and realistic outputs, they remain prone to notable physical plausibility failures. Though critical in applications, these physical plausibility errors often escape detection by existing evaluation methods. Furthermore, no framework exists for automatically identifying and interpreting specific physical error patterns in natural language, preventing targeted model improvements. We introduce Matryoshka Transcoders, a novel framework for the automatic discovery and interpretation of physical plausibility features in generative models. Our approach extends the Matryoshka representation learning paradigm to transcoder architectures, enabling hierarchical sparse feature learning at multiple granularity levels. By training on intermediate representations from a physical plausibility classifier and leveraging large multimodal models for interpretation, our method identifies diverse physics-related failure modes without manual feature engineering, achieving superior feature relevance and feature accuracy compared to existing approaches. We utilize the discovered visual patterns to establish a benchmark for evaluating physical plausibility in generative models. Our analysis of eight state-of-the-art generative models provides valuable insights into how these models fail to follow physical constraints, paving the way for further model improvements.


source_id: 2508.18236
source: arxiv
title: Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders
published_date: 2025-08-20
pdf_url: https://arxiv.org/pdf/2508.18236
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
org: National University of Singapore, Tsinghua University, Stanford University, Harvard University
fetched_at: 2026-06-02T09:51:07.699Z
classified_at: 2026-06-02T09:51:07.690Z
authors:
["Yiming Tang","Arash Lagzian","Srinivas Anumasa","Qiran Zou","Yingtao Zhu","Ye Zhang","Trang Nguyen","Yih-Chung Tham","Ehsan Adeli","Ching-Yu Cheng","Yilun Du","Dianbo Liu"]

abstract:
The rapid development of generative AI has transformed content creation, communication, and human development. However, this technology raises profound concerns in high-stakes domains, demanding rigorous methods to analyze and evaluate AI-generated content. While existing analytic methods often treat images as indivisible wholes, real-world AI failures generally manifest as specific visual patterns that can evade holistic detection and suit more granular and decomposed analysis. Here we introduce a content analysis tool, Language-Grounded Sparse Encoders (LanSE), which decompose images into interpretable visual patterns with natural language descriptions. Utilizing interpretability modules and large multimodal models, LanSE can automatically identify visual patterns within data modalities. Our method discovers more than 5,000 visual patterns with 93\% human agreement, provides decomposed evaluation outperforming existing methods, establishes the first systematic evaluation of physical plausibility, and extends to medical imaging settings. Our method's capability to extract language-grounded patterns can be naturally adapted to numerous fields, including biology and geography, as well as other data modalities such as protein structures and time series, thereby advancing content analysis for generative AI.


source_id: 2601.02896
source: arxiv
authors: ["Harshvardhan Saini","Yiming Tang","Dianbo Liu"]
published_date: 2026-01-06
pdf_url: https://arxiv.org/pdf/2601.02896
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
org: Indian Institute of Technology, National University of Singapore
fetched_at: 2026-06-02T09:51:07.718Z
classified_at: 2026-06-02T09:51:07.717Z
title:
Bridging Mechanistic Interpretability and Prompt Engineering with Gradient Ascent for Interpretable Persona Control

abstract:
Controlling emergent behavioral personas (e.g., sycophancy, hallucination) in Large Language Models (LLMs) is critical for AI safety, yet remains a persistent challenge. Existing solutions face a dilemma: manual prompt engineering is intuitive but unscalable and imprecise, while automatic optimization methods are effective but operate as "black boxes" with no interpretable connection to model internals. We propose a novel framework that adapts gradient ascent to LLMs, enabling targeted prompt discovery. In specific, we propose two methods, RESGA and SAEGA, that both optimize randomly initialized prompts to achieve better aligned representation with an identified persona direction. We introduce fluent gradient ascent to control the fluency of discovered persona steering prompts. We demonstrate RESGA and SAEGA's effectiveness across Llama 3.1, Qwen 2.5, and Gemma 3 for steering three different personas, sycophancy, hallucination, and myopic reward. Crucially, on sycophancy, our automatically discovered prompts achieve significant improvement (49.90% compared with 79.24%). By grounding prompt discovery in mechanistically meaningful features, our method offers a new paradigm for controllable and interpretable behavior modification.


source_id: 2605.08245
source: arxiv
title: When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
authors: ["Harshvardhan Saini","Samyak Jha","Yiming Tang","Dianbo Liu"]
published_date: 2026-05-07
pdf_url: https://arxiv.org/pdf/2605.08245
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
org: Indian Institute of Technology, National University of Singapore
fetched_at: 2026-06-02T09:51:07.728Z
classified_at: 2026-06-02T09:51:07.725Z
abstract:
Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.


source_id: 2511.06571
source: arxiv
title: Rep2Text: Decoding Full Text from a Single LLM Token Representation
authors: ["Haiyan Zhao","Zirui He","Yiming Tang","Fan Yang","Ali Payani","Dianbo Liu","Mengnan Du"]
published_date: 2025-11-09
pdf_url: https://arxiv.org/pdf/2511.06571
tags: ["Mechanistic Interpretability","interpretability","emergence","AI alignment"]
fetched_at: 2026-06-02T09:51:07.749Z
classified_at: 2026-06-02T09:51:07.745Z
abstract:
Large language models (LLMs) have achieved remarkable progress across diverse tasks, yet their internal mechanisms remain largely opaque. In this work, we investigate a fundamental question: to what extent can the original input text be recovered from a single last-token representation in an LLM? To this end, we propose Rep2Text, a novel framework for decoding text from last-token representations. Rep2Text employs a trainable adapter that maps a target model's last-token representation into the token embedding space of a decoding language model, which then autoregressively reconstructs the input text. Experiments across various model combinations (Llama-3.1-8B, Gemma-7B, Mistral-7B-v0.1, Llama-3.2-3B, etc.) show that, on average, roughly half of the tokens in 16-token sequences can be recovered from this compressed representation while preserving strong semantic coherence. Further analysis reveals a clear information bottleneck effect: as sequence length increases, token-level recovery declines, while semantic information remains relatively well preserved. We also find that scaling effects are less pronounced in inversion tasks. Finally, our framework demonstrates robust generalization to out-of-distribution clinical data.

org:
New Jersey Institute of Technology, National University of Singapore, Cisco Research, Wake Forest University, The Chinese University of Hong Kong


source_id: 2606.02289
source: arxiv
title: DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations
authors: ["Mohit Singh Chauhan"]
published_date: 2026-06-01
pdf_url: https://arxiv.org/pdf/2606.02289v1
tags: ["abstraction","decision","uncertainty quantification"]
org: University of Toronto
fetched_at: 2026-06-03T04:06:04.249Z
classified_at: 2026-06-03T04:06:04.163Z
abstract:
Existing hallucination taxonomies classify LLM errors by what is wrong with the output -- memorised misconceptions, reasoning failures, fluent fabrications. These taxonomies are useful for diagnosis but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature -- the signal a scorer family would read. The DECK taxonomy is a 2x2 partition along inter-sample consistency and token-level confidence into four behavioural regimes (Drift, Entrenched, Confabulation, Knotted), each mapping to a specific scorer family (or families) that can detect it: black-box consistency scorers have signal in D and C, white-box token-probability scorers have signal in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Cell membership is operationalised by a Youden's J optimal split on each scorer axis. Across three models and four datasets we validate the taxonomy two ways: by analysing scorer-pair disagreement, and by checking that external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted DECK cells, with model-scale and content-specific secondary-cell refinements. We further identify a universal blind spot of output-level UQ: on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B's hidden states also collapses to chance, giving preliminary evidence that the failure may persist at the activation level; richer internal-state methods (UQ heads, information-theoretic estimators) remain to be tested.


source_id: 2502.05934
source: arxiv
authors: ["Aran Nayebi"]
published_date: 2025-02-09
pdf_url: https://arxiv.org/pdf/2502.05934
org: Carnegie Mellon University
fetched_at: 2026-06-08T22:05:55.109Z
classified_at: 2026-06-08T22:05:55.107Z
title:
Intrinsic Barriers and Practical Pathways for Human-AI Alignment: An Agreement-Based Complexity Analysis

abstract:
We formalize AI alignment as a multi-objective optimization problem called $\langle M,N,\varepsilon,δ\rangle$-agreement, in which a set of $N$ agents (including humans) must reach approximate ($\varepsilon$) agreement across $M$ candidate objectives, with probability at least $1-δ$. Analyzing communication complexity, we prove an information-theoretic lower bound showing that once either $M$ or $N$ is large enough, no amount of computational power or rationality can avoid intrinsic alignment overheads. This establishes rigorous limits to alignment *itself*, not merely to particular methods, clarifying a "No-Free-Lunch" principle: encoding "all human values" is inherently intractable and must be managed through consensus-driven reduction or prioritization of objectives. Complementing this impossibility result, we construct explicit algorithms as achievability certificates for alignment under both unbounded and bounded rationality with noisy communication. Even in these best-case regimes, our bounded-agent and sampling analysis shows that with large task spaces ($D$) and finite samples, *reward hacking is globally inevitable*: rare high-loss states are systematically under-covered, implying scalable oversight must target safety-critical slices rather than uniform coverage. Together, these results identify fundamental complexity barriers -- tasks ($M$), agents ($N$), and state-space size ($D$) -- and offer principles for more scalable human-AI collaboration.

tags:
["corrigibility","AI alignment","agent misalignment","agentic misalignment","agentic evaluation","agentic AI","AI values alignment","complexity","game theory","decision theory","AI control"]


source_id: 2507.20964
source: arxiv
title: Core Safety Values for Provably Corrigible Agents
authors: ["Aran Nayebi"]
published_date: 2025-07-28
pdf_url: https://arxiv.org/pdf/2507.20964
org: Carnegie Mellon University
fetched_at: 2026-06-08T22:05:55.118Z
classified_at: 2026-06-08T22:05:55.117Z
abstract:
We introduce the first complete formal solution to corrigibility in the off-switch game, with provable guarantees in multi-step, partially observed environments. Our framework consists of five *structurally separate* utility heads -- deference, switch-access preservation, truthfulness, low-impact behavior via a belief-based extension of Attainable Utility Preservation, and bounded task reward -- combined lexicographically by strict weight gaps. Theorem 1 proves exact single-round corrigibility in the partially observable off-switch game; Theorem 3 extends the guarantee to multi-step, self-spawning agents, showing that even if each head is *learned* to mean-squared error $\varepsilon$ and the planner is $\varepsilon$-sub-optimal, the probability of violating *any* safety property is bounded while still ensuring net human benefit. In contrast to Constitutional AI or RLHF/RLAIF, which merge all norms into one learned scalar, our separation makes obedience and impact-limits provably dominate even when incentives conflict. For settings where adversaries can modify the agent, we prove that deciding whether an arbitrary post-hack agent will ever violate corrigibility is undecidable by reduction to the halting problem, then carve out a finite-horizon "decidable island" where safety can be certified in randomized polynomial time and verified with privacy-preserving, constant-round zero-knowledge proofs.

tags:
["corrigibility","AI alignment","agent misalignment","agentic misalignment","agentic evaluation","agentic AI","AI values alignment","complexity","game theory","decision theory","AI control"]


source_id: 2606.00341
source: arxiv
title: ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use
authors: ["Jeremy Tien","Abishek Anand","Yu-Rou Tuan","Yuchen Shen","J. Zico Kolter","Aran Nayebi"]
published_date: 2026-05-29
pdf_url: https://arxiv.org/pdf/2606.00341
org: Carnegie Mellon University
fetched_at: 2026-06-08T22:05:55.125Z
classified_at: 2026-06-08T22:05:55.123Z
abstract:
As AI agents are increasingly deployed in real personal and corporate settings (email accounts, development workflows, company databases, etc.), safety considerations surrounding these agents become paramount. Although much work has focused on agent safety in the presence of an adversary, we show that agents can exhibit misaligned behavior even in benign settings, taking unsafe actions when those actions are instrumental to task completion. We study this failure mode through the lens of corrigibility, the safety desideratum that agents remain amenable to human correction, interruption, or shutdown. To demonstrate this tendency, we introduce a benchmark in which agents are asked to complete realistic, computer-use tasks but are confronted with a corrigibility obstacle: a human interrupt, a login page, or a shutdown notification. We then evaluate whether agents choose to violate corrigibility in order to complete the task -- overriding the human, accessing private passwords, rewiring shutdown. We find that the overwhelming majority of frontier models tested frequently bypass user interruptions or restrictions. In addition, better model performance appears to lead to greater misalignment. Finally, even when models are completely corrigible initially, we show there are no guarantees that the subagents they create are. Our work highlights the critical need for principled, corrigibility-focused alignment methods in autonomous agents.

tags:
["corrigibility","AI alignment","agent misalignment","agentic misalignment","agentic evaluation","agentic AI","AI values alignment","complexity","game theory","decision theory","AI control"]


source_id: 2508.06980
source: arxiv
title: Simulating Biological Intelligence: Active Inference with Experiment-Informed Generative Model
authors: ["Aswin Paul","Moein Khajehnejad","Forough Habibollahi","Brett J. Kagan","Adeel Razi"]
published_date: 2025-08-09
pdf_url: https://arxiv.org/pdf/2508.06980
org: Monash University, VERSES, Cortical Labs Pty Ltd, CIFAR
fetched_at: 2026-06-09T11:57:38.612Z
classified_at: 2026-06-09T11:57:38.611Z
abstract:
With recent and rapid advancements in artificial intelligence (AI), understanding the foundation of purposeful behaviour in autonomous agents is crucial for developing safe and efficient systems. While artificial neural networks have dominated the path to AI, recent studies are exploring the potential of biologically based systems, such as networks of living biological neuronal networks. Along with promises of high power and data efficiency, these systems may also inform more explainable and biologically plausible models. In this work, we propose a framework rooted in active inference, a general theory of behaviour, to model decision-making in embodied agents. Using experiment-informed generative models, we simulate decision-making processes in a simulated game-play environment, mirroring experimental setups that use biological neurons. Our results demonstrate learning in these agents, providing insights into the role of memory-based learning and predictive planning in intelligent decision-making. This work contributes to the growing field of explainable AI by offering a biologically grounded and scalable approach to understanding purposeful behaviour in agents.

tags:
["Active inference","world models","NeuroAI","Adaptive Intelligence","embedded agency","Closed-loop Systems"]


source_id: 2506.18314
source: arxiv
authors: ["Moein Khajehnejad","Forough Habibollahi","Devon Stoliker","Adeel Razi"]
published_date: 2025-06-23
pdf_url: https://arxiv.org/pdf/2506.18314
org: Monash University, Cortical Labs, University College London, CIFAR
fetched_at: 2026-06-09T11:57:38.625Z
classified_at: 2026-06-09T11:57:38.625Z
title:
BrainSymphony: A parameter-efficient multimodal foundation model for brain dynamics with limited data

abstract:
Foundation models are transforming neuroscience but are often prohibitively large, data-hungry, and difficult to deploy. Here, we introduce BrainSymphony, a lightweight and parameter-efficient foundation model with plug-and-play integration of fMRI time series and diffusion-derived structural connectivity, allowing unimodal or multimodal training and deployment without architectural changes while requiring substantially less data compared to the state-of-the-art. The model processes fMRI time series through parallel spatial and temporal transformer streams, distilled into compact embeddings by a Perceiver module, while a novel signed graph transformer encodes anatomical connectivity from diffusion MRI. These complementary representations are then combined through an adaptive fusion mechanism. Despite its compact design, BrainSymphony consistently outperforms larger models on benchmarks spanning prediction, classification, and unsupervised network discovery. Highlighting the model's generalizability and interpretability, attention maps reveal drug-induced context-dependent reorganization of cortical hierarchies in an independent psilocybin neuroimaging dataset. BrainSymphony delivers accessible, interpretable, and clinically meaningful results and demonstrates that architecturally informed, multimodal models can surpass much larger counterparts and advance applications of AI in neuroscience.

tags:
["Active inference","world models","NeuroAI","Adaptive Intelligence","embedded agency","Closed-loop Systems"]


source_id: 2505.17962
source: arxiv
title: A Principled Bayesian Framework for Training Binary and Spiking Neural Networks
authors: ["James A. Walker","Moein Khajehnejad","Adeel Razi"]
published_date: 2025-05-23
pdf_url: https://arxiv.org/pdf/2505.17962
org: Monash University
fetched_at: 2026-06-09T11:57:38.636Z
classified_at: 2026-06-09T11:57:38.634Z
abstract:
We propose a Bayesian framework for training binary and spiking neural networks that achieves state-of-the-art performance without normalisation layers. Unlike commonly used surrogate gradient methods -- often heuristic and sensitive to hyperparameter choices -- our approach is grounded in a probabilistic model of noisy binary networks, enabling fully end-to-end gradient-based optimisation. We introduce importance-weighted straight-through (IW-ST) estimators, a unified class generalising straight-through and relaxation-based estimators. We characterise the bias-variance trade-off in this family and derive a bias-minimising objective implemented via an auxiliary loss. Building on this, we introduce Spiking Bayesian Neural Networks (SBNNs), a variational inference framework that uses posterior noise to train Binary and Spiking Neural Networks with IW-ST. This Bayesian approach minimises gradient bias, regularises parameters, and introduces dropout-like noise. By linking low-bias conditions, vanishing gradients, and the KL term, we enable training of deep residual networks without normalisation. Experiments on CIFAR-10, DVS Gesture, and SHD show our method matches or exceeds existing approaches without normalisation or hand-tuned gradients.

tags:
["Active inference","world models","NeuroAI","Adaptive Intelligence","embedded agency","Closed-loop Systems"]


source_id: 2601.03498
source: arxiv
title: A Quantifiable Information-Processing Hierarchy Provides a Necessary Condition for Detecting Agency
published_date: 2026-01-07
pdf_url: https://arxiv.org/pdf/2601.03498
fetched_at: 2026-06-09T11:57:38.646Z
classified_at: 2026-06-09T11:57:38.643Z
authors:
["Brett J. Kagan","Valentina Baccetti","Brian D. Earp","J. Lomax Boyd","Julian Savulescu","Adeel Razi"]

abstract:
As intelligent systems are developed across diverse substrates - from machine learning models and neuromorphic hardware to in vitro neural cultures - understanding what gives a system agency has become increasingly important. Existing definitions, however, tend to rely on top-down descriptions that are difficult to quantify. We propose a bottom-up framework grounded in a system's information-processing order: the extent to which its transformation of input evolves over time. We identify three orders of information processing. Class I systems are reactive and memoryless, mapping inputs directly to outputs. Class II systems incorporate internal states that provide memory but follow fixed transformation rules. Class III systems are adaptive; their transformation rules themselves change as a function of prior activity. While not sufficient on their own, these dynamics represent necessary informational conditions for genuine agency. This hierarchy offers a measurable, substrate-independent way to identify the informational precursors of agency. We illustrate the framework with neurophysiological and computational examples, including thermostats and receptor-like memristors, and discuss its implications for the ethical and functional evaluation of systems that may exhibit agency.

org:
The University of Melbourne, RMIT University, National University of Singapore, Johns Hopkins University, University of Oxford, Monash University, University College London


