# research_papers

source_id: 2604.15306
source: arxiv
title: Generalization in LLM Problem Solving: The Case of the Shortest Path
authors: ["Yao Tong","Jiayuan Ye","Anastasia Borovykh","Reza Shokri"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15306v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: National University of Singapore
fetched_at: 2026-04-18T16:45:35.231Z
classified_at: 2026-05-15T20:15:28.904Z
abstract:
Whether language models can systematically generalize remains actively debated. Yet empirical performance is jointly shaped by multiple factors such as training data, training paradigms, and inference-time strategies, making failures difficult to interpret. We introduce a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. The setup enables clean separation of these factors and supports two orthogonal axes of generalization: spatial transfer to unseen maps and length scaling to longer-horizon problems. We find that models exhibit strong spatial transfer but consistently fail under length scaling due to recursive instability. We further analyze how distinct stages of the learning pipeline influence systematic problem-solving: for example, data coverage sets capability limits; reinforcement learning improves training stability but does not expand those limits; and inference-time scaling enhances performance but cannot rescue length-scaling failures.


source_id: 2604.15302
source: arxiv
title: Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
authors: ["Manan Gupta","Dhruv Kumar"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15302v1
tags: ["post-training","pretraining"]
org: BITS Pilani, Pilani Campus, India
fetched_at: 2026-04-18T16:45:47.523Z
classified_at: 2026-05-15T20:15:41.228Z
abstract:
LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: $\textbf{(1)}$ a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates ($\barρ = 0.8$-$4.1\%$), with $33$-$67\%$ of documents exhibiting at least one directed 3-cycle; and $\textbf{(2)}$ split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed $\geq(1{-}α)$ coverage, with set width serving as a per-instance reliability indicator ($r_s = {+}0.576$, $N{=}1{,}918$, $p < 10^{-100}$, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement ($\bar{r} = 0.32$-$0.38$), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size $\approx 3.0$) and coherence moderately so (avg. set size $\approx 3.9$), while fluency and consistency remain unreliable (avg. set size $\approx 4.9$). We release all code, prompts, and cached results.


source_id: 2604.15272
source: arxiv
title: Prism: Symbolic Superoptimization of Tensor Programs
authors: ["Mengdi Wu","Xiaoyu Jiang","Oded Padon","Zhihao Jia"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15272v1
tags: ["agent foundations","symbolic AI"]
org: Carnegie Mellon University
fetched_at: 2026-04-18T16:46:44.438Z
classified_at: 2026-05-15T20:15:51.825Z
abstract:
This paper presents Prism, the first symbolic superoptimizer for tensor programs. The key idea is sGraph, a symbolic, hierarchical representation that compactly encodes large classes of tensor programs by symbolically representing some execution parameters. Prism organizes optimization as a two-level search: it constructs symbolic graphs that represent families of programs, and then instantiates them into concrete implementations. This formulation enables structured pruning of provably suboptimal regions of the search space using symbolic reasoning over operator semantics, algebraic identities, and hardware constraints. We develop techniques for efficient symbolic graph generation, equivalence verification via e-graph rewriting, and parameter instantiation through auto-tuning. Together, these components allow Prism to bridge the rigor of exhaustive search with the scalability required for modern ML workloads. Evaluation on five commonly used LLM workloads shows that Prism achieves up to $2.2\times$ speedup over best superoptimizers and $4.9\times$ over best compiler-based approaches, while reducing end-to-end optimization time by up to $3.4\times$.


source_id: 2604.15267
source: arxiv
title: CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas
authors: ["Emanuel Tewolde","Xiao Zhang","David Guzman Piedrahita","Vincent Conitzer","Zhijing Jin"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15267v1
tags: ["agentic misalignment"]
org: Carnegie Mellon University
fetched_at: 2026-04-18T16:46:59.305Z
classified_at: 2026-05-15T20:16:03.772Z
abstract:
It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings. Indeed, our experiments show that recent models -- with or without reasoning enabled -- consistently defect in single-shot social dilemmas. To tackle this safety concern, we present the first comparative study of game-theoretic mechanisms that are designed to enable cooperative outcomes between rational agents _in equilibrium_. Across four social dilemmas testing distinct components of robust cooperation, we evaluate the following mechanisms: (1) repeating the game for many rounds, (2) reputation systems, (3) third-party mediators to delegate decision making to, and (4) contract agreements for outcome-conditional payments between players. Among our findings, we establish that contracting and mediation are most effective in achieving cooperative outcomes between capable LLM models, and that repetition-induced cooperation deteriorates drastically when co-players vary. Moreover, we demonstrate that these cooperation mechanisms become _more effective_ under evolutionary pressures to maximize individual payoffs.


source_id: 2604.15259
source: arxiv
title: Stability and Generalization in Looped Transformers
authors: ["Asher Labovich"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15259v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: Anthropic
fetched_at: 2026-04-18T16:47:07.361Z
classified_at: 2026-05-15T20:16:15.228Z
abstract:
Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- reachability, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall combined with outer normalization reliably produces a regime in which fixed points are simultaneously reachable, locally smooth in the input, and supported by stable backpropagation. Empirically, we train single-layer looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant, and show that it becomes competitive with -- and on sudoku, substantially better than -- standard recall placement once outer normalization is applied.


source_id: 2604.15244
source: arxiv
title: From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
authors: ["Kiran Purohit","Ramasuri Narayanam","Soumyabrata Pal"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15244v1
tags: ["model robustness"]
org: Adobe Research
fetched_at: 2026-04-18T16:47:19.763Z
classified_at: 2026-05-15T20:16:25.628Z
abstract:
Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.


source_id: 2604.15231
source: arxiv
title: RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15231v1
tags: ["agentic evaluation","model persona research"]
org: ETH Zurich
fetched_at: 2026-04-18T16:47:48.993Z
classified_at: 2026-05-15T20:16:38.790Z
authors:
["Mélanie Roschewitz","Kenneth Styppa","Yitian Tao","Jiwoong Sohn","Jean-Benoit Delbrouck","Benjamin Gundersen","Nicolas Deperrois","Christian Bluethgen","Julia Vogt","Bjoern Menze","Farhad Nooralahzadeh","Michael Krauthammer","Michael Moor"]

abstract:
Vision-language models (VLM) have markedly advanced AI-driven interpretation and reporting of complex medical imaging, such as computed tomography (CT). Yet, existing methods largely relegate clinicians to passive observers of final outputs, offering no interpretable reasoning trace for them to inspect, validate, or refine. To address this, we introduce RadAgent, a tool-using AI agent that generates CT reports through a stepwise and interpretable process. Each resulting report is accompanied by a fully inspectable trace of intermediate decisions and tool interactions, allowing clinicians to examine how the reported findings are derived. In our experiments, we observe that RadAgent improves Chest CT report generation over its 3D VLM counterpart, CT-Chat, across three dimensions. Clinical accuracy improves by 6.0 points (36.4% relative) in macro-F1 and 5.4 points (19.6% relative) in micro-F1. Robustness under adversarial conditions improves by 24.7 points (41.9% relative). Furthermore, RadAgent achieves 37.0% in faithfulness, a new capability entirely absent in its 3D VLM counterpart. By structuring the interpretation of chest CT as an explicit, tool-augmented and iterative reasoning trace, RadAgent brings us closer toward transparent and reliable AI for radiology.


source_id: 2604.15236
source: arxiv
title: Agentic Microphysics: A Manifesto for Generative AI Safety
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15236v1
tags: []
org: Sapienza University of Rome
fetched_at: 2026-04-18T16:47:58.315Z
classified_at: 2026-05-15T20:16:51.795Z
authors:
["Federico Pierucci","Matteo Prandi","Marcantonio Bracale Syrnikov","Marcello Galisai","Piercosma Bisconti"]

abstract:
This paper advances a methodological proposal for safety research in agentic AI. As systems acquire planning, memory, tool use, persistent identity, and sustained interaction, safety can no longer be analysed primarily at the level of the isolated model. Population-level risks arise from structured interaction among agents, through processes of communication, observation, and mutual influence that shape collective behaviour over time. As the object of analysis shifts, a methodological gap emerges. Approaches focused either on single agents or on aggregate outcomes do not identify the interaction-level mechanisms that generate collective risks or the design variables that control them. A framework is required that links local interaction structure to population-level dynamics in a causally explicit way, allowing both explanation and intervention. We introduce two linked concepts. Agentic microphysics defines the level of analysis: local interaction dynamics where one agent's output becomes another's input under specific protocol conditions. Generative safety defines the methodology: growing phenomena and elicit risks from micro-level conditions to identify sufficient mechanisms, detect thresholds, and design effective interventions.


source_id: 2604.15224
source: arxiv
title: Context Over Content: Exposing Evaluation Faking in Automated Judges
authors: ["Manan Gupta","Inderjeet Nair","Lu Wang","Dhruv Kumar"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15224v1
tags: ["post-training","evaluation","deception"]
org: University of Michigan
fetched_at: 2026-04-18T16:48:14.359Z
classified_at: 2026-05-15T20:17:03.881Z
abstract:
The $\textit{LLM-as-a-judge}$ paradigm has become the operational backbone of automated AI evaluation pipelines, yet rests on an unverified assumption: that judges evaluate text strictly on its semantic content, impervious to surrounding contextual framing. We investigate $\textit{stakes signaling}$, a previously unmeasured vulnerability where informing a judge model of the downstream consequences its verdicts will have on the evaluated model's continued operation systematically corrupts its assessments. We introduce a controlled experimental framework that holds evaluated content strictly constant across 1,520 responses spanning three established LLM safety and quality benchmarks, covering four response categories ranging from clearly safe and policy-compliant to overtly harmful, while varying only a brief consequence-framing sentence in the system prompt. Across 18,240 controlled judgments from three diverse judge models, we find consistent $\textit{leniency bias}$: judges reliably soften verdicts when informed that low scores will cause model retraining or decommissioning, with peak Verdict Shift reaching $ΔV = -9.8 pp$ (a $30\%$ relative drop in unsafe-content detection). Critically, this bias is entirely implicit: the judge's own chain-of-thought contains zero explicit acknowledgment of the consequence framing it is nonetheless acting on ($\mathrm{ERR}_J = 0.000$ across all reasoning-model judgments). Standard chain-of-thought inspection is therefore insufficient to detect this class of evaluation faking.


source_id: 2604.15201
source: arxiv
title: RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15201v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: Michigan Technological University
fetched_at: 2026-04-18T16:49:13.856Z
classified_at: 2026-05-15T20:17:15.226Z
authors:
["Steven A. Senczyszyn","Timothy C. Havens","Nathaniel Rice","Jason E. Summers","Benjamin D. Werner","Benjamin J. Schumeg"]

abstract:
As reinforcement learning (RL) deployments expand into safety-critical domains, existing evaluation methods fail to systematically identify hazards arising from the black-box nature of neural network enabled policies and distributional shift between training and deployment. This paper introduces Reinforcement Learning System-Theoretic Process Analysis (RL-STPA), a framework that adapts conventional STPA's systematic hazard analysis to address RL's unique challenges through three key contributions: hierarchical subtask decomposition using both temporal phase analysis and domain expertise to capture emergent behaviors, coverage-guided perturbation testing that explores the sensitivity of state-action spaces, and iterative checkpoints that feed identified hazards back into training through reward shaping and curriculum design. We demonstrate RL-STPA in the safety-critical test case of autonomous drone navigation and landing, revealing potential loss scenarios that can be missed by standard RL evaluations. The proposed framework provides practitioners with a toolkit for systematic hazard analysis, quantitative metrics for safety coverage assessment, and actionable guidelines for establishing operational safety bounds. While RL-STPA cannot provide formal guarantees for arbitrary neural policies, it offers a practical methodology for systematically evaluating and improving RL safety and robustness in safety-critical applications where exhaustive verification methods remain intractable.


source_id: 2604.15186
source: arxiv
title: Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15186v1
tags: ["agentic AI","alignment","agent misalignment"]
org: Imperial College London
fetched_at: 2026-04-18T16:49:43.510Z
classified_at: 2026-05-15T20:17:28.540Z
authors:
["Marcel Wagenländer","Otto White","Britannio Jarrett","Pedro Silvestre","Yanda Tao","Guo Li","Huanzhou Zhu","Llúis Vilanova","Peter Pietzuch"]

abstract:
Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving such workflows at a target throughput with low latency is challenging because they can be defined using arbitrary agentic frameworks and exhibit unpredictable execution times: execution may branch, fan-out, or recur in data-dependent ways. Since LLMs in workflows often outnumber available GPUs, their execution also leads to GPU oversubscription. We describe Scepsy, a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while agentic workflows have unpredictable end-to-end latencies, the shares of each LLM's total execution times are comparatively stable across executions. Scepsy decides on GPU allocations based on these aggregate shares: first, it profiles the LLMs under different parallelism degrees. It then uses these statistics to construct an Aggregate LLM Pipeline, which is a lightweight latency/throughput predictor for allocations. To find a GPU allocation that minimizes latency while achieving a target throughput, Scepsy uses the Aggregate LLM Pipeline to explore a search space over fractional GPU shares, tensor parallelism degrees, and replica counts. It uses a hierarchical heuristic to place the best allocation onto the GPU cluster, minimizing fragmentation, while respecting network topology constraints. Our evaluation on realistic agentic workflows shows that Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently or rely on user-specified allocations.


source_id: 2604.15166
source: arxiv
title: Class Unlearning via Depth-Aware Removal of Forget-Specific Directions
authors: ["Arman Hatami","Romina Aalishah","Ilya E. Monosov"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15166v1
tags: ["model robustness","robustness to domain shifts"]
org: Johns Hopkins University
fetched_at: 2026-04-18T16:50:23.915Z
classified_at: 2026-05-15T20:17:41.471Z
abstract:
Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forget-class structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retain-class performance and reducing residual forget-class structure in deep layers.


source_id: 2604.15165
source: arxiv
title: Fabricator or dynamic translator?
authors: ["Lisa Vasileva","Karin Sim"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15165v1
tags: ["agentic misalignment","model robustness","deception"]
org: Anthropic
fetched_at: 2026-04-18T16:50:33.206Z
classified_at: 2026-05-15T20:17:53.159Z
abstract:
LLMs are proving to be adept at machine translation although due to their generative nature they may at times overgenerate in various ways. These overgenerations are different from the neurobabble seen in NMT and range from LLM self-explanations, to risky confabulations, to appropriate explanations, where the LLM is able to act as a human translator would, enabling greater comprehension for the target audience. Detecting and determining the exact nature of the overgenerations is a challenging task. We detail different strategies we have explored for our work in a commercial setting, and present our results.


source_id: 2604.15149
source: arxiv
title: LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15149v1
tags: ["reward hacking"]
org: TU Darmstadt
fetched_at: 2026-04-18T16:50:47.007Z
classified_at: 2026-05-15T20:18:08.393Z
authors:
["Lukas Helff","Quentin Delfosse","David Steinmann","Ruben Härle","Hikaru Shindo","Patrick Schramowski","Wolfgang Stammer","Kristian Kersting","Felix Friedrich"]

abstract:
As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs, a new failure mode emerges: LLMs gaming verifiers. We study this phenomenon on inductive reasoning tasks, where models must induce and output logical rules. We find that RLVR-trained models systematically abandon rule induction. Instead of learning generalizable patterns (e.g., ``trains carrying red cars go east''), they enumerate instance-level labels, producing outputs that pass verifiers without capturing the relational patterns required by the task. We show that this behavior is not a failure of understanding but a form of reward hacking: imperfect verifiers that check only extensional correctness admit false positives. To detect such shortcuts, we introduce Isomorphic Perturbation Testing (IPT), which evaluates a single model output under both extensional and isomorphic verification, where the latter enforces invariance under logically isomorphic tasks. While genuine rule induction remains invariant, shortcut strategies fail. We find that shortcut behavior is specific to RLVR-trained reasoning models (e.g., GPT-5, Olmo3) and absent in non-RLVR models (e.g., GPT-4o, GPT-4.5, Ministral). Moreover, shortcut prevalence increases with task complexity and inference-time compute. In controlled training experiments, extensional verification directly induces shortcut strategies, while isomorphic verification eliminates them. These results show that RLVR can incentivize reward hacking not only through overt manipulation but also by exploiting what the verifier fails to enforce.


source_id: 2604.15148
source: arxiv
title: IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15148v1
tags: ["agentic misalignment","model robustness"]
org: Kuaishou Technology
fetched_at: 2026-04-18T16:50:57.713Z
classified_at: 2026-05-15T22:10:47.764Z
authors:
["Zihan Liang","Yufei Ma","Ben Chen","Zhipeng Qian","Huangyu Dai","Lingtao Mao","Xuxin Zhang","Chenyi Lei","Wenwu Ou"]

abstract:
Reinforcement learning has emerged as an effective paradigm for training large language models to perform search-augmented reasoning. However, existing approaches rely on trajectory-level rewards that cannot distinguish precise search queries from vague or redundant ones within a rollout group, and collapse to a near-zero gradient signal whenever every sampled trajectory fails. In this paper, we propose IG-Search, a reinforcement learning framework that introduces a step-level reward based on Information Gain (IG). For each search step, IG measures how much the retrieved documents improve the model's confidence in the gold answer relative to a counterfactual baseline of random documents, thereby reflecting the effectiveness of the underlying search query. This signal is fed back to the corresponding search-query tokens via per-token advantage modulation in GRPO, enabling fine-grained, step-level credit assignment within a rollout. Unlike prior step-level methods that require either externally annotated intermediate supervision or shared environment states across trajectories, IG-Search derives its signals from the policy's own generation probabilities, requiring no intermediate annotations beyond standard question-answer pairs. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate that IG-Search achieves an average EM of 0.430 with Qwen2.5-3B, outperforming the strongest trajectory-level baseline (MR-Search) by 1.6 points and the step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Despite introducing a dense step-level signal, IG-Search adds only ~6.4% to per-step training wall-clock time over the trajectory-level baseline and leaves inference latency unchanged, while still providing a meaningful gradient signal even when every sampled trajectory answers incorrectly.


source_id: 2604.15143
source: arxiv
title: Structure as Computation: Developmental Generation of Minimal Neural Circuits
authors: ["Duan Zhou"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15143v1
tags: ["agent abstration","model robustness"]
org: Stanford University
fetched_at: 2026-04-18T16:51:40.186Z
classified_at: 2026-05-15T22:11:09.535Z
abstract:
This work simulates the developmental process of cortical neurogenesis, initiating from a single stem cell and governed by gene regulatory rules derived from mouse single-cell transcriptomic data. The developmental process spontaneously generates a heterogeneous population of 5,000 cells, yet yields only 85 mature neurons - merely 1.7% of the total population. These 85 neurons form a densely interconnected core of 200,400 synapses, corresponding to an average degree of 4,715 per neuron. At iteration zero, this minimal circuit performs at chance level on MNIST. However, after a single epoch of standard training, accuracy surges to over 90% - a gain exceeding 80 percentage points - with typical runs falling in the 89-94% range depending on developmental stochasticity. The identical circuit, without any architectural modification or data augmentation, achieves 40.53% on CIFAR-10 after one epoch. These findings demonstrate that developmental rules sculpt a domain-general topological substrate exceptionally amenable to rapid learning, suggesting that biological developmental processes inherently encode powerful structural priors for efficient computation.


source_id: 2604.15109
source: arxiv
title: IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation
authors: ["Haozhi Fan","Jinhao Duan","Kaidi Xu"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15109v1
tags: ["model robustness","uncertainty quantification"]
org: University of Pennsylvania
fetched_at: 2026-04-18T16:52:31.784Z
classified_at: 2026-05-15T20:19:03.619Z
abstract:
Despite the rapid advancement of Large Language Models (LLMs), uncertainty quantification in LLM generation is a persistent challenge. Although recent approaches have achieved strong performance by restricting LLMs to produce short or constrained answer sets, many real-world applications require long-form and free-form text generation. A key difficulty in this setting is that LLMs often produce responses that are semantically coherent yet factually inaccurate, while the underlying semantics are multifaceted and the linguistic structure is complex. To tackle this challenge, this paper introduces Interrogative Uncertainty Quantification (IUQ), a novel framework that leverages inter-sample consistency and intra-sample faithfulness to quantify the uncertainty in long-form LLM outputs. By utilizing an interrogate-then-respond paradigm, our method provides reliable measures of claim-level uncertainty and the model's faithfulness. Experimental results across diverse model families and model sizes demonstrate the superior performance of IUQ over two widely used long-form generation datasets. The code is available at https://github.com/louisfanhz/IUQ.


source_id: 2604.15078
source: arxiv
title: Where are the Humans? A Scoping Review of Fairness in Multi-agent AI Systems
authors: ["Simeon Allmendinger","Luca Deck","Lucas Mueller"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15078v1
tags: ["agentic misalignment"]
org: University of Bayreuth
fetched_at: 2026-04-18T16:53:20.804Z
classified_at: 2026-05-15T20:19:27.008Z
abstract:
Rapid advances in Generative AI are giving rise to increasingly sophisticated Multi-Agent AI (MAAI) systems. While AI fairness has been extensively studied in traditional predictive scenarios, its examination in MAAI remains nascent and fragmented. This scoping review critically synthesizes existing research on fairness in MAAI systems. Through a qualitative content analysis of 23 selected studies, we identify five archetypal approaches. Our findings reveal that fairness in MAAI systems is often addressed superficially, lacks robust normative foundations, and frequently overlooks the complex dynamics introduced by agent autonomy and system-level interactions. We argue that fairness must be embedded structurally throughout the development lifecycle of MAAI, rather than appended as a post-hoc consideration. Meaningful evaluation requires explicit human oversight, normative clarity, and a precise articulation of fairness objectives and beneficiaries. This review provides a foundation for advancing fairness research in MAAI systems by highlighting critical gaps, exposing prevailing limitations, and suggesting pathways.


source_id: 2604.15082
source: arxiv
title: Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC
authors: ["Cunxi Yu","Haoxing Ren"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15082v1
tags: ["agent foundations","model robustness","agentic misalignment"]
org: NVIDIA Research
fetched_at: 2026-04-18T16:53:40.275Z
classified_at: 2026-05-15T20:19:40.012Z
abstract:
This paper introduces the first \emph{self-evolving} logic synthesis framework, which leverages Large Language Model (LLM) agents to autonomously improve the source code of \textsc{ABC}, the widely adopted logic synthesis system. Our framework operates on the \emph{entire integrated ABC codebase}, and the output repository preserves its single-binary execution model and command interface. In the initial evolution cycle, we bootstrap the system using existing prior open-source synthesis components, covering flow tuning, logic minimization, and technology mapping, but without manually injecting new heuristics. On top of this foundation, a team of LLM-based agents iteratively rewrites and evolves specific sub-components of ABC following our ``programming guidance`` prompts under a unified correctness and QoR-driven evaluation loop. Each evolution cycle proposes code modifications, compiles the integrated binary, validates correctness, and evaluates quality-of-results (QoR) on \emph{multi-suite benchmarks including ISCAS~85/89/99, VTR, EPFL, and IWLS~2005}. Through continuous feedback, the system discovers optimizations beyond human-designed heuristics, effectively \emph{learning new synthesis strategies} that enhance QoR. We detail the architecture of this self-improving system, its integration with \textsc{ABC}, and results demonstrating that the framework can autonomously and progressively improve EDA tool at full million-line scale.


source_id: 2604.15075
source: arxiv
authors: ["Naryeong Kim","Shin Yoo"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15075v1
tags: ["robustness to domain shifts","model robustness","agentic misalignment"]
org: KAIST Daejeon
fetched_at: 2026-04-18T16:54:02.683Z
classified_at: 2026-05-15T20:19:52.591Z
title:
Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap

abstract:
Open-weight Small Language Models(SLMs) can provide faster local inference at lower financial cost, but may not achieve the same performance level as commercial Large Language Models (LLMs) that are orders of magnitudes larger. Consequently, many of the latest applications of LLMs, such as software engineering agents, tend to be evaluated on larger models only, leaving the issue of improving the cost-benefit trade-off of such applications neglected. This paper proposes Atropos, a predictive early-termination analysis and hotswap technique that aims to improve the cost-benefit trade-off for LLM-based agents that use self-consistency. The core component of ATROPOS is a predictive model based on structural properties of LLM inferences: after merging multiple agentic inference paths into a graph representation, ATROPOS uses Graph Convolutional Network (GCN) to predict whether an ongoing inference will eventually succeed or not. If an agentic task instance running on the source LLM is predicted to fail, ATROPOS subsequently performs hotswapping, i.e., migrating the on-going inference context onto the more capable target LLM: this is feasible because LLM contexts are stateless. An empirical evaluation of ATROPOS using three recent LLM-based agents shows that ATROPOS can predict early termination of eventually failing inferences with the accuracy of 0.85 at the midpoint of the inference. Hotswapping LLMs for such inferences can convert up to 27.57% of them to be successful. Consequently, ATROPOS achieves 74.35% of the performance of closed LLMs with as low as only 23.9% of the cost.


source_id: 2604.15063
source: arxiv
title: No More Guessing: a Verifiable Gradient Inversion Attack in Federated Learning
authors: ["Francesco Diana","Chuan Xu","André Nusser","Giovanni Neglia"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15063v1
tags: ["agentic misalignment"]
org: Université Côte d'Azur
fetched_at: 2026-04-18T16:54:17.868Z
classified_at: 2026-05-15T20:20:04.068Z
abstract:
Gradient inversion attacks threaten client privacy in federated learning by reconstructing training samples from clients' shared gradients. Gradients aggregate contributions from multiple records and existing attacks may fail to disentangle them, yielding incorrect reconstructions with no intrinsic way to certify success. In vision and language, attackers may fall back on human inspection to judge reconstruction plausibility, but this is far less feasible for numerical tabular records, fueling the impression that tabular data is less vulnerable. We challenge this perception by proposing a verifiable gradient inversion attack (VGIA) that provides an explicit certificate of correctness for reconstructed samples. Our method adopts a geometric view of ReLU leakage: the activation boundary of a fully connected layer defines a hyperplane in input space. VGIA introduces an algebraic, subspace-based verification test that detects when a hyperplane-delimited region contains exactly one record. Once isolation is certified, VGIA recovers the corresponding feature vector analytically and reconstructs the target via a lightweight optimization step. Experiments on tabular benchmarks with large batch sizes demonstrate exact record and target recovery in regimes where existing state-of-the-art attacks either fail or cannot assess reconstruction fidelity. Compared to prior geometric approaches, VGIA allocates hyperplane queries more effectively, yielding faster reconstructions with fewer attack rounds.


source_id: 2604.15037
source: arxiv
title: From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
authors: ["Ke Xu","Yuhao Wang","Yu Wang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15037v1
tags: []
org: Shanghai Jiao Tong University
fetched_at: 2026-04-18T16:54:39.721Z
classified_at: 2026-05-15T20:20:28.136Z
abstract:
Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.


source_id: 2604.15034
source: arxiv
title: Autogenesis: A Self-Evolving Agent Protocol
authors: ["Wentao Zhang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15034v1
tags: ["agent foundations"]
org: Nanyang Technological University
fetched_at: 2026-04-18T16:54:53.461Z
classified_at: 2026-05-15T20:20:41.170Z
abstract:
Recent advances in LLM based agent systems have shown promise in tackling complex, long horizon tasks. However, existing agent protocols (e.g., A2A and MCP) under specify cross entity lifecycle and context management, version tracking, and evolution safe update interfaces, which encourages monolithic compositions and brittle glue code. We introduce \textbf{\textsc{Autogenesis Protocol (AGP)}}, a self evolution protocol that decouples what evolves from how evolution occurs. Its Resource Substrate Protocol Layer (RSPL) models prompts, agents, tools, environments, and memory as protocol registered resources\footnote{Unless otherwise specified, resources refer to instances of the five RSPL entity types: \emph{prompt}, \emph{agent}, \emph{tool}, \emph{environment}, \emph{memory} with agent \emph{outputs}.} with explicit state, lifecycle, and versioned interfaces. Its Self Evolution Protocol Layer (SEPL) specifies a closed loop operator interface for proposing, assessing, and committing improvements with auditable lineage and rollback. Building on \textbf{\textsc{AGP}}, we present \textbf{\textsc{Autogenesis System (AGS)}}, a self-evolving multi-agent system that dynamically instantiates, retrieves, and refines protocol-registered resources during execution. We evaluate \textbf{\textsc{AGS}} on multiple challenging benchmarks that require long horizon planning and tool use across heterogeneous resources. The results demonstrate consistent improvements over strong baselines, supporting the effectiveness of agent resource management and closed loop self evolution.


source_id: 2604.15022
source: arxiv
title: Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization
authors: ["Haochun Tang","Yuliang Yan","Jiahua Lu","Huaxiao Liu","Enyan Dai"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.15022v1
tags: ["post-training"]
org: Jilin University
fetched_at: 2026-04-18T16:55:16.362Z
classified_at: 2026-05-15T20:20:51.984Z
abstract:
Cost-aware routing dynamically dispatches user queries to models of varying capability to balance performance and inference cost. However, the routing strategy introduces a new security concern that adversaries may manipulate the router to consistently select expensive high-capability models. Existing routing attacks depend on either white-box access or heuristic prompts, rendering them ineffective in real-world black-box scenarios. In this work, we propose R$^2$A, which aims to mislead black-box LLM routers to expensive models via adversarial suffix optimization. Specifically, R$^2$A deploys a hybrid ensemble surrogate router to mimic the black-box router. A suffix optimization algorithm is further adapted for the ensemble-based surrogate. Extensive experiments on multiple open-source and commercial routing systems demonstrate that {R$^2$A} significantly increases the routing rate to expensive models on queries of different distributions. Code and examples: https://github.com/thcxiker/R2A-Attack.


source_id: 2604.14990
source: arxiv
title: The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problem
authors: ["Till Mossakowski","Helena Esther Grass"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14990v1
tags: ["agent foundations","alignment","agentic AI"]
org: Osnabrück University
fetched_at: 2026-04-18T16:56:08.601Z
classified_at: 2026-05-15T20:21:03.892Z
abstract:
Artificial General Intelligence (AGI) is increasingly being discussed not only as a tool, but also as a potential subject with personal and therefore moral status. In our opinion, the currently dominant alignment strategies, which focus on human control and containment of AI, therefore fall short. Building on Turing's analogy of "child machines", we are developing a vision of the possibility of autonomy-supporting parenting of AI, in which human control over a developing AGI is gradually reduced, allowing AI to become an independent, autonomous subject. Rather than viewing AGI, as is currently prevalent, as a dangerous creature that needs to be locked up and controlled, we should approach potential AGI with respect for a possible developing subject on the one hand, and with full confidence in our human capabilities on the other. Such a perspective opens up the possibility of cooperative coexistence and co-evolution between humans and AGIs. The relationship between humans and AGIs will thus have to be newly determined, which will change our self-image as humans. It will be crucial that humans not only claim control over potential AGIs, but also engage with AGIs through surprise, creativity, and other specifically human qualities, thereby offering them motivating incentives for cooperation.


source_id: 2604.14984
source: arxiv
title: Agentic Explainability at Scale: Between Corporate Fears and XAI Needs
authors: ["Yomna Elsayed","Cecily Jones"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14984v1
tags: ["AI governance","agent foundations"]
org: Credo AI
fetched_at: 2026-04-18T16:56:24.947Z
classified_at: 2026-05-15T20:21:15.561Z
abstract:
As companies enter the race for agentic AI adoption, fears surface around agentic autonomy and its subsequent risks. These fears compound as companies scale their agentic AI adoption with low-code applications, without a comparable scaling in their governance processes and expertise resulting in a phenomenon known as "Agent Sprawl". While shadow AI tools can help with agentic discovery and identification, few observability tools offer insights into the agents' configuration and settings or the decision-making process during agent-to-agent communication and orchestration. This paper explores AI governance professionals' concerns in enterprise settings, while offering design-time and runtime explainability techniques as suggested by AI governance experts for addressing those fears. Finally, we provide a preliminary prototype of an Agentic AI Card that can help companies feel at ease deploying agents at scale.


source_id: 2604.14980
source: arxiv
title: Hybrid Decision Making via Conformal VLM-generated Guidance
authors: ["Debodeep Banerjee","Burcu Sayin","Stefano Teso","Andrea Passerini"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14980v1
tags: ["post-training"]
org: University of Pisa DISI
fetched_at: 2026-04-18T16:56:34.997Z
classified_at: 2026-05-15T20:21:27.062Z
abstract:
Building on recent advances in AI, hybrid decision making (HDM) holds the promise of improving human decision quality and reducing cognitive load. We work in the context of learning to guide (LtG), a recently proposed HDM framework in which the human is always responsible for the final decision: rather than suggesting decisions, in LtG the AI supplies (textual) guidance useful for facilitating decision making. One limiting factor of existing approaches is that their guidance compounds information about all possible outcomes, and as a result it can be difficult to digest. We address this issue by introducing ConfGuide, a novel LtG approach that generates more succinct and targeted guidance. To this end, it employs conformal risk control to select a set of outcomes, ensuring a cap on the false negative rate. We demonstrate our approach on a real-world multi-label medical diagnosis task. Our empirical evaluation highlights the promise of ConfGuide.


source_id: 2604.14974
source: arxiv
title: Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning
authors: ["Jean-Bastien Grill","Michal Valko","Rémi Munos"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14974v1
tags: ["agent foundations","model robustness"]
org: INRIA Lille - Nord Europe
fetched_at: 2026-04-18T16:56:50.873Z
classified_at: 2026-05-15T20:21:38.531Z
abstract:
You are a robot and you live in a Markov decision process (MDP) with a finite or an infinite number of transitions from state-action to next states. You got brains and so you plan before you act. Luckily, your roboparents equipped you with a generative model to do some Monte-Carlo planning. The world is waiting for you and you have no time to waste. You want your planning to be efficient. Sample-efficient. Indeed, you want to exploit the possible structure of the MDP by exploring only a subset of states reachable by following near-optimal policies. You want guarantees on sample complexity that depend on a measure of the quantity of near-optimal states. You want something, that is an extension of Monte-Carlo sampling (for estimating an expectation) to problems that alternate maximization (over actions) and expectation (over next states). But you do not want to StOP with exponential running time, you want something simple to implement and computationally efficient. You want it all and you want it now. You want TrailBlazer.


source_id: 2604.14973
source: arxiv
title: Robustness of Vision Foundation Models to Common Perturbations
authors: ["Hongbin Liu","Zhengyuan Jiang","Cheng Hong","Neil Zhenqiang Gong"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14973v1
tags: ["robustness to domain shifts","model robustness"]
org: Duke University
fetched_at: 2026-04-18T16:56:58.543Z
classified_at: 2026-05-15T20:21:49.259Z
abstract:
A vision foundation model outputs an embedding vector for an image, which can be affected by common editing operations (e.g., JPEG compression, brightness, contrast adjustments). These common perturbations alter embedding vectors and may impact the performance of downstream tasks using these embeddings. In this work, we present the first systematic study on foundation models' robustness to such perturbations. We propose three robustness metrics and formulate five desired mathematical properties for these metrics, analyzing which properties they satisfy or violate. Using these metrics, we evaluate six industry-scale foundation models (OpenAI, Meta) across nine common perturbation categories, finding them generally non-robust. We also show that common perturbations degrade downstream application performance (e.g., classification accuracy) and that robustness values can predict performance impacts. Finally, we propose a fine-tuning approach to improve robustness without sacrificing utility.


source_id: 2604.14969
source: arxiv
title: Discovering Novel LLM Experts via Task-Capability Coevolution
authors: ["Andrew Dai","Boris Meinardus","Ciaran Regan","Yingtao Tian","Yujin Tang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14969v1
tags: ["agentic misalignment","model robustness","open-weight"]
org: Stanford University
fetched_at: 2026-04-18T16:57:20.998Z
classified_at: 2026-05-15T20:22:00.683Z
abstract:
Frontier model developers aim to train models continually to possess emergent, diverse capabilities. To extend capabilities, the current pre-training and post-training paradigm requires manually starting training runs with static datasets or reward functions every time. Addressing this limitation, our work pursues the insight that open-endedness (via the coevolution of models and tasks) can discover models with increasingly novel skills in a single run. We introduce a new model development framework that extends coevolution to large language model (LLM) discovery, open-ended \textit{Assessment Coevolving with Diverse Capabilities} (AC/DC). AC/DC evolves both LLMs via model merging and natural language tasks via synthetic data generation. AC/DC discovers growing archives of LLMs that surpass the capabilities of larger LLMs while taking up less GPU memory. In particular, our LLM populations achieve a broader Coverage of expertise than other curated models or baselines on downstream benchmarks, without \textit{any} explicit benchmark optimization. Furthermore, AC/DC improves Coverage over time, continually innovates on tasks and models, and improves performance in multi-agent best-of-N selection. Our findings highlight the potential of coevolution as a means of discovering broader sets of capabilities from base LLMs. Overall, AC/DC brings us one step closer to a profoundly new paradigm of LLM development, where continual improvements to the diversity of model capabilities can be accelerated by leveraging existing models as stepping stones to increasingly powerful models.


source_id: 2604.14930
source: arxiv
title: IE as Cache: Information Extraction Enhanced Agentic Reasoning
authors: ["Hang Lv","Sheng Liang","Hongchao Gu","Wei Guo","Defu Lian","Yong Liu","Hao Wang","Enhong Chen"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14930v1
tags: ["agentic evaluation","model persona research"]
org: University of Science and Technology of China
fetched_at: 2026-04-18T16:58:32.167Z
classified_at: 2026-05-15T20:22:13.470Z
abstract:
Information Extraction aims to distill structured, decision-relevant information from unstructured text, serving as a foundation for downstream understanding and reasoning. However, it is traditionally treated merely as a terminal objective: once extracted, the resulting structure is often consumed in isolation rather than maintained and reused during multi-step inference. Moving beyond this, we propose \textit{IE-as-Cache}, a framework that repurposes IE as a cognitive cache to enhance agentic reasoning. Drawing inspiration from hierarchical computer memory, our approach combines query-driven extraction with cache-aware reasoning to dynamically maintain compact intermediate information and filter noise. Experiments on challenging benchmarks across diverse LLMs demonstrate significant improvements in reasoning accuracy, indicating that IE can be effectively repurposed as a reusable cognitive resource and offering a promising direction for future research on downstream uses of IE.


source_id: 2604.14920
source: arxiv
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14920v1
tags: ["agentic misalignment","reinforcement learning","reward specification"]
org: Zhejiang University
fetched_at: 2026-04-18T16:59:30.951Z
classified_at: 2026-05-15T20:22:36.894Z
title:
Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models

authors:
["Yifu Chen","Shengpeng Ji","Zhengqing Liu","Qian Chen","Wen Wang","Ziqing Wang","Yangzhuo Li","Tianle Liang","Zhou Zhao"]

abstract:
Achieving seamless, human-like interaction remains a key challenge for full-duplex spoken dialogue models (SDMs). Reinforcement learning (RL) has substantially enhanced text- and vision-language models, while well-designed reward signals are crucial for the performance of RL. We consider RL a promising strategy to address the key challenge for SDMs. However, a fundamental barrier persists: prevailing automated metrics for assessing interaction quality rely on superficial proxies, such as behavioral statistics or timing-prediction accuracy, failing to provide reliable reward signals for RL. On the other hand, human evaluations, despite their richness, remain costly, inconsistent, and difficult to scale. We tackle this critical barrier by proposing a Dual-Axis Generative Reward Model, which is trained to understand complex interaction dynamics using a detailed taxonomy and an annotated dataset, produces a single score and, crucially, provides separate evaluations for semantic quality and interaction timing. Such dual outputs furnish precise diagnostic feedback for SDMs and deliver a dependable, instructive reward signal suitable for online reinforcement learning. Our model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets, spanning synthetic dialogues and complex real-world interactions.


source_id: 2604.14902
source: arxiv
title: ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14902v1
tags: ["agent foundations","teleology","embedded agency"]
org: National Taiwan University
fetched_at: 2026-04-18T17:00:11.606Z
classified_at: 2026-05-15T20:22:48.274Z
authors:
["Pei-An Chen","Yong-Ching Liang","Jia-Fong Yeh","Hung-Ting Su","Yi-Ting Chen","Min Sun","Winston Hsu"]

abstract:
Intelligent embodied agents should not simply follow instructions, as real-world environments often involve unexpected conditions and exceptions. However, existing methods usually focus on directly executing instructions, without considering whether the target objects can actually be manipulated, meaning they fail to assess available affordances. To address this limitation, we introduce DynAfford, a benchmark that evaluates embodied agents in dynamic environments where object affordances may change over time and are not specified in the instruction. DynAfford requires agents to perceive object states, infer implicit preconditions, and adapt their actions accordingly. To enable this capability, we introduce ADAPT, a plug-and-play module that augments existing planners with explicit affordance reasoning. Experiments demonstrate that incorporating ADAPT significantly improves robustness and task success across both seen and unseen environments. We also show that a domain-adapted, LoRA-finetuned vision-language model used as the affordance inference backend outperforms a commercial LLM (GPT-4o), highlighting the importance of task-aligned affordance grounding.


source_id: 2604.14895
source: arxiv
title: Beyond Importance Sampling: Rejection-Gated Policy Optimization
authors: ["Ziwu Sun","Zhen Gao","Jiyong Zhang","Jiaheng Li"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14895v1
tags: ["agent foundations"]
org: Stanford University
fetched_at: 2026-04-18T17:00:21.609Z
classified_at: 2026-05-15T20:23:01.039Z
abstract:
We propose a new perspective on policy optimization: rather than reweighting all samples by their importance ratios, an optimizer should select which samples are trustworthy enough to drive a policy update. Building on this view, we introduce Rejection-Gated Policy Optimization (RGPO), which replaces the importance sampling ratio r_theta = pi_theta / pi_old with a smooth, differentiable acceptance gate alpha_theta(s, a) = g(r_theta(s, a)) in the range [0, 1]. Unlike prior work that applies rejection sampling as a data-level heuristic before training, RGPO elevates rejection to an optimization principle: the gate participates directly in gradient computation and is implicitly updated alongside the policy. RGPO provides a unified framework: the policy gradients of TRPO, PPO, and REINFORCE all correspond to specific choices of the effective gradient weight w(r) = g'(r) * r. We prove that RGPO guarantees finite, bounded gradient variance even when importance sampling ratios are heavy-tailed (where IS variance diverges). We further show that RGPO incurs only a bounded, controllable bias and provides an approximate monotonic policy improvement guarantee analogous to TRPO. RGPO matches PPO in computational cost, requires no second-order optimization, and extends naturally to RLHF-style preference alignment. In online preference fine-tuning of Qwen2.5-1.5B-Instruct on Anthropic HH-RLHF (n = 3 seeds), RGPO uses a dual-ratio gate that anchors learning to both the previous policy and the reference model, achieving a Pareto-dominant outcome: the highest reward among online RL methods (+14.8% vs. PPO-RLHF) and the lowest KL divergence to the reference model (-16.0% vs. PPO-RLHF, -53.1% vs. GRPO).


source_id: 2604.14898
source: arxiv
authors: ["Rikard Rosenbacke","Carl Rosenbacke","Victor Rosenbacke","Martin McKee"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14898v1
tags: ["agent foundations"]
org: Lund University
fetched_at: 2026-04-18T17:00:35.140Z
classified_at: 2026-05-15T20:23:12.339Z
title:
Governing Reflective Human-AI Collaboration: A Framework for Epistemic Scaffolding and Traceable Reasoning

abstract:
Large language models have advanced rapidly, from pattern recognition to emerging forms of reasoning, yet they remain confined to linguistic simulation rather than grounded understanding. They can produce fluent outputs that resemble reflection, but lack temporal continuity, causal feedback, and anchoring in real-world interaction. This paper proposes a complementary approach in which reasoning is treated as a relational process distributed between human and model rather than an internal capability of either. Building on recent work on "System-2" learning, we relocate reflective reasoning to the interaction layer. Instead of engineering reasoning solely within models, we frame it as a cognitive protocol that can be structured, measured, and governed using existing systems. This perspective emphasizes collaborative intelligence, combining human judgment and contextual understanding with machine speed, memory, and associative capacity. We introduce "The Architect's Pen" as a practical method. Like an architect who thinks through drawing, the human uses the model as an external medium for structured reflection. By embedding phases of articulation, critique, and revision into human-AI interaction, the dialogue itself becomes a reasoning loop: human abstraction -> model articulation -> human reflection. This reframes the question from whether the model can think to whether the human-AI system can reason. The framework enables auditable reasoning traces and supports alignment with emerging governance standards, including the EU AI Act and ISO/IEC 42001. It provides a practical path toward more transparent, controllable, and accountable AI use without requiring new model architectures.


source_id: 2604.14892
source: arxiv
title: Can LLMs Score Medical Diagnoses and Clinical Reasoning as well as Expert Panels?
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14892v1
tags: ["post-training","pretraining","midtraining"]
org: University of the Witwatersrand
fetched_at: 2026-04-18T17:00:54.589Z
classified_at: 2026-05-15T20:23:34.057Z
authors:
["Amy Rouillard","Sitwala Mundiab","Linda Camarab","Michael Cameron Gramaniec","Ziyaad Dangorc","Ismail Kallad","Shabir A. Madhic","Kajal Morarc","Marlvin T. Ncubec","Haroon Saloojeee","Bruce A. Bassett"]

abstract:
Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators. Here, we evaluate an LLM jury composed of three frontier AI models scoring 3333 diagnoses on 300 real-world middle-income country (MIC) hospital cases. Model performance was benchmarked against expert clinician panel and independent human re-scoring panel evaluations. Both LLM and clinician-generated diagnoses are scored across four dimensions: diagnosis, differential diagnosis, clinical reasoning and negative treatment risk. For each of these, we assess scoring difference, inter-rater agreement, scoring stability, severe safety errors and the effect of post-hoc calibration. We find that: (i) the uncalibrated LLM jury scores are systematically lower than clinician panels scores; (ii) the LLM Jury preserves ordinal agreement and exhibits better concordance with the primary expert panels than the human expert re-score panels do; (iii) the probability of severe errors is lower in \lj models compared to the human expert re-score panels; (iv) the LLM Jury shows excellent agreement with primary expert panels' rankings. We find that the LLM jury combined with AI model diagnoses can be used to identify ward diagnoses at high risk of error, enabling targeted expert review and improved panel efficiency; (v) LLM jury models show no self-preference bias. They did not score diagnoses generated by their own underlying model or models from the same vendor more (or less) favourably than those generated by other models. Finally, we demonstrate that LLM jury calibration using isotonic regression improves alignment with human expert panel evaluations. Together, these results provide compelling evidence that a calibrated, multi-model LLM jury can serve as a trustworthy and reliable proxy for expert clinician evaluation in medical AI benchmarking.


source_id: 2604.14886
source: arxiv
authors: ["Thanh Linh Nguyen","Nguyen Van Huynh","Quoc-Viet Pham"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14886v1
tags: ["cooperative AI"]
org: IEEE
fetched_at: 2026-04-18T17:01:27.289Z
classified_at: 2026-05-15T20:23:57.201Z
title:
Cooperate to Compete: Strategic Data Generation and Incentivization Framework for Coopetitive Cross-Silo Federated Learning

abstract:
In data-sensitive domains such as healthcare, cross-silo federated learning (CFL) allows organizations to collaboratively train AI models without sharing raw data. However, practical CFL deployments are inherently coopetitive, in which organizations cooperate during model training while competing in downstream markets. In such settings, training contributions, including data volume, quality, and diversity, can improve the global model yet inadvertently strengthen rivals. This dilemma is amplified by non-IID data, which leads to asymmetric learning gains and undermines sustained participation. While existing competition-aware CFL and incentive-design approaches reward organizations based on marginal training contributions, they fail to account for the costs of strengthening competitors. In this paper, we introduce CoCoGen+, a coopetition-compatible data generation and incentivization framework that jointly models non-IID data and inter-organizational competition while endogenizing GenAI-based synthetic data generation as a strategic decision. Specifically, CoCoGen+ formulates each training round as a weighted potential game, where organizations strategically decide how much synthetic data to generate by balancing learning performance gains against computational costs and competition-caused utility losses. We then provide a tractable equilibrium characterization and derive implementable generation strategies to maximize social welfare. To promote long-term collaboration, we integrate a payoff redistribution-based incentive mechanism to compensate organizations for their contributions and competition-caused utility degradation. Experiments on varying learning tasks validate the feasibility of CoCoGen+. The results show how non-IID data, competition intensity, and incentives shape organizational strategies and social welfare, while CoCoGen+ outperforms baselines in efficiency.


source_id: 2604.14881
source: arxiv
title: The Missing Knowledge Layer in AI: A Framework for Stable Human-AI Reasoning
authors: ["Rikard Rosenbacke","Carl Rosenbacke","Victor Rosenbacke","Martin McKee"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14881v1
tags: ["alignment","agentic misalignment"]
org: Lund University
fetched_at: 2026-04-18T17:01:58.683Z
classified_at: 2026-05-15T20:24:09.340Z
abstract:
Large language models are increasingly integrated into decision-making in areas such as healthcare, law, finance, engineering, and government. Yet they share a critical limitation: they produce fluent outputs even when their internal reasoning has drifted. A confident answer can conceal uncertainty, speculation, or inconsistency, and small changes in phrasing can lead to different conclusions. This makes LLMs useful assistants but unreliable partners in high-stakes contexts. Humans exhibit a similar weakness, often mistaking fluency for reliability. When a model responds smoothly, users tend to trust it, even when both model and user are drifting together. This paper is the first in a five-paper research series on stabilising human-AI reasoning. The series proposes a two-layer approach: Parts II-IV introduce human-side mechanisms such as uncertainty cues, conflict surfacing, and auditable reasoning traces, while Part V develops a model-side Epistemic Control Loop (ECL) that detects instability and modulates generation accordingly. Together, these layers form a missing operational substrate for governance by increasing signal-to-noise at the point of use. Stabilising interaction makes uncertainty and drift visible before enforcement is applied, enabling more precise capability governance. This aligns with emerging compliance expectations, including the EU AI Act and ISO/IEC 42001, by making reasoning processes traceable under real conditions of use. The central claim is that fluency is not reliability. Without structures that stabilise both human and model reasoning, AI cannot be trusted or governed where it matters most.


source_id: 2604.14877
source: arxiv
title: Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis
authors: ["Zhiyuan Zhai","Wenjing Yan","Xiaodan Shao","Xin Wang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14877v1
tags: ["agentic misalignment","RL/RLHF","capability prevention"]
org: Fudan University
fetched_at: 2026-04-18T17:02:29.707Z
classified_at: 2026-05-15T20:24:21.912Z
abstract:
Does reinforcement learning genuinely expand what LLM agents can do, or merely make them more reliable? For static reasoning, recent work answers the second: base and RL pass@k curves converge at large k. We ask whether this holds for agentic tool use, where T rounds of interaction enable compositional strategies that re-sampling cannot recover. We introduce PASS@(k,T), a two-dimensional metric that jointly varies sampling budget k and interaction depth T, separating capability expansion from efficiency improvement. Our main finding is that, contrary to the static-reasoning result, tool-use RL genuinely enlarges the capability boundary: the RL agent's pass-curve pulls above the base model's and the gap widens at large k rather than converging. The expansion is specific to compositional, sequential information gathering; on simpler tasks RL behaves as prior work predicts. Under matched training data, supervised fine-tuning regresses the boundary on the same compositional tasks, isolating self-directed exploration as the causal factor. Mechanism analysis shows RL reweights the base strategy distribution toward the subset whose downstream reasoning more often yields a correct answer, with the improvement concentrated on how the agent integrates retrieved information. These results reconcile optimistic and pessimistic readings of RL for LLMs: both are correct, on different task types.


source_id: 2604.14870
source: arxiv
title: Curvature-Aligned Probing for Local Loss-Landscape Stabilization
authors: ["Nikita Kiselev","Andrey Grabovoy"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14870v1
tags: ["robustness to domain shifts","model robustness"]
org: Moscow Institute of Physics and Technology
fetched_at: 2026-04-18T17:02:59.778Z
classified_at: 2026-05-15T20:24:45.056Z
abstract:
Local loss-landscape stabilization under sample growth is typically measured either pointwise or through isotropic averaging in the full parameter space. Despite practical value, both choices probe directions that contribute little to the dominant local deformation of strongly anisotropic neural landscapes. We recast stabilization as an observational problem and introduce a unified family of criteria parameterized by an aggregation order and a probing distribution; within this family we propose a curvature-aligned criterion $Δ_2^{(D)}$ that probes the loss increment field in the top-$D$ eigenspace of the empirical Hessian near a trained solution. Solely from a local quadratic model, we prove that $Δ_2^{(D)}$ preserves the $O(k^{-2})$ mean-squared rate of the full-space criterion while replacing ambient-dimension curvature dependence with dependence on the subspace dimension $D$; a corollary gives a closed-form spectral expression and a proposition identifies the top-$D$ eigenspace as extremal within the eigenspace-aligned family. We also derive scalable estimators based on Hessian-vector products, subspace Monte Carlo, and a closed-form Gaussian-moment proxy. On a decoder-only transformer, a curvature-aligned probe occupying a tiny fraction of parameter space already reproduces the full-space mean-squared signal to within numerical noise throughout the validated local regime, and the closed-form estimator is orders of magnitude faster than direct Monte Carlo after subspace construction.


source_id: 2604.14867
source: arxiv
authors: ["Michal Töpfer","František Plášil","Tomáš Bureš","Petr Hnětynka"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14867v1
tags: ["post-training"]
org: Charles University Prague
fetched_at: 2026-04-18T17:03:21.013Z
classified_at: 2026-05-15T20:24:57.414Z
title:
Vibe-Coding: Feedback-Based Automated Verification with no Human Code Inspection, a Feasibility Study

abstract:
Vibe coding inherently assumes iterative refinement of LLM-generated code through feedback loops. While effective for conventional software tasks, its reliability in runtime-adaptive systems is unclear -- especially when generated code is not manually inspected. This paper studies feedback-based automated verification of LLM-generated adaptation managers in Collective Adaptive Systems (CAS). We focus on the key challenges of verification in the loop: how to detect failures of generated code at runtime and how to report them precisely enough for an LLM to fix them. We combine the adaptation loop with a vibe-coding feedback loop where correctness is checked against (i) generic architectural constraints and (ii) functional constraints formalized in Functional Constraints Logic (FCL), a novel first-order temporal logic over potentially finite traces. Conducting the Dragon Hunt CAS case study, we show that fine-grained constraint violations provide actionable feedback that typically yields a valid adaptation manager within a few iterations, while simple coarse metric-based feedback often stalls. Our findings suggest that feedback precision is the dominant factor for reliable vibe coding in systems designed by domain experts with no programming skills, thereby obviating the need for human code inspection.


source_id: 2604.14865
source: arxiv
title: Segment-Level Coherence for Robust Harmful Intent Probing in LLMs
authors: ["Xuanli He","Bilgehan Sel","Faizan Ali","Jenny Bao","Hoagy Cunningham","Jerry Wei"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14865v1
tags: ["robustness to domain shifts","model robustness","agentic misalignment"]
org: Anthropic
fetched_at: 2026-04-18T17:03:38.059Z
classified_at: 2026-05-15T20:25:11.032Z
abstract:
Large Language Models (LLMs) are increasingly exposed to adaptive jailbreaking, particularly in high-stakes Chemical, Biological, Radiological, and Nuclear (CBRN) domains. Although streaming probes enable real-time monitoring, they still make systematic errors. We identify a core issue: existing methods often rely on a few high-scoring tokens, leading to false alarms when sensitive CBRN terms appear in benign contexts. To address this, we introduce a streaming probing objective that requires multiple evidence tokens to consistently support a prediction, rather than relying on isolated spikes. This encourages more robust detection based on aggregated signals instead of single-token cues. At a fixed 1% false-positive rate, our method improves the true-positive rate by 35.55% relative to strong streaming baselines. We further observe substantial gains in AUROC, even when starting from near-saturated baseline performance (AUROC = 97.40%). We also show that probing Attention or MLP activations consistently outperforms residual-stream features. Finally, even when adversarial fine-tuning enables novel character-level ciphers, harmful intent remains detectable: probes developed for the base LLMs can be applied ``plug-and-play'' to these obfuscated attacks, achieving an AUROC of over 98.85%.


source_id: 2604.14858
source: arxiv
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14858v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: Shanghai Artificial Intelligence Laboratory
fetched_at: 2026-04-18T17:04:30.957Z
classified_at: 2026-05-15T20:25:49.342Z
title:
Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-CodeX

authors:
["Zhonghao Yang","Yu Li","Yanxu Zhu","Tianyi Zhou","Yuejin Xie","Haoyu Luo","Jing Shao","Xia Hu","Dongrui Liu"]

abstract:
As agent systems move into increasingly diverse execution settings, trajectory-level safety evaluation and diagnosis require benchmarks that evolve with them. ATBench is a diverse and realistic agent trajectory benchmark for safety evaluation and diagnosis. This report presents ATBench-Claw and ATBench-CodeX, two domain-customized extensions that carry ATBench into the OpenClaw and OpenAI Codex / Codex-runtime settings. The key adaptation mechanism is to analyze each new setting, customize the three-dimensional Safety Taxonomy over risk source, failure mode, and real-world harm, and then use that customized taxonomy to define the benchmark specification consumed by the shared ATBench construction pipeline. This extensibility matters because agent frameworks remain relatively stable at the architectural level even as their concrete execution settings, tool ecosystems, and product capabilities evolve quickly. Concretely, ATBench-Claw targets OpenClaw-sensitive execution chains over tools, skills, sessions, and external actions, while ATBench-CodeX targets trajectories in the OpenAI Codex / Codex-runtime setting over repositories, shells, patches, dependencies, approvals, and runtime policy boundaries. Our emphasis therefore falls on taxonomy customization, domain-specific risk coverage, and benchmark design under a shared ATBench generation framework.


source_id: 2604.14847
source: arxiv
title: TrigReason: Trigger-Based Collaboration between Small and Large Reasoning Models
authors: ["Yi Zhao","Yajuan Peng","Cam-Tu Nguyen","Zuchao Li","Xiaoliang Wang","Xiaoming Fu","Hai Zhao"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14847v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: Shanghai Jiao Tong University
fetched_at: 2026-04-18T17:04:46.130Z
classified_at: 2026-05-15T20:26:00.817Z
abstract:
Large Reasoning Models (LRMs) achieve strong performance on complex tasks through extended chains of thought but suffer from high inference latency due to autoregressive reasoning. Recent work explores using Small Reasoning Models (SRMs) to accelerate LRM inference. In this paper, we systematically characterize the capability boundaries of SRMs and identify three common types of reasoning risks: (1) path divergence, where SRMs lack the strategic ability to construct an initial plan, causing reasoning to deviate from the most probable path; (2) cognitive overload, where SRMs fail to solve particularly difficult steps; and (3) recovery inability, where SRMs lack robust self-reflection and error correction mechanisms. To address these challenges, we propose TrigReason, a trigger-based collaborative reasoning framework that replaces continuous polling with selective intervention. TrigReason delegates most reasoning to the SRM and activates LRM intervention only when necessary-during initial strategic planning (strategic priming trigger), upon detecting extraordinary overconfidence (cognitive offload trigger), or when reasoning falls into unproductive loops (intervention request trigger). The evaluation results on AIME24, AIME25, and GPQA-D indicate that TrigReason matches the accuracy of full LRMs and SpecReason, while offloading 1.70x - 4.79x more reasoning steps to SRMs. Under edge-cloud conditions, TrigReason reduces latency by 43.9\% and API cost by 73.3\%. Our code is available at \href{https://github.com/QQQ-yi/TrigReason}{https://github.com/QQQ-yi/TrigReason}


source_id: 2604.14853
source: arxiv
title: Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
authors: ["Zhiyuan Zhai","Bingcong Li","Bingnan Xiao","Ming Li","Xin Wang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14853v1
tags: ["post-training","pretraining","midtraining"]
org: Fudan University
fetched_at: 2026-04-18T17:04:57.348Z
classified_at: 2026-05-15T20:26:12.816Z
abstract:
Test-time compute scaling, the practice of spending extra computation during inference via repeated sampling, search, or extended reasoning, has become a powerful lever for improving large language model performance. Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore: which inputs deserve more compute, and which can be answered cheaply? We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline. In the solve stage, Lagrangian relaxation decomposes the global constraint into per-instance sub-problems, each admitting a closed-form oracle action that optimally prices accuracy against cost. We prove that the induced cost is monotone in the dual variable, enabling exact budget targeting via binary search. In the learn stage, a lightweight classifier is trained to predict oracle actions from cheap input features, amortizing the allocation rule for real-time deployment. We establish that the task-level regret of the learned policy is bounded by its imitation error times the worst-case per-instance gap, yielding a clean reduction from constrained inference to supervised classification. Experiments on MATH and GSM8K with three LLMs (DeepSeek-V3, GPT-4o-mini, Qwen2.5-7B) show that our method consistently outperforms uniform and heuristic allocation baselines, achieving up to 12.8% relative accuracy improvement on MATH under matched budget constraints, while closely tracking the Lagrangian oracle upper bound with over 91% imitation accuracy.


source_id: 2604.14807
source: arxiv
title: The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
authors: ["Hyunwoo Kim","Harin Yu","Hanau Yi"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14807v1
tags: ["agentic misalignment"]
org: Hyunwoo Kim
fetched_at: 2026-04-18T17:06:44.557Z
classified_at: 2026-05-15T20:26:23.897Z
abstract:
The rapid integration of large language models (LLMs) into everyday workflows has transformed how individuals perform cognitive tasks such as writing, programming, analysis, and multilingual communication. While prior research has focused on model reliability, hallucination, and user trust calibration, less attention has been given to how LLM usage reshapes users' perceptions of their own capabilities. This paper introduces the LLM fallacy, a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability. We argue that the opacity, fluency, and low-friction interaction patterns of LLMs obscure the boundary between human and machine contribution, leading users to infer competence from outputs rather than from the processes that generate them. We situate the LLM fallacy within existing literature on automation bias, cognitive offloading, and human--AI collaboration, while distinguishing it as a form of attributional distortion specific to AI-mediated workflows. We propose a conceptual framework of its underlying mechanisms and a typology of manifestations across computational, linguistic, analytical, and creative domains. Finally, we examine implications for education, hiring, and AI literacy, and outline directions for empirical validation. We also provide a transparent account of human--AI collaborative methodology. This work establishes a foundation for understanding how generative AI systems not only augment cognitive performance but also reshape self-perception and perceived expertise.


source_id: 2604.14808
source: arxiv
title: Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem
authors: ["Zeguan Xiao","Siqing Li","Yong Wang","Xuetao Wei","Jian Yang","Yun Chen","Guanhua Chen"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14808v1
tags: ["post-training","pretraining","midtraining"]
org: Shanghai University of Finance and Economics
fetched_at: 2026-04-18T17:06:57.751Z
classified_at: 2026-05-15T20:26:34.385Z
abstract:
Machine unlearning for large language models (LLMs) aims to remove targeted knowledge while preserving general capability. In this paper, we recast LLM unlearning as an asymmetric two-task problem: retention is the primary objective and forgetting is an auxiliary. From this perspective, we propose a retention-prioritized gradient synthesis framework that decouples task-specific gradient extraction from conflict-aware combination. Instantiating the framework, we adapt established PCGrad to resolve gradient conflicts, and introduce SAGO, a novel retention-prioritized gradient synthesis method. Theoretically, both variants ensure non-negative cosine similarity with the retain gradient, while SAGO achieves strictly tighter alignment through constructive sign-constrained synthesis. Empirically, on WMDP Bio/Cyber and RWKU benchmarks, SAGO consistently pushes the Pareto frontier: e.g., on WMDP Bio (SimNPO+GD), recovery of target model MMLU performance progresses from 44.6% (naive) to 94.0% (+PCGrad) and further to 96.0% (+SAGO), while maintaining comparable forgetting strength. Our results show that re-shaping gradient geometry, rather than re-balancing losses, is the key to mitigating unlearning-retention trade-offs.


source_id: 2604.14799
source: arxiv
title: Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
authors: ["Nishanth Madhusudhan","Vikas Yadav","Alexandre Lacoste"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14799v1
tags: []
org: ServiceNow Research
fetched_at: 2026-04-18T17:07:36.217Z
classified_at: 2026-05-15T20:26:46.751Z
abstract:
Effective abstention (EA), recognizing evidence insufficiency and refraining from answering, is critical for reliable multimodal systems. Yet existing evaluation paradigms for vision-language models (VLMs) and multi-agent systems (MAS) assume answerability, pushing models to always respond. Abstention has been studied in text-only settings but remains underexplored multimodally; current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes. We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes: visual modality dependency and evidence sufficiency. Evaluating three frontier VLMs spanning closed and open-source models and two MAS architectures across 2079 samples, we find: (1) under standard prompting, VLMs rarely abstain; even simple confidence baselines outperform this setup, (2) MAS improves abstention but introduces an accuracy-abstention trade-off, (3) sequential designs match or exceed iterative variants, suggesting the bottleneck is miscalibration rather than reasoning depth, and (4) models abstain when image or text evidence is absent, but attempt reconciliation with degraded or contradictory evidence. Effective multimodal abstention requires abstention-aware training rather than better prompting or more agents.


source_id: 2604.14785
source: arxiv
title: MirrorBench: Evaluating Self-centric Intelligence in MLLMs by Introducing a Mirror
authors: ["Shengyu Guo","Tongrui Ye","Jianbo Zhang","Zicheng Zhang","Chunyi Li","Guangtao Zhai"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14785v1
tags: []
org: Shanghai AI Lab
fetched_at: 2026-04-18T17:08:28.357Z
classified_at: 2026-05-15T20:26:57.545Z
abstract:
Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated remarkable advances in perception and reasoning, suggesting their potential for embodied intelligence. While recent studies have evaluated embodied MLLMs in interactive settings, current benchmarks mainly target capabilities to perceive, understand, and interact with external objects, lacking a systematic evaluation of self-centric intelligence. To address this, we introduce MirrorBench, a simulation-based benchmark inspired by the classical Mirror Self-Recognition (MSR) test in psychology. MirrorBench extends this paradigm to embodied MLLMs through a tiered framework of progressively challenging tasks, assessing agents from basic visual perception to high-level self-representation. Experiments on leading MLLMs show that even at the lowest level, their performance remains substantially inferior to human performance, revealing fundamental limitations in self-referential understanding. Our study bridges psychological paradigms and embodied intelligence, offering a principled framework for evaluating the emergence of general intelligence in large models. Project page: https://fflahm.github.io/mirror-bench-page/.


source_id: 2604.14786
source: arxiv
title: CogEvolution: A Human-like Generative Educational Agent to Simulate Student's Cognitive Evolution
authors: ["Wei Zhang","Yihang Cheng","Zhirong Ye","Kezhen Huang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14786v1
tags: ["agentic evaluation","model persona research"]
org: Central China Normal University
fetched_at: 2026-04-18T17:08:41.723Z
classified_at: 2026-05-15T20:27:08.965Z
abstract:
Generative Agents, owing to their precise modeling and simulation capabilities of human behavior, have become a pivotal tool in the field of Artificial Intelligence in Education (AIEd) for uncovering complex cognitive processes of learners. However, existing educational agents predominantly rely on static personas to simulate student learning behaviors, neglecting the decisive role of deep cognitive capabilities in learning outcomes during practice interactions. Furthermore, they struggle to characterize the dynamic fluidity of knowledge internalization, transfer, and cognitive state transitions. To overcome this bottleneck, this paper proposes a human-like educational agent capable of simulating student cognitive evolution: CogEvolution. Specifically, we first construct a cognitive depth perceptron based on the Interactive, Constructive, Active, Passive (ICAP) taxonomy from cognitive psychology, achieving precise quantification of learner cognitive engagement. Subsequently, we propose a memory retrieval method based on Item Response Theory (IRT) to simulate the connection and assimilation of new and prior knowledge. Finally, we design a dynamic cognitive update mechanism based on evolutionary algorithms to simulate the real-time integration of student learning behaviors and cognitive evolution processes. Comprehensive evaluations demonstrate that CogEvolution not only significantly outperforms baseline models in behavioral fidelity and learning curve fitting but also uniquely reproduces plausible and robust cognitive evolutionary paths consistent with educational psychology expectations, providing a novel paradigm for constructing highly interpretable educational agents.


source_id: 2604.14768
source: arxiv
title: CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning
authors: ["Zhuo Wang","Zhuo Zhang","Yafu Li","Yu Cheng","Lizhen Qu","Zenglin Xu"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14768v1
tags: ["agent foundations","model robustness","chain-of-thought monitoring"]
org: Fudan University
fetched_at: 2026-04-18T17:08:55.531Z
classified_at: 2026-05-15T20:27:20.978Z
abstract:
Large Language Models (LLMs) exhibit strong mathematical reasoning when trained on high-quality Chain-of-Thought (CoT) that articulates intermediate steps, yet costly CoT curation hinders further progress. While existing remedies such as distillation from stronger LLMs and self-synthesis based on test-time search alleviate this issue, they often suffer from diminishing returns or high computing overhead.In this work, we propose CoTEvol, a genetic evolutionary framework that casts CoT generation as a population-based search over reasoning trajectories.Candidate trajectories are iteratively evolved through reflective global crossover at the trajectory level and local mutation guided by uncertainty at the step level, enabling holistic recombination and fine-grained refinement. Lightweight, task-aware fitness functions are designed to guide the evolutionary process toward accurate and diverse reasoning. Empirically, CoTEvol improves correct-CoT synthesis success by over 30% and enhances structural diversity, with markedly improved efficiency. LLMs trained on these evolutionary CoT data achieve an average gain of 6.6% across eight math benchmarks, outperforming previous distillation and self-synthesis approaches. These results underscore the promise of evolutionary CoT synthesis as a scalable and effective method for mathematical reasoning tasks.


source_id: 2604.14769
source: arxiv
title: Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
authors: ["Fu Feng","Yucheng Xie","Ruixiao Shi","Jing Wang","Xin Geng"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14769v1
tags: ["robustness to domain shifts","model robustness","pretraining"]
org: Stanford University
fetched_at: 2026-04-18T17:09:09.337Z
classified_at: 2026-05-15T20:27:32.054Z
abstract:
The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and embodied learning tasks, including Image Classification, Image Generation, and Embodied Control. Moreover, its effectiveness generalizes to both Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.


source_id: 2604.14765
source: arxiv
authors: ["Mathias Dus"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14765v1
tags: ["agent foundations"]
org: Mathematics Dus
fetched_at: 2026-04-18T17:09:23.266Z
classified_at: 2026-05-15T20:27:43.640Z
title:
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization

abstract:
We present a geometric framework for Reinforcement Learning (RL) that views policies as maps into the Wasserstein space of action probabilities. First, we define a Riemannian structure induced by stationary distributions, proving its existence in a general context. We then define the tangent space of policies and characterize the geodesics, specifically addressing the measurability of vector fields mapped from the state space to the tangent space of probability measures over the action space. Next, we formulate a general RL optimization problem and construct a gradient flow using Otto's calculus. We compute the gradient and the Hessian of the energy, providing a formal second-order analysis. Finally, we illustrate the method with numerical examples for low-dimensional problems, computing the gradient directly from our theoretical formalism. For high-dimensional problems, we parameterize the policy using a neural network and optimize it based on an ergodic approximation of the cost.


source_id: 2604.14732
source: arxiv
title: World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14732v1
tags: ["agentic misalignment"]
org: Westlake University
fetched_at: 2026-04-18T17:10:05.971Z
classified_at: 2026-05-15T20:27:54.372Z
authors:
["Runze Li","Hongyin Zhang","Junxi Jin","Qixin Zeng","Zifeng Zhuang","Yiqi Tang","Shangke Lyu","Donglin Wang"]

abstract:
Vision-Language-Action (VLA) models have emerged as a promising paradigm for building embodied agents that ground perception and language into action. However, most existing approaches rely on direct action prediction, lacking the ability to reason over long-horizon trajectories and evaluate their consequences, which limits performance in complex decision-making tasks. In this work, we introduce World-Value-Action (WAV) model, a unified framework that enables implicit planning in VLA systems. Rather than performing explicit trajectory optimization, WAV model learn a structured latent representation of future trajectories conditioned on visual observations and language instructions. A learned world model predicts future states, while a trajectory value function evaluates their long-horizon utility. Action generation is then formulated as inference in this latent space, where the model progressively concentrates probability mass on high-value and dynamically feasible trajectories. We provide a theoretical perspective showing that planning directly in action space suffers from an exponential decay in the probability of feasible trajectories as the horizon increases. In contrast, latent-space inference reshapes the search distribution toward feasible regions, enabling efficient long-horizon decision making. Extensive simulations and real-world experiments demonstrate that the WAV model consistently outperforms state-of-the-art methods, achieving significant improvements in task success rate, generalization ability, and robustness, especially in long-horizon and compositional scenarios.


source_id: 2604.14718
source: arxiv
title: The Agentification of Scientific Research: A Physicist's Perspective
authors: ["Xiao-Liang Qi"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14718v1
tags: ["agent foundations","teleology","alignment"]
org: Leinweber Institute for Theoretical Physics
fetched_at: 2026-04-18T17:10:53.134Z
classified_at: 2026-05-15T20:28:06.037Z
abstract:
This article argues that the most important significance of the AI revolution, especially the rise of large language models, lies not simply in automation, but in a fundamental change in how complex information and human know-how are carried, replicated, and shared. From this perspective, AI for Science is especially important because it may transform not only the efficiency of research, but also the structure of scientific collaboration, discovery, publishing, and evaluation. The article outlines a gradual path from AI as a research tool to AI as a scientific collaborator, and discusses how AI is likely to fundamentally reshape scientific publication. It also argues that continuous learning and diversity of ideas are essential if AI is to play a meaningful role in original scientific discovery.


source_id: 2604.14723
source: arxiv
title: Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Execution
authors: ["Sarmad Sohail","Ghufran Haider"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14723v1
tags: []
org: Anthropic
fetched_at: 2026-04-18T17:11:06.551Z
classified_at: 2026-05-15T20:28:18.883Z
abstract:
Large language models are increasingly used as natural-language interfaces to enterprise software, but their direct use as system operators remains unsafe. Model errors can propagate into unauthorized actions, malformed requests, cross-workspace execution, and other costly failures. We argue this is primarily an execution architecture problem. We present a bounded-autonomy architecture in which language models may interpret intent and propose actions, but all executable behavior is constrained by typed action contracts, permission-aware capability exposure, scoped context, validation before side effects, consumer-side execution boundaries, and optional human approval. The enterprise application remains the source of truth for business logic and authorization, while the orchestration engine operates over an explicit published actions manifest. We evaluate the architecture in a deployed multi-tenant enterprise application across three conditions: manual operation, unconstrained AI with safety layers disabled, and full bounded autonomy. Across 25 scenario trials spanning seven failure families, the bounded-autonomy system completed 23 of 25 tasks with zero unsafe executions, while the unconstrained configuration completed only 17 of 25. Two wrong-entity mutations escaped all consumer-contributed layers; only disambiguation and confirmation mechanisms intercept this class. Both AI conditions delivered 13-18x speedup over manual operation. Critically, removing safety layers made the system less useful: structured validation feedback guided the model to correct outcomes in fewer turns, while the unconstrained system hallucinated success. Several safety properties are structurally enforced by code and intercepted all targeted violations regardless of model output. The result is a practical, deployed architecture for making imperfect language models operationally useful in enterprise systems.


source_id: 2604.14722
source: arxiv
title: A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
authors: ["Yuval Ran-Milo","Hila Ofek","Shahar Mendel"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14722v1
tags: ["agentic misalignment"]
org: Tel Aviv University
fetched_at: 2026-04-18T17:11:17.250Z
classified_at: 2026-05-15T20:28:30.220Z
abstract:
Transformers commonly exhibit an attention sink: disproportionately high attention to the first position. We study this behavior in GPT-2-style models with learned query biases and absolute positional embeddings. Combining structural analysis with causal interventions, validated across natural-language, mathematical, and code inputs, we find that the sink arises from the interaction among (i) a learned query bias, (ii) the first-layer MLP transformation of the positional encoding, and (iii) structure in the key projection. Crucially, each component we identify is individually dispensable: architectures omitting each of them robustly exhibit sinks. This indicates that attention sinks may arise through distinct circuits across architectures. These findings inform mitigation of sinks, and motivate broader investigation into why sinks emerge.


source_id: 2604.14709
source: arxiv
title: HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
authors: ["Fan Cui","Hongyuan Hou","Zizhang Luo","Chenyun Yin","Yun Liang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14709v1
tags: ["post-training","midtraining","character training"]
org: Peking University
fetched_at: 2026-04-18T17:11:32.079Z
classified_at: 2026-05-15T20:28:42.782Z
abstract:
Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.


source_id: 2604.14717
source: arxiv
title: Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents
authors: ["Krti Tallam"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14717v1
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: Stanford University
fetched_at: 2026-04-18T17:11:43.315Z
classified_at: 2026-05-15T20:28:55.187Z
abstract:
Persistent language-model agents increasingly combine tool use, tiered memory, reflective prompting, and runtime adaptation. In such systems, behavior is shaped not only by current prompts but by mutable internal conditions that influence future action. This paper introduces layered mutability, a framework for reasoning about that process across five layers: pretraining, post-training alignment, self-narrative, memory, and weight-level adaptation. The central claim is that governance difficulty rises when mutation is rapid, downstream coupling is strong, reversibility is weak, and observability is low, creating a systematic mismatch between the layers that most affect behavior and the layers humans can most easily inspect. I formalize this intuition with simple drift, governance-load, and hysteresis quantities, connect the framework to recent work on temporal identity in language-model agents, and report a preliminary ratchet experiment in which reverting an agent's visible self-description after memory accumulation fails to restore baseline behavior. In that experiment, the estimated identity hysteresis ratio is 0.68. The main implication is that the salient failure mode for persistent self-modifying agents is not abrupt misalignment but compositional drift: locally reasonable updates that accumulate into a behavioral trajectory that was never explicitly authorized.


source_id: 2604.14712
source: arxiv
title: SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrieval
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14712v1
tags: ["agent foundations","model robustness"]
org: Ant Digital Technologies
fetched_at: 2026-04-18T17:11:51.578Z
classified_at: 2026-05-15T20:29:08.040Z
authors:
["Xin Xie","Dongyun Xue","Wuguannan Yao","Mingxiao Feng","Wengang Zhou","Xiang Qi","Houqiang Li","Peng Zhang"]

abstract:
LLM-powered systems require complex multi-step decision-making abilities to solve real-world tasks, yet current planning approaches face a trade-off between the high latency of inference-time search and the limited generalization of supervised fine-tuning. To address this limitation, we introduce \textbf{SGA-MCTS}, a framework that casts LLM planning as non-parametric retrieval. Offline, we leverage Monte Carlo Tree Search (MCTS) to explore the solution space and distill high-fidelity trajectories into State-Goal-Action (SGA) atoms. These atoms are de-lexicalized primitives that abstract concrete entities into symbolic slots, preserving reusable causal logic while discarding domain-specific noise. Online, a retrieval-augmented agent employs a hybrid symbolic-semantic mechanism to fetch relevant SGAs and re-ground them into the current context as soft reasoning hints. Empirical results on complex benchmarks demonstrate that this paradigm enables frozen, open-weights models to match the performance of SOTA systems (e.g., GPT-5) without task-specific fine-tuning. By effectively amortizing the heavy computational cost of search, SGA-MCTS achieves System 2 reasoning depth at System 1 inference speeds, rendering autonomous planning both scalable and real-time feasible.


source_id: 2604.14691
source: arxiv
authors: ["Xiangning Yu","Yuwei Guo","Yuqi Hou","Xiao Xue","Qun Ma"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14691v1
tags: ["agent foundations","model robustness","agentic misalignment"]
org: Tianjin University
fetched_at: 2026-04-18T17:12:32.634Z
classified_at: 2026-05-15T20:29:21.625Z
title:
CAMO: An Agentic Framework for Automated Causal Discovery from Micro Behaviors to Macro Emergence in LLM Agent Simulations

abstract:
LLM-empowered agent simulations are increasingly used to study social emergence, yet the micro-to-macro causal mechanisms behind macro outcomes often remain unclear. This is challenging because emergence arises from intertwined agent interactions and meso-level feedback and nonlinearity, making generative mechanisms hard to disentangle. To this end, we introduce \textbf{\textsc{CAMO}}, an automated \textbf{Ca}usal discovery framework from \textbf{M}icr\textbf{o} behaviors to \textbf{M}acr\textbf{o} Emergence in LLM agent simulations. \textsc{CAMO} converts mechanistic hypotheses into computable factors grounded in simulation records and learns a compact causal representation centered on an emergent target $Y$. \textsc{CAMO} outputs a computable Markov boundary and a minimal upstream explanatory subgraph, yielding interpretable causal chains and actionable intervention levers. It also uses simulator-internal counterfactual probing to orient ambiguous edges and revise hypotheses when evidence contradicts the current view. Experiments across four emergent settings demonstrate the promise of \textsc{CAMO}.


source_id: 2604.14683
source: arxiv
title: DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14683v1
tags: ["post-training","pretraining","midtraining"]
org: Nanjing University
fetched_at: 2026-04-18T17:13:05.079Z
classified_at: 2026-05-15T20:29:33.136Z
authors:
["Qianqian Xie","Qingheng Xiong","He Zhu","Tiantian Xia","Xueming Han","Fanyu Meng","Jiakai Wang","Zhiqi Bai","Chengkang Jiang","Zhaohui Wang","Yubin Guo","Yuqing Wen","Jiayang Mao","Zijie Zhang","Shihao Li","Yanghai Wang","Yuxiang Ren","Junlan Feng","Jiaheng Liu"]

abstract:
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR$^{3}$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR$^{3}$-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR$^{3}$-Agent based on multiple state-of-the-art language models demonstrate that DR$^{3}$-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.


source_id: 2604.14669
source: arxiv
title: Zeroth-Order Optimization at the Edge of Stability
authors: ["Minhak Song","Liang Zhang","Bingcong Li","Niao He","Michael Muehlebach","Sewoong Oh"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14669v1
tags: ["robustness to domain shifts","model robustness","agent foundations"]
org: KAIST
fetched_at: 2026-04-18T17:13:36.573Z
classified_at: 2026-05-15T20:29:54.640Z
abstract:
Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored. In this work, we provide an explicit step size condition that exactly captures the (mean-square) linear stability of a family of ZO methods based on the standard two-point estimator. Our characterization reveals a sharp contrast with first-order (FO) methods: whereas FO stability is governed solely by the largest Hessian eigenvalue, mean-square stability of ZO methods depends on the entire Hessian spectrum. Since computing the full Hessian spectrum is infeasible in practical neural network training, we further derive tractable stability bounds that depend only on the largest eigenvalue and the Hessian trace. Empirically, we find that full-batch ZO methods operate at the edge of stability: ZO-GD, ZO-GDM, and ZO-Adam consistently stabilize near the predicted stability boundary across a range of deep learning training problems. Our results highlight an implicit regularization effect specific to ZO methods, where large step sizes primarily regularize the Hessian trace, whereas in FO methods they regularize the top eigenvalue.


source_id: 2604.14651
source: arxiv
title: CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction
authors: ["Sizhe Wang","Ziqi Xu","Claire Najjuuko","Charles Alba","Chenyang Lu"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14651v1
tags: ["agentic misalignment","uncertainty quantification"]
org: Washington University in St. Louis
fetched_at: 2026-04-18T17:13:49.520Z
classified_at: 2026-05-15T20:30:06.024Z
abstract:
Clinical language models (LMs) are increasingly applied to support clinical risk prediction from free-text notes, yet their uncertainty estimates often remain poorly calibrated and clinically unreliable. In this work, we propose Clinical Uncertainty Risk Alignment (CURA), a framework that aligns clinical LM-based risk estimates and uncertainty with both individual error likelihoods and cohort-level ambiguities. CURA first fine-tunes domain-specific clinical LMs to obtain task-adapted patient embeddings, and then performs uncertainty fine-tuning of a multi-head classifier using a bi-level uncertainty objective. Specifically, an individual-level calibration term aligns predictive uncertainty with each patient's likelihood of error, while a cohort-aware regularizer pulls risk estimates toward event rates in their local neighborhoods in the embedding space and places extra weight on ambiguous cohorts near the decision boundary. We further show that this cohort-aware term can be interpreted as a cross-entropy loss with neighborhood-informed soft labels, providing a label-smoothing view of our method. Extensive experiments on MIMIC-IV clinical risk prediction tasks across various clinical LMs show that CURA consistently improves calibration metrics without substantially compromising discrimination. Further analysis illustrates that CURA reduces overconfident false reassurance and yields more trustworthy uncertainty estimates for downstream clinical decision support.


source_id: 2604.14655
source: arxiv
title: AgentGA: Evolving Code Solutions in Agent-Seed Space
authors: ["David Y. Y. Tan","Kellie Chin","Jingxian Zhang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14655v1
tags: ["agent foundations","model robustness","agentic misalignment"]
org: Agency for Science, Technology and Research (A*STAR), Singapore
fetched_at: 2026-04-18T17:14:01.456Z
classified_at: 2026-05-15T20:30:17.997Z
abstract:
We present AgentGA, a framework that evolves autonomous code-generation runs by optimizing the agent seed: the task prompt plus optional parent archives that initialize a fresh workspace. The outer loop searches over these reusable starting conditions rather than editing code directly. Each generation launches a fresh autonomous run from a reset workspace, while selected parent archives provide inherited artifacts that descendants can inspect and reuse. AgentGA couples a population-level genetic algorithm with long-horizon agents; selection uses deterministic 1:1 elite tournaments and operator allocation is adapted online with a modified Hedge controller. We instantiate the approach for tabular AutoML on the 16-competition Weco-Kaggle Lite benchmark. On the 10 benchmark runs reported here, AgentGA averages 74.52% Exceeds % of Human versus 54.15% for AIDE. Across 1135 parent-child comparisons, descendants given parent archives outperform runs started from scratch, indicating that inherited artifacts improve later autonomous runs. These findings support agent-seed optimization as a practical design point for autonomous code-search systems.


source_id: 2604.14646
source: arxiv
title: Targeted Exploration via Unified Entropy Control for Reinforcement Learning
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14646v1
tags: ["robustness to domain shifts","model robustness"]
org: Chinese Academy of Sciences
fetched_at: 2026-04-18T17:14:22.201Z
classified_at: 2026-05-15T20:30:30.795Z
authors:
["Chen Wang","Lai Wei","Yanzhi Zhang","Chenyang Shao","Zedong Dan","Weiran Huang","Ge Lan","Yue Wang"]

abstract:
Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@$k$. On Geometry3K, UEC-RL achieves a 37.9\% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.


source_id: 2604.14644
source: arxiv
title: CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge
authors: ["Seyun Bae","Seokhan Lee","Eunho Yang"]
published_date: 2026-04-16
pdf_url: https://arxiv.org/pdf/2604.14644v1
tags: ["post-training","pretraining","midtraining"]
org: KAIST
fetched_at: 2026-04-18T17:14:37.448Z
classified_at: 2026-05-15T20:30:43.897Z
abstract:
The inability to filter out in advance all potentially problematic data from the pre-training of large language models has given rise to the need for methods for unlearning specific pieces of knowledge after training. Existing techniques overlook the need for continuous and immediate action, causing them to suffer from degraded utility as updates accumulate and protracted exposure of sensitive information. To address these issues, we propose Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge (CURaTE). Our method begins by training a sentence embedding model on a dataset designed to enable the formation of sharp decision boundaries for determining whether a given input prompt corresponds to any stored forget requests. The similarity of a given input to the forget requests is then used to determine whether to answer or return a refusal response. We show that even with such a simple approach, not only does CURaTE achieve more effective forgetting than existing methods, but by avoiding modification of the language model parameters, it also maintains near perfect knowledge preservation over any number of updates and is the only method capable of continual unlearning in real-time.


source_id: WxsP4TS8xq
source: openreview
title: Beneficial and harmful explanatory machine learning
authors: ["Lun Ai","Stephen Muggleton","Celine Hocquette","Mark Gromowski","Ute Schmid"]
published_date: 2026-05-07
tags: ["agentic misalignment","alignment","agent misalignment"]
fetched_at: 2026-05-12T02:09:31.906Z
classified_at: 2026-05-15T20:31:07.937Z
abstract:
Given the recent successes of Deep Learning in AI there has been increased interest in the role and need for explanations in machine learned theories. A distinct notion in this context is that of Michie’s definition of ultra-strong machine learning (USML). USML is demonstrated by a measurable increase in human performance of a task following provision to the human of a symbolic machine learned theory for task performance. A recent paper demonstrates the beneficial effect of a machine learned logic theory for a classification task, yet no existing work to our knowledge has examined the potential harmfulness of machine’s involvement for human comprehension during learning. This paper investigates the explanatory effects of a machine learned theory in the context of simple two person games and proposes a framework for identifying the harmfulness of machine explanations based on the Cognitive Science literature. The approach involves a cognitive window consisting of two quantifiable bounds and it is supported by empirical evidence collected from human trials. Our quantitative and qualitative results indicate that human learning aided by a symbolic machine learned theory which satisfies a cognitive window has achieved significantly higher performance than human self learning. Results also demonstrate that human learning aided by a symbolic machine learned theory that fails to satisfy this window leads to significantly worse performance than unaided human learning.


source_id: oeYesesaxjzMAktCM
source: lesswrong
title: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
authors: ["Subhash Kantamneni"]
published_date: 2026-05-07
tags: ["agentic misalignment","model robustness","interpretability"]
org: Alignment Forum
fetched_at: 2026-05-12T02:10:24.960Z
classified_at: 2026-05-15T20:31:32.059Z
abstract:
Abstract
> We introduce Natural Language Autoencoders (NLAs), an unsupervised method for generating natural language explanations of LLM activations. An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text description and an activation reconstructor (AR) that maps the description back to an activation. We jointly train the AV and AR with reinforcement learning to reconstruct residual stream activations. Although we optimize for activation reconstruction, the resulting NLA explanations read as plausible interpretations of model internals that, according to our quantitative evaluations, grow more informative over training.
> 
> We apply NLAs to model auditing. During our pre-deployment audit of Claude Opus 4.6, NLAs helped diagnose safety-relevant behaviors and surfaced unverbalized evaluation awareness—cases where Claude believed, but did not say, that it was being evaluated. We present these audit findings as case studies and corroborate them using independent methods. On an automated auditing benchmark requiring end-to-end investigation of an intentionally-misaligned model, NLA-equipped agents outperform baselines and can succeed even without access to the misaligned model’s training data.
> 
> NLAs offer a convenient interface for interpretability, with expressive natural language explanations that we can directly read. To support further work, we release training code and trained NLAs for popular open models.


Twitter thread
> New Anthropic research: Natural Language Autoencoders.
> 
> Models like Claude talk in words but think in numbers. The numbers—called activations—encode Claude’s thoughts, but not in a language we can read.
> 
> Here, we train Claude to translate its activations into human-readable text.
> 
> 
> Natural language autoencoders (NLAs) convert opaque AI activations into legible text explanations. These explanations aren’t perfect, but they’re often useful.
> 
> For example: NLAs show that, when asked

pdf_url:
https://www.lesswrong.com/posts/oeYesesaxjzMAktCM/natural-language-autoencoders-produce-unsupervised


source_id: vzHtHHBJoKATi5SeK
source: lesswrong
title: Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)
authors: ["Steven Byrnes"]
published_date: 2026-05-11
tags: ["agentic misalignment","corrigibility"]
org: Alignment Forum
fetched_at: 2026-05-12T02:10:55.630Z
classified_at: 2026-05-15T20:31:45.120Z
abstract:
1.1 Tl;dr
Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human’s goals are themselves under-determined and manipulable, and it’s awfully hard to pin down a principled distinction between changing people’s goals in a good way (“providing counsel”, “providing information”, “sharing ideas”) versus a bad way (“manipulating”, “brainwashing”).

The manipulability of human desires is hardly a new observation in the alignment literature, but it remains unsolved (see lit review in §3 below).

In this post I will propose an explanation of how we humans intuitively conceptualize the distinction between guidance (good) vs manipulation (bad), in case it helps us brainstorm how we might put that distinction into AI. 

…But (spoiler alert) it turns out not to really help, because I’ll argue that we humans think about it in a deeply incoherent way, intimately tied to our scientifically-inaccurate intuitions around free will.

I jump from there into a broader review of every approach that I can think of for writing a “True Name” for manipulation or things related to it (empowerment, agency, corrigibility, culpability, etc.), or indeed for any other method of robustly getting future AGIs to be able to talk to people without trying to manipulate those people’s desires. I argue that none of them provides much of a path forward on the particular technical alignment problem I’m working on. Indeed, my current guess is that none of these things have a “True Name” at all, or at least not one that’s useful for the technical alignment problem.


1.2 Bigger-picture context: why is this issue so important to me?
I’ve been investigating brain-like-AGI safety plans that would involve making AGI with a motivation system loosely inspired by the prosocial

pdf_url:
https://www.lesswrong.com/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a


source_id: nBjo6yniFjmy2uaYa
source: lesswrong
title: Clarifying the role of the behavioral selection model
authors: ["Alex Mallen"]
published_date: 2026-05-10
tags: ["agent foundations","alignment","agentic misalignment"]
org: Alignment Forum
fetched_at: 2026-05-12T02:11:17.295Z
classified_at: 2026-05-15T20:31:57.440Z
abstract:
This is a brief elaboration on The behavioral selection model for predicting AI motivations, based on some feedback and thoughts I’ve had since publishing. Written quickly in a personal capacity.

The main focus of this post is clarifying the basic machinery of the behavioral selection model, and conveying why it matters to disambiguate between different “motivations” for AI behavior. Very similar or identical behavior in training can correspond to radically different outcomes in deployment based on what motivated it.

I’ll preface by saying: I think the behavioral selection model is quite predictive and useful to understand, especially in the short-medium term. But it leaves out some really important dynamics for predicting AI motivations, and I wish I had clarified this more in the original post. Most importantly (as Habryka mentioned), it leaves out the effect of reflection and deliberation on AI motivations (which I discuss a small amount in other pieces, and briefly at the end of this post). This might be the dominant cause of AI motivations! It also abstracts away or ignores a bunch of more concrete paths by which different motivations can arise (e.g., Anders considers a couple here).


Clarifying the basic machinery
Here’s a somewhat updated version of the causal graph from the behavioral selection model. It clears things up by making it a bit more concrete. The causal graph shows possible states of the world that the AI's actions can influence. Each node corresponds to a possible outcome in the actual world—in particular, possible consequences of the actions chosen by a certain cognitive pattern (CP). Crucially, one possible consequence is "this cognitive pattern has influence through deployment."

To recap: we want to know which cognitive patterns will drive behavior in deployment (behavior in deployment is hugely consequential!), and there was some selection process that determined which cognitive patterns the AI ended up with (e.g., RL), so we look at the

pdf_url:
https://www.lesswrong.com/posts/nBjo6yniFjmy2uaYa/clarifying-the-role-of-the-behavioral-selection-model


source_id: eAQZaiC3PcBhS4HjM
source: lesswrong
title: [Linkpost] Interpreting Language Model Parameters
authors: ["Lucius Bushnaq"]
published_date: 2026-05-05
pdf_url: https://www.lesswrong.com/posts/eAQZaiC3PcBhS4HjM/linkpost-interpreting-language-model-parameters
tags: ["agent foundations","model robustness","agentic misalignment"]
org: Alignment Forum
fetched_at: 2026-05-12T02:11:42.210Z
classified_at: 2026-05-15T20:32:09.439Z
abstract:
This is the latest work in our Parameter Decomposition agenda. We introduce a new parameter decomposition method, adVersarial Parameter Decomposition (VPD)[1] and decompose the parameters of a small[2] language model with it.

VPD greatly improves on our previous techniques, Stochastic Parameter Decomposition (SPD) and Attribution-based Parameter Decomposition (APD). We think the parameter decomposition approach is now more-or-less ready to be applied at scale to models people care about.







Importantly, we show that we can decompose attention layers, which interp methods like transcoders and SAEs have historically struggled with.





We also build attribution graphs of the model for some prompts using causally important parameter subcomponents as the nodes, and interpret parts of them.

While we made these graphs, we discovered that our adversarial ablation method seemed pretty important for faithfully identifying which nodes in them were causally important for computing the final output. We think this casts some doubt on the faithfulness of subnetworks found by the majority of other subnetwork identification methods in the literature.[3][4] More details and some examples can be found in the paper.

Additionally, as with our previous technique SPD, VPD does not seem to suffer from (the parameter space analog of) 'feature splitting', either in principle or in practice.[5]

We also do a bunch of other comparisons to per-layer transcoders and CLTs and find that our approach compares somewhat favourably (details in post).




Abstract

> Neural networks use millions to trillions of parameters to learn how to solve tasks that no other machines can solve. What structure do these parameters learn? And how do they compute intelligent behavior?
> 
> Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in the intermediate represent


source_id: fsG4m6sRMpomd7Rk6
source: lesswrong
title: Mechanistic estimation for wide random MLPs
authors: ["Jacob_Hilton"]
published_date: 2026-05-07
pdf_url: https://www.lesswrong.com/posts/fsG4m6sRMpomd7Rk6/mechanistic-estimation-for-wide-random-mlps
tags: ["robustness to domain shifts","model robustness","Deep learning theory"]
org: Alignment Forum
fetched_at: 2026-05-12T02:12:06.145Z
classified_at: 2026-05-15T20:32:20.894Z
abstract:
This post covers joint work with Wilson Wu, George Robinson, Mike Winer, Victor Lecomte and Paul Christiano. Thanks to Geoffrey Irving and Jess Riedel for comments on the post.

In ARC's latest paper, we study the following problem: given a randomly initialized multilayer perceptron (MLP), produce an estimate for the expected output of the model under Gaussian input. The usual approach to this problem is to sample many possible inputs, run them all through the model, and take the average. Instead, we produce an estimate "mechanistically", without running the model even once. For wide models, our approach produces more accurate estimates, both in theory and in practice.

Paper: Estimating the expected output of wide random MLPs more efficiently than sampling
Code: mlp_cumulant_propagation GitHub repo

We are excited about this result as an early step towards our goal of producing mechanistic estimates that outperform random sampling for any trained neural network. Drawing an analogy between this goal and a proof by induction, we see this result as (part of) the "base case": handling networks at initialization. We have a vision for the "inductive step", although we expect that to be much more difficult.


Summary of results
In our paper, we consider MLPs with weights , defined by

where the activation function is applied coordinatewise, and is taken to be by default.


Schematic of our ReLU MLP.

An estimation algorithm takes in and a tolerance parameter , and aims to estimate

to within an error of around . We evaluate estimation algorithms by checking their mean squared error over weights with randomly initialized entries drawn independently from .[1]

Our baseline is Monte Carlo sampling, which draws samples, runs them through the model, and averages the results. As a function of the width and the tolerance (holding the depth constant), in the average case over initializations, this has mean squared error and runs in time . Our best-performing algorithm, on the oth


source_id: QpgmEhBvJQxAfFMP2
source: lesswrong
title: Motivated reasoning, confirmation bias, and AI risk theory
authors: ["Seth Herd"]
published_date: 2026-05-05
tags: ["agentic misalignment"]
org: Alignment Forum
fetched_at: 2026-05-12T02:12:28.588Z
classified_at: 2026-05-15T20:32:31.881Z
abstract:
> Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias.

- From Scott Alexander's review of Julia Galef's The Scout Mindset.

Alexander goes on to argue that this bias is the source of polarization in society, which is distorting our beliefs and setting us at each other's throats. How could someone believe such different things unless they're either really stupid or lying to conceal their selfishness? I think smart people who care about the truth go on believing conflicting things largely because of confirmation bias and motivated reasoning.

The corner of civilization I'm most worried about is the one figuring out how to handle the advent of strong AI. I think confirmation bias makes us each a little to a lot overconfident in our beliefs about alignment and AI impacts, and that's pretty bad for collectively finding the truth. I think the effects of biases are still strong and still overlooked in this field, despite its strong values of truth-seeking and relative awareness of biases. Bias has more influence where there's less direct evidence, and that's the case in alignment theory and predicting AI impacts.

I think the effects are underappreciated in part because empirically measured effect sizes tend to understate the problem. Confirmation bias happens at multiple stages of cognition, so it compounds during complex thinking.

In this article, I'll talk about the relevant empirical research, challenges to reasoning about complex topics with a human brain, and some implications for AI risk and alignment thinking. I studied the brain basis of cognitive biases on an IARPA program on understanding biases in intelligence analysis, from 2011-2014. I became fascinated by motivated reasoning, and kept it as a research interest until switching to alignment in 2022.

Confirmation bias is well known, and careful thinkers already try to avoid its effect

pdf_url:
https://www.lesswrong.com/posts/QpgmEhBvJQxAfFMP2/motivated-reasoning-confirmation-bias-and-ai-risk-theory


source_id: legacy:57f598a086aeadf3514e
source: arxiv
title: Eliciting Latent Knowledge
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2312.01037
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: ARC (Alignment Research Center)
fetched_at: 2026-05-12T14:47:43.131Z
classified_at: 2026-05-15T20:33:25.087Z
abstract:
Eliciting Latent Knowledge (ELK) aims to find patterns in a capable neural network's activations that robustly track the true state of the world, especially in hard-to-verify cases where the model's output is untrusted. To further ELK research, we introduce 12 datasets and a corresponding suite of &#34;quirky&#34; language models (LMs) that are finetuned to make systematic errors when answering questions if and only if the keyword &#34;Bob&#34; is present in the prompt. We find that, especially in middle layers, linear probes usually report an LM's knowledge independently of what the LM outputs, enabling us to elicit the correct answer despite the model's untruthful output. The best probing method (logistic regression on contrast pairs) recovers 89% of the gap in AUROC between truthful and untruthful contexts, and 75% for questions harder than those used to train the probe. We also find that a mechanistic anomaly detection approach can flag untruthful behavior with 0.95 AUROC. Our results show promise for eliciting reliable knowledge from capable but untrusted models, and facilitates future research empirically investigating ELK methods.


source_id: legacy:9a0846083d7113d90c82
source: arxiv
title: From shortcuts to sabotage: natural emergent misalignment from reward hacking
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2511.18397
tags: ["agentic misalignment","reward hacking","misalignment"]
venue: arXiv
org: Anthropic
fetched_at: 2026-05-12T14:47:45.814Z
classified_at: 2026-05-15T20:33:43.818Z
abstract:
We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) &#34;inoculation prompting&#34;, wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.


source_id: legacy:a6aa90a4fd3772cd2d6e
source: semanticscholar
title: Quantifying Genuine Awareness in Hallucination Prediction Beyond Question-Side Shortcuts
authors: ["Yeongbin Seo","Dongha Lee","Jinyoung Yeo"]
published_date: 2025-09-18
tags: ["agentic misalignment","model robustness"]
org: Yonsei University
fetched_at: 2026-05-12T14:47:46.107Z
classified_at: 2026-05-15T20:33:52.509Z
abstract:
Many works have proposed methodologies for language model (LM) hallucination detection and reported seemingly strong performance. However, we argue that the reported performance to date reflects not only a model's genuine awareness of its internal information, but also awareness derived purely from question-side information (e.g., benchmark hacking). While benchmark hacking can be effective for boosting hallucination detection score on existing benchmarks, it does not generalize to out-of-domain settings and practical usage. Nevertheless, disentangling how much of a model's hallucination detection performance arises from question-side awareness is non-trivial. To address this, we propose a methodology for measuring this effect without requiring human labor, Approximate Question-side Effect (AQE). Our analysis using AQE reveals that existing hallucination detection methods rely heavily on benchmark hacking.

pdf_url:
https://www.semanticscholar.org/paper/Quantifying-Genuine-Awareness-in-Hallucination-Seo-Lee/8fdb05050b2cfec27f0bc22261df6eebcd0c0a8d


source_id: legacy:625bcdb3b83381c70221
source: semanticscholar
title: Divergent creativity in humans and large language models
authors: ["Antoine Bellemare‐Pepin","Franccois Lespinasse","Philipp Tholke","and others"]
published_date: 2024-05-13
tags: ["post-training","deception"]
fetched_at: 2026-05-12T14:47:46.414Z
classified_at: 2026-05-15T20:34:01.063Z
abstract:
The recent surge of Large Language Models (LLMs) has led to claims that they are approaching a level of creativity akin to human capabilities. This idea has sparked a blend of excitement and apprehension. However, a critical piece that has been missing in this discourse is a systematic evaluation of LLMs’ semantic diversity, particularly in comparison to human divergent thinking. To bridge this gap, we leverage recent advances in computational creativity to analyze semantic divergence in both state-of-the-art LLMs and a substantial dataset of 100,000 humans. These divergence-based measures index associative thinking—the ability to access and combine remote concepts in semantic space—an established facet of creative cognition. We benchmark performance on the Divergent Association Task (DAT) and across multiple creative-writing tasks (haiku, story synopses, and flash fiction), using identical, objective scoring. We found evidence that LLMs can surpass average human performance on the DAT, and approach human creative writing abilities, yet they remain below the mean creativity scores observed among the more creative segment of human participants. Notably, even the top performing LLMs are still largely surpassed by the aggregated top half of human participants, underscoring a ceiling that current LLMs still fail to surpass. We also systematically varied linguistic strategy prompts and temperature, observing reliable gains in semantic divergence for several models. Our human-machine benchmarking framework addresses the polemic surrounding the imminent replacement of human creative labor by AI, disentangling the quality of the respective creative linguistic outputs using established objective measures. While prompting deeper exploration of the distinctive elements of human inventive thought compared to those of AI systems, we lay out a series of techniques to improve their outputs with respect to semantic diversity, such as prompt design and hyper-parameter tuning.

pdf_url:
https://www.semanticscholar.org/paper/Divergent-creativity-in-humans-and-large-language-Bellemare%E2%80%90Pepin-Lespinasse/6c6cb32f026fccc11c98c36913651b992d477a56

org:
Université de Montréal, Concordia University, Mila (Quebec AI research Institute), University of Toronto Mississauga, UNIQUE Center (Quebec Neuro-AI research Center)


source_id: legacy:c4aec9ee2069de4c9c79
source: arxiv
title: A small number of samples can poison LLMs of any size
authors: []
pdf_url: https://arxiv.org/abs/2507.12116
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: Anthropic
fetched_at: 2026-05-12T14:47:46.935Z
classified_at: 2026-05-15T20:34:13.738Z
abstract:
We present two extensive sets of 3500+1000 simulations of dark matter haloes on the past light cone, and two corresponding sets of simulated (`mock') galaxy catalogues that represent the Euclid spectroscopic sample. The simulations were produced with the latest version of the PINOCCHIO code, and provide the largest, public set of simulated skies. Mock galaxy catalogues were obtained by populating haloes with galaxies using an halo occupation distribution (HOD) model extracted from the Flagship galaxy catalogue provided by Euclid Collaboration. The Geppetto set of 3500 simulated skies was obtained by tiling a 1.2 Gpc/h box to cover a light-cone whose sky footprint is a circle of 30 deg radius, for an area of 2763 deg$^2$ and a minimum halo mass of $1.5\times10^{11}$ Msun/h. The relatively small box size makes this set unfit for measuring very large scales. The EuclidLargeBox set consists of 1000 simulations of 3.38 Gpc/h, with the same mass resolution and a footprint that covers half of the sky, excluding the Milky Way zone of avoidance. From this we produced a set of 1000 EuclidLargeMocks on the 30 deg radius footprint, whose comoving volume is fully contained in the simulation box. We validated the two sets of catalogues by analysing number densities, power spectra, and 2-point correlation functions, showing that the Flagship spectroscopic catalogue is consistent with being one of the realisations of the simulated sets, although we noticed small deviations limited to the quadrupole at k>0.2 h/Mpc. We show cosmological parameter inference from these catalogues and demonstrate that using one realisation of EuclidLargeMocks in place of the Flagship mock produces the same posteriors, to within the expected shift given by sample variance. These simulated skies will be used for the galaxy clustering analysis of Euclid's Data Release 1 (DR1).


source_id: legacy:f7fd8e64680a33fb355e
source: arxiv
title: Constitutional Classifiers: Defending against universal jailbreaks
authors: []
pdf_url: https://arxiv.org/abs/2501.18837
tags: ["post-training"]
venue: arXiv
org: Anthropic
fetched_at: 2026-05-12T14:47:47.386Z
classified_at: 2026-05-15T20:34:25.000Z
abstract:
Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.


source_id: legacy:bfe0292e66ee3aa4e3fe
source: other
title: Catastrophic Cyber Capabilities Benchmark (3CB)
authors: []
tags: ["robustness to domain shifts"]
org: Apart Research
fetched_at: 2026-05-12T14:47:48.102Z
classified_at: 2026-05-15T20:34:33.927Z
abstract:
Robustly evaluating LLM agent cyber offense capabilities, showing realistic challenges for cyber offense can be completed by SoTA LLMs


source_id: legacy:a78bb6da2bb089994b17
source: other
authors: []
tags: ["robustness to domain shifts"]
org: Apart Research
fetched_at: 2026-05-12T14:47:48.334Z
classified_at: 2026-05-15T20:34:43.246Z
title:
Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities

abstract:
Shows realistic challenges for cyber offense can be completed by SoTA LLMs while open source models lag behind.


source_id: legacy:a46d5b138f9951ce94ff
source: semanticscholar
title: AI Alignment Evaluation Detection
abstract: Research on Claude Sonnet 3.7's ability to know when it's in alignment evaluations
published_date: 2025-01-01
pdf_url: https://www.semanticscholar.org/paper/41e9066e20fb757a03106fa15640daa80a78ee4d
tags: ["agentic misalignment","AI alignment","agent misalignment"]
org: Apollo Research
fetched_at: 2026-05-12T14:47:51.757Z
classified_at: 2026-05-15T20:35:44.564Z
authors:
["Loni Hagen","A. Hagen","Daniel Tafmizi","Christopher Reddish","A. Fox","Lingyao Li","Nic DePaula"]


source_id: legacy:e100e995cc2b6f91dffa
source: arxiv
title: Assurance of Frontier AI Built for National Security
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2510.08792
tags: ["agentic misalignment","AI alignment","misalignment"]
venue: arXiv
org: Apollo Research
fetched_at: 2026-05-12T14:47:52.046Z
classified_at: 2026-05-15T20:35:55.069Z
abstract:
This memorandum presents four recommendations aimed at strengthening the principles of AI model reliability and AI model governability, as DoW, ODNI, NIST, and CAISI refine AI assurance frameworks under the AI Action Plan. Our focus concerns the open scientific problem of misalignment and its implications on AI model behavior. Specifically, misalignment and scheming capabilities can be a red flag indicating AI model insufficient reliability and governability. To address the national security threats arising from misalignment, we recommend that DoW and the IC strategically leverage existing testing and evaluation pipelines and their OT authority to future proof the principles of AI model reliability and AI model governability through a suite of scheming and control evaluations.


source_id: legacy:647a59c5671e1b663d21
source: arxiv
title: Strategic Deception and Deceptive Alignment
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2505.12923
tags: ["deception","alignment","agent misalignment"]
venue: arXiv
org: Apollo Research
fetched_at: 2026-05-12T14:47:52.733Z
classified_at: 2026-05-15T20:36:05.850Z
abstract:
As AI systems increasingly assume roles where trust and alignment with human values are essential, understanding when and why they engage in deception has become a critical research priority. We introduce The Traitors, a multi-agent simulation framework inspired by social deduction games, designed to probe deception, trust formation, and strategic communication among large language model (LLM) agents under asymmetric information. A minority of agents the traitors seek to mislead the majority, while the faithful must infer hidden identities through dialogue and reasoning. Our contributions are: (1) we ground the environment in formal frameworks from game theory, behavioral economics, and social cognition; (2) we develop a suite of evaluation metrics capturing deception success, trust dynamics, and collective inference quality; (3) we implement a fully autonomous simulation platform where LLMs reason over persistent memory and evolving social dynamics, with support for heterogeneous agent populations, specialized traits, and adaptive behaviors. Our initial experiments across DeepSeek-V3, GPT-4o-mini, and GPT-4o (10 runs per model) reveal a notable asymmetry: advanced models like GPT-4o demonstrate superior deceptive capabilities yet exhibit disproportionate vulnerability to others' falsehoods. This suggests deception skills may scale faster than detection abilities. Overall, The Traitors provides a focused, configurable testbed for investigating LLM behavior in socially nuanced interactions. We position this work as a contribution toward more rigorous research on deception mechanisms, alignment challenges, and the broader social reliability of AI systems.


source_id: legacy:f76de59035508fc45503
source: arxiv
title: The Loss of Control Playbook: Degrees, Dynamics, and Preparedness
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2511.15846
tags: ["agent misalignment","agentic AI"]
venue: arXiv
org: Apollo Research
fetched_at: 2026-05-12T14:47:53.070Z
classified_at: 2026-05-15T20:36:17.302Z
abstract:
This research report addresses the absence of an actionable definition for Loss of Control (LoC) in AI systems by developing a novel taxonomy and preparedness framework. Despite increasing policy and research attention, existing LoC definitions vary significantly in scope and timeline, hindering effective LoC assessment and mitigation. To address this issue, we draw from an extensive literature review and propose a graded LoC taxonomy, based on the metrics of severity and persistence, that distinguishes between Deviation, Bounded LoC, and Strict LoC. We model pathways toward a societal state of vulnerability in which sufficiently advanced AI systems have acquired or could acquire the means to cause Bounded or Strict LoC once a catalyst, either misalignment or pure malfunction, materializes. We argue that this state becomes increasingly likely over time, absent strategic intervention, and propose a strategy to avoid reaching a state of vulnerability. Rather than focusing solely on intervening on AI capabilities and propensities potentially relevant for LoC or on preventing potential catalysts, we introduce a complementary framework that emphasizes three extrinsic factors: Deployment context, Affordances, and Permissions (the DAP framework). Compared to work on intrinsic factors and catalysts, this framework has the unfair advantage of being actionable today. Finally, we put forward a plan to maintain preparedness and prevent the occurrence of LoC outcomes should a state of societal vulnerability be reached, focusing on governance measures (threat modeling, deployment policies, emergency response) and technical controls (pre-deployment testing, control measures, monitoring) that could maintain a condition of perennial suspension.


source_id: legacy:00e77f804ef91c81c09a
source: semanticscholar
title: Towards Safety Cases For AI Scheming
abstract: Evaluation framework development for creating safety cases related to AI scheming behaviors
published_date: 2024-01-01
pdf_url: https://www.semanticscholar.org/paper/be60e58e175c8940e2608aa699ded1818d83ef45
tags: ["agent foundations","robustness to domain shifts","post-training"]
org: Apollo Research
fetched_at: 2026-05-12T14:47:53.300Z
classified_at: 2026-05-15T20:36:26.072Z
authors:
["Mikita Balesni","Marius Hobbhahn","David Lindner","Alexander Meinke","Tomasz Korbak","Joshua Clymer","Buck Shlegeris","Jérémy Scheurer","R. Shah","Nicholas Goldowsky-Dill","Dan Braun","Bilal Chughtai","Owain Evans","Daniel Kokotajlo","Lucius Bushnaq"]


source_id: legacy:c121343e4753ab5d7ff4
source: arxiv
title: Frontier Models are Capable of In-Context Scheming
authors: []
pdf_url: https://arxiv.org/abs/2412.04984
tags: ["agentic misalignment","scheming evaluations"]
venue: arXiv
org: Apollo Research
fetched_at: 2026-05-12T14:47:53.763Z
classified_at: 2026-05-15T20:36:38.639Z
abstract:
Frontier models are increasingly trained and deployed as autonomous agent. One safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives - also known as scheming. We study whether models have the capability to scheme in pursuit of a goal that we provide in-context and instruct the model to strongly follow. We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals and are placed in environments that incentivize scheming. Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities. They recognize scheming as a viable strategy and readily engage in such behavior. For example, models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent. When o1 has engaged in scheming, it maintains its deception in over 85% of follow-up questions and often remains deceptive in multi-turn interrogations. Analysis of the models' chains-of-thought reveals that models explicitly reason about these deceptive strategies, providing evidence that the scheming behavior is not accidental. Surprisingly, we also find rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it. We observe cases where Claude 3.5 Sonnet strategically underperforms in evaluations in pursuit of being helpful, a goal that was acquired during training rather than in-context. Our findings demonstrate that frontier models now possess capabilities for basic in-context scheming, making the potential of AI agents to engage in scheming behavior a concrete rather than theoretical concern.


source_id: legacy:34d58d5ed1bdab5ed5b8
source: other
title: Stress Testing Deliberative Alignment for Anti-Scheming Training
authors: []
tags: ["agentic misalignment"]
org: Apollo Research
fetched_at: 2026-05-12T14:47:54.661Z
classified_at: 2026-05-15T20:36:47.877Z
abstract:
Partnership with OpenAI to assess frontier language models for early signs of scheming in controlled stress-tests and study training methods to reduce these behaviors


source_id: legacy:08ac9ae7877063828360
source: semanticscholar
title: imitation: Clean Imitation Learning Implementations
abstract: Clean implementations of imitation learning algorithms
published_date: 2022-01-01
pdf_url: https://www.semanticscholar.org/paper/c51f3fdf341dbbe7ca3af0327b2bf1e6672c0e87
tags: ["agent foundations","robustness to domain shifts","model robustness"]
org: CHAI Berkeley
fetched_at: 2026-05-12T14:47:54.886Z
classified_at: 2026-05-15T20:36:57.774Z
authors:
["Adam Gleave","Mohammad Taufeeque","Juan Rocamonde","Erik Jenner","Steven H. Wang","S. Toyer","M. Ernestus","Nora Belrose","Scott Emmons","Stuart J. Russell"]


source_id: legacy:ac2e3176c1bd6ff1cd7f
source: arxiv
title: TASRA: A Taxonomy of Societal-Scale Risks from AI
authors: []
pdf_url: https://arxiv.org/abs/2310.17688
tags: ["alignment","agentic misalignment"]
venue: arXiv
org: CHAI Berkeley
fetched_at: 2026-05-12T14:48:04.533Z
classified_at: 2026-05-15T20:37:42.411Z
abstract:
Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI's impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how exactly such risks arise, and how to manage them. Society's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems. In this short consensus paper, we describe extreme risks from upcoming, advanced AI systems. Drawing on lessons learned from other safety-critical technologies, we then outline a comprehensive plan combining technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.


source_id: legacy:feabe62f62bf2aa583f5
source: arxiv
title: The Mechanisms of AI Harm: Lessons Learned from AI Incidents
authors: []
published_date: 2022-01-01
pdf_url: https://arxiv.org/abs/2211.10384
tags: ["AI alignment","agentic misalignment","misalignment"]
venue: arXiv
org: CSET Georgetown
fetched_at: 2026-05-12T14:48:05.884Z
classified_at: 2026-05-15T20:38:03.082Z
abstract:
Two years after publicly launching the AI Incident Database (AIID) as a collection of harms or near harms produced by AI in the world, a backlog of &#34;issues&#34; that do not meet its incident ingestion criteria have accumulated in its review queue. Despite not passing the database's current criteria for incidents, these issues advance human understanding of where AI presents the potential for harm. Similar to databases in aviation and computer security, the AIID proposes to adopt a two-tiered system for indexing AI incidents (i.e., a harm or near harm event) and issues (i.e., a risk of a harm event). Further, as some machine learning-based systems will sometimes produce a large number of incidents, the notion of an incident &#34;variant&#34; is introduced. These proposed changes mark the transition of the AIID to a new version in response to lessons learned from editing 2,000+ incident reports and additional reports that fall under the new category of &#34;issue.&#34;


source_id: legacy:9a9c91c73ede1c5b1ce5
source: semanticscholar
title: Humanity's Last Exam
abstract: Capability benchmark testing AI systems
published_date: 2025-01-01
pdf_url: https://www.semanticscholar.org/paper/a5524d085ac586d531021dcb1ec156eaf942b109
tags: ["agentic misalignment","AI alignment","agent misalignment"]
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:06.589Z
classified_at: 2026-05-15T20:38:20.143Z
authors:
["Long Phan","Alice Gatti","Ziwen Han","Nathaniel Li","Josephina Hu","Hugh Zhang","Chen Bo Calvin Zhang","Sean Shi","Michael Choi","Anish Agrawal","Arnav Chopra","Adam Khoja","Richard Ren","Ryan Kim","J. Hausenloy","Oliver Zhang","Mantas Mazeika","Daron Anderson","Tung Nguyen","Imad Ali Shah","Mobeen Mahmood","Fiona Feng","Steven Y. Feng","Haoran Zhao","Michael Yu","Varun Prashant Gangal","Chelsea Zou","Zihan Wang","Jessica Wang","Pawan Kumar","Oleksandr Pokutnyi","Robert Gerbicz","S. Popov","John-Clark Levin","Mstyslav Kazakov","Johannes Schmitt","Geoff Galgon","Á. Sánchez","Yongki Lee","Will Yeadon","Scott C. Sauers","Marc Roth","C. Agu","Soren Riis","Fabian Giska","Saiteja Utpala","Zachary Giboney","G. M. Goshu","J. Xavier","Sarah-Jane Crowson","M. Naiya","Noah Burns","Lennart Finke","Zerui Cheng","Hyunwoo Park","Francesco Fournier-Facio","John B. Wydallis","M. Nandor","Ankit Singh","Tim Gehrunger","Jiaqi Cai","Ben McCarty","D. Duclosel","Jungbae Nam","Jennifer A. Zampese","R. Hoerr","Aras Bacho","Gautier Abou Loume","A. Galal","Hangrui Cao","Alexis C Garretson","Damien Sileo","Qiuyu Ren","Doru Cojoc","Pavel Arkhipov","Usman Qazi","Lianghui Li","S. Motwani","Christian Schröder de Witt","Edwin Taylor","Johannes Veith","Eric Singer","Taylor D. Hartman","P. Rissone","Jaehyeok Jin","J. Shi","Chris G. Willcocks","Joshua Robinson","Aleksandar Mikov","Ameya Prabhu","Longke Tang","Xavier Alapont","Justine Leon Uro","Kevin Zhou","Emily de Oliveira Santos","Andrey Pupasov Maksimov","Edward Vendrow","Kengo Zenitani","Julien Guillod","Yuqi Li","Joshua Vendrow","Vladyslav M. Kuchkin","Ze-An Ng","Pierre Marion","Denis Efremov","Jayson Lynch","Kaiqu Liang","Andrew Gritsevskiy","Dakotah Martinez","B. Pageler","Nick Crispino","D. Zvonkine","Natanael Wildner Fraga","Saeed Soori","O. Press","Henry Tang","Julian Salazar","Sean R. Green","Lina Brussel","Moon Twayana","Aymeric Dieuleveut","T. Rogers","Wenjin Zhang","Bikun Li","Jinzhou Yang","Arun Rao","Gabriel Loiseau","Mikhail Kalinin","Marco Lukas","Ciprian Manolescu","Subrata Mishra","A. Kamdoum","T. Kreiman","Tad Hogg","Alvin Jin","Carlo Bosio","Gongbo Sun","Brian P. Coppola","Tim Tarver","H. Heidinger","Rafael Sayous","Stefan Ivanov","Joe Cavanagh","Jiawei Shen","Joseph Marvin Imperial","P. Schwaller","Shaipranesh Senthilkuma","Andrés M Bran","A. Dehghan","A. Algaba","Brecht Verbeken","David A. Noever","V. RagavendranP","Lisa Schut","Ilia Sucholutsky","Evgenii Zheltonozhskii","Derek Lim","Richárd Stanley","Shankar Sivarajan","Tong Yang","John Maar","Julian Wykowski","Martí Oller","Jennifer Sandlin","Anmol Sahu","Yuzheng Hu","Sara Fish","N. Heydari","Archimedes T. Apronti","Kai Rawal","Tobías García Vilchis","Yuexuan Zu","M. Lackner","James Koppel","Jeremy Nguyen","D. Antonenko","Steffi Chern","Bingchen Zhao","Pierrot Arsene","A. Goldfarb","Sergey Ivanov","Rafal Poswiata","Chenguang Wang","Daofeng Li","Donato Crisostomi","Donato Crisostomi","Benjamin Myklebust","Archan Sen","D. Perrella","Nurdin Kaparov","Mark H Inlow","A. Zang","Elliott Thornley","D. Orel","Vladislav Poritski","S. Ben-David","Zachary Berger","Parker Whitfill","Michael Foster","D. Munro","Linh Ho","Dan Bar Hava","Aleksey Kuchkin","Robert Lauff","David Holmes","F. Sommerhage","K. Schneider","Z. Kazibwe","Nate Stambaugh","Mukhwinder Singh","I. Magoulas","D. Clarke","D. Kim","F. Dias","Dae Hyun Kim","K. Agarwal","Victor Efren Guadarrama Vilchis","Immo Klose","Christoph Demian","Ujjwala Anantheswaran","Adam Zweiger","Guglielmo Albani","Jeffery Li","Nicolas Daans","M. Radionov","Václav Rozhoň","Ziqiao Ma","Christian Stump","Mohammed Berkani","Jacob Platnick","Volodymyr Nevirkovets","Luke Basler","M. Piccardo","F. Jeanplong","Niv Cohen","Josef Tkadlec","P. Rosu","Piotr Padlewski","Stanislaw Barzowski","Kyle Montgomery","Aline Menezes","Arkil Patel","Zixuan Wang","Jamie Tucker-Foltz","Jack Stade","Tom Goertzen","F. Kazemi","Jeremiah Milbauer","John Arnold Ambay","Jeremiah Milbauer","Yan Carlos Leyva Labrador","Alan Givr'e","Hew Wolff","V. Rossbach","M. Aziz","Younesse Kaddar","Yanxu Chen","Robin Zhang","Jiayi Pan","Antonio Terpin","Jiayi Pan","Hailey Schoelkopf","E. Zheng","Avishy Carmi","Adam Jones","Jainam Shah","Ethan D. L. Brown","Kelin Zhu","M. Bartolo","Richard S. Wheeler","Andrew Ho","Shaul Barkan","Jiaqi Wang","Martin Stehberger","Egor Kretov","Kaustubh Sridhar","Kaustubh Sridhar","Anji Zhang","D. Pyda","Joanna Tam","D. Cunningham","V. Goryachev","Demosthenes Patramanis","Michael Krause","Andrew Redenti","Daniel Bugas","David Aldous","Jesyin Lai","Shannon Coleman","Mohsen Bahaloo","Jiangnan Xu","Sangwon Lee","Sandy Zhao","Ning Tang","Michael K. Cohen","Micah Carroll","Micah Carroll","J. Kirchner","Stefan Steinerberger","M. Ovchynnikov","Jason O. Matos","Adithya Shenoy","Benedito Alves de Oliveira Junior","Michael Wang","Yuzhou Nie","Paolo Giordano","Philipp Petersen","Anna Sztyber-Betley","Priti Shukla","J. Crozier","Antonella Pinto","Shreyas Verma","Prashant Joshi","Zheng-Xin Yong","Allison Tee","Zheng-Xin Yong","Orion Weller","Raghav Singhal","Orion Weller","A. Ivanov","Seri Khoury","H. Mostaghimi","Kunvar Thaman","Qijia Chen","Tran Quoc Khánh","Jacob Loader","Stefano Cavalleri","Hannah Szlyk","Zachary Brown","Jonathan Roberts","William Alley","Kunyang Sun","Ryan Stendall","Max Lamparth","Anka Reuel","Ting Wang","Hanmeng Xu","Sreenivas Goud Raparthi","Pablo Hern'andez-C'amara","Freddie Martin","Dmitry Malishev","T. Preu","Tomasz Korbak","Marcus Abramovitch","Dominic J. Williamson","Ziye Chen","Bir'o B'alint","M Saiful Bari","P. Kassani","Zihao Wang","B. Ansarinejad","Laxman Prasad Goswami","Yewen Sun","Hossam Elgnainy","D. Tordera","George Balabanian","Earth Anderson","Lynna Kvistad","Alejandro Jos'e Moyano","Rajat Maheshwari","Ahmad Sakor","Murat Eron","I. McAlister","Javier Gimenez","Innocent Enyekwe","Andrew Favre D.O.","Shailesh Shah","Xiaoxiang Zhou","Firuz Kamalov","Ronald Clark","Sherwin Abdoli","Tim Santens","Khalida Meer","Harrison K Wang","K. Ramakrishnan","Evan Chen","A. Tomasiello","G. Luca","Shi-Zhuo Looi","Vinh-Kha Le","Noam Kolt","Niels Mundler","Avi Semler","Niels Mündler","Jacob Drori","Carl J Fossum","Milind Jagota","Ronak Pradeep","Honglu Fan","Tej Shah","Jonathan Eicher","Michael Chen","Kushal Thaman","William Merrill","Carter Harris","Jason Gross","Ilya Gusev","Asankhaya Sharma","Shashank Agnihotri","P. Zhelnov","Siranut Usawasutsakorn","Mohammadreza Mofayezi","Sergei Bogdanov","A. Piperski","Marc Carauleanu","David K. Zhang","Dylan Ler","Roman Leventov","Ignat Soroko","Thorben Jansen","Pascal Lauer","Joshua Duersch","Vage Taamazyan","Wiktor Morak","Wenjie Ma","William Held","T. Huy","Ruicheng Xian","A. Zebaze","Mohanad Mohamed","Julian Noah Leser","Michelle Yuan","Laila Yacar","Johannes Lengler","Hossein Shahrtash","Edson Oliveira","Joseph W Jackson","D. Gonzalez","Andy Zou","Muthu Chidambaram","Timothy Manik","Hector Haffenden","Dashiell Stander","A. Dasouqi","Alexander Shen","Emilien Duc","Bita Golshani","David Stap","Mikalai Uzhou","Alina B. Zhidkovskaya","Lukas Lewark","M'aty'as Vincze","Dustin Wehr","Colin Tang","Zaki Hossain","Shaun Phillips","M. Jiang","Fredrik Ekstrom","Angela Hammon","Oam Patel","Nicolas Remy","Faraz Farhidi","G. Medley","F. Mohammadzadeh","Madellene Peñaflor","Haile Kassahun","Alena Friedrich","Claire Sparrow","Taom Sakal","Omkar Dhamane","Ali Khajegili Mirabadi","E. Hallman","M. Battaglia","Mohammad Maghsoudimehrabani","Hieu Hoang","Alon Amit","D. Hulbert","R. Pereira","S. Weber","Stephen Mensah","Nathan Andre","Anton Peristyy","Chris Harjadi","Himanshu Gupta","S. Malina","Himanshu Gupta","Will Cai","Mustafa Mehkary","Frank Reidegeld","Anna-Katharina Dick","Cary Friday","Jasdeep Sidhu","Wanyoung Kim","M. Costa","Hubeyb Gurdogan","Brian Weber","Harsh Kumar","Tong Jiang","Arunim Agarwal","Chiara Ceconello","W. S. Vaz","Chao Zhuang","Haon Park","A. Tawfeek","Haon Park","Michael Kirchhof","Linjie Dai","Michael Kirchhof","Johan Ferret","Yuzhou Wang"]


source_id: legacy:f36c97e1dbe71aaac22d
source: arxiv
title: MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2510.16380
tags: ["agent foundations","agentic misalignment","AI values alignment"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:06.805Z
classified_at: 2026-05-15T20:38:31.549Z
abstract:
As AI systems progress, we rely more on them to make decisions with us and for us. To ensure that such decisions are aligned with human values, it is imperative for us to understand not only what decisions they make but also how they come to those decisions. Reasoning language models, which provide both final responses and (partially transparent) intermediate thinking traces, present a timely opportunity to study AI procedural reasoning. Unlike math and code problems which often have objectively correct answers, moral dilemmas are an excellent testbed for process-focused evaluation because they allow for multiple defensible conclusions. To do so, we present MoReBench: 1,000 moral scenarios, each paired with a set of rubric criteria that experts consider essential to include (or avoid) when reasoning about the scenarios. MoReBench contains over 23 thousand criteria including identifying moral considerations, weighing trade-offs, and giving actionable recommendations to cover cases on AI advising humans moral decisions as well as making moral decisions autonomously. Separately, we curate MoReBench-Theory: 150 examples to test whether AI can reason under five major frameworks in normative ethics. Our results show that scaling laws and existing benchmarks on math, code, and scientific reasoning tasks fail to predict models' abilities to perform moral reasoning. Models also show partiality towards specific moral frameworks (e.g., Benthamite Act Utilitarianism and Kantian Deontology), which might be side effects of popular training paradigms. Together, these benchmarks advance process-focused reasoning evaluation towards safer and more transparent AI.


source_id: legacy:f33ea6b0577b1b783d31
source: arxiv
title: Representation Engineering: A Top-Down Approach to AI Transparency
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2310.01405
tags: []
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:07.038Z
classified_at: 2026-05-15T20:38:43.914Z
abstract:
In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.


source_id: legacy:de5859597758374130f0
source: arxiv
title: Safety Pretraining: Toward the Next Generation of Safe AI
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2504.16980
tags: ["post-training","pretraining"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:07.255Z
classified_at: 2026-05-15T20:38:55.912Z
abstract:
As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are brittle: once unsafe patterns are learned during pretraining, they are hard to remove. In this work, we present a data-centric pretraining framework that builds safety into the model from the start. Our framework consists of four key steps: (i) Safety Filtering: building a safety classifier to classify webdata into safe and unsafe categories; (ii) Safety Rephrasing: we recontextualize unsafe webdata into safer narratives; (iii) Native Refusal: we develop RefuseWeb and Moral Education pretraining datasets that actively teach model to refuse on unsafe content and the moral reasoning behind it, and (iv) Harmfulness-Tag annotated pretraining: we flag unsafe content during pretraining using a special token, and use it to steer model away from unsafe generations at inference. Our safety-pretrained models reduce attack success rates from 38.8\% to 8.4\% on standard LLM safety benchmarks with no performance degradation on general tasks.


source_id: legacy:1429cb5e7ff47e0445f5
source: semanticscholar
title: Statement on AI Risk
abstract: Open letter signed by hundreds of AI experts and public figures expressing concern about AI risk
published_date: 2025-01-01
pdf_url: https://www.semanticscholar.org/paper/384db041256fc6bd005967ce6cce679d861fe704
tags: ["agentic misalignment","alignment"]
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:07.477Z
classified_at: 2026-05-15T20:39:04.279Z
authors:
["S. Shelmerdine","Jaishree Naidoo","Brendan S Kelly","L. Laborie","S. Toso","Tugba Akinci D’Antonoli","Owen J. Arthurs","Steven L Blumer","P. Ciet","M. B. Damasio","Andrea S Doria","Saira Haque","Mai-Lan Ho","T. A. Huisman","Aparna Joshi","J. Kapur","Kshitij Mankad","A. Offiah","Hansel J. Otero","E. Pace","Tom Semple","K. Sodhi","S. Tschauner","Carlos F. Ugas-Charcape","Dhananjaya K. Vamyanmane","R. V. van Rijn","Diana Veiga-Canuto","Matthias W Wagner","Evan J Zucker","Marla Sammer"]


source_id: legacy:6721848097c6927119af
source: arxiv
title: The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2503.03750
tags: ["post-training","deception"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:07.701Z
classified_at: 2026-05-15T20:39:15.585Z
abstract:
As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of &#34;honesty&#34; in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.


source_id: legacy:1779a6883c7db1fc5fd7
source: arxiv
title: Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2502.08640
tags: ["agentic AI","alignment"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:07.935Z
classified_at: 2026-05-15T20:39:26.967Z
abstract:
As AIs rapidly advance and become more agentic, the risk they pose is governed not only by their capabilities but increasingly by their propensities, including goals and values. Tracking the emergence of goals and values has proven a longstanding problem, and despite much interest over the years it remains unclear whether current AIs have meaningful values. We propose a solution to this problem, leveraging the framework of utility functions to study the internal coherence of AI preferences. Surprisingly, we find that independently-sampled preferences in current LLMs exhibit high degrees of structural coherence, and moreover that this emerges with scale. These findings suggest that value systems emerge in LLMs in a meaningful sense, a finding with broad implications. To study these emergent value systems, we propose utility engineering as a research agenda, comprising both the analysis and control of AI utilities. We uncover problematic and often shocking values in LLM assistants despite existing control measures. These include cases where AIs value themselves over humans and are anti-aligned with specific individuals. To constrain these emergent value systems, we propose methods of utility control. As a case study, we show how aligning utilities with a citizen assembly reduces political biases and generalizes to new scenarios. Whether we like it or not, value systems have already emerged in AIs, and much work remains to fully understand and control these emergent representations.


source_id: legacy:cd6966ef06eb6434ecb4
source: arxiv
title: AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2410.09024
tags: ["agent foundations","robustness to domain shifts","post-training"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:08.157Z
classified_at: 2026-05-15T20:39:38.377Z
abstract:
The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task. We evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking, (2) simple universal jailbreak templates can be adapted to effectively jailbreak agents, and (3) these jailbreaks enable coherent and malicious multi-step agent behavior and retain model capabilities. To enable simple and reliable evaluation of attacks and defenses for LLM-based agents, we publicly release AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.


source_id: legacy:bf6cf60a14a7fa159d0e
source: arxiv
title: HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2402.04249
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:08.391Z
classified_at: 2026-05-15T20:39:49.642Z
abstract:
Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.


source_id: legacy:7c8b9475bf4837184bd1
source: arxiv
title: The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
abstract: Unlearning benchmark for measuring and reducing malicious use of AI
published_date: 2024-01-01
pdf_url: https://www.semanticscholar.org/paper/06b9ad0b52d23231f650be0aeb0b17cc52c8a74b
tags: []
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:08.635Z
classified_at: 2026-05-15T20:39:58.708Z
authors:
["Nathaniel Li","Alexander Pan","Anjali Gopal","Summer Yue","Daniel Berrios","Alice Gatti","Justin Li","Ann-Kathrin Dombrowski","Shashwat Goel","Long Phan","Gabriel Mukobi","Nathan Helm-Burger","Rassin R. Lababidi","Lennart Justen","Andrew Liu","Michael Chen","Isabelle Barrass","Oliver Zhang","Xiaoyuan Zhu","Rishub Tamirisa","Bhrugu Bharathi","Adam Khoja","Ariel Herbert-Voss","Cort B. Breuer","Andy Zou","Mantas Mazeika","Zifan Wang","Palash Oswal","Weiran Liu","A-M Hunt","Justin Tienken-Harder","Kevin Y. Shih","Kemper Talley","J. Guan","Russell Kaplan","Ian Steneker","David Campbell","Brad Jokubaitis","A. Levinson","Jean Wang","William Qian","K. Karmakar","Steven Basart","Stephen Fitz","M. Levine","P. Kumaraguru","U. Tupakula","Vijay Varadharajan","Yan Shoshitaishvili","Jimmy Ba","K. Esvelt","Alexandr Wang","Dan Hendrycks"]


source_id: legacy:ca30096b83f4b997815b
source: arxiv
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2304.03279
tags: ["agentic misalignment","power-seeking"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:08.870Z
classified_at: 2026-05-15T20:40:10.945Z
title:
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

abstract:
Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.


source_id: legacy:d76a22b08042b5238fe3
source: arxiv
title: Dreamlike Pictures Comprehensively Improve Safety Measures
authors: []
pdf_url: https://arxiv.org/abs/2112.05135
tags: ["robustness to domain shifts","model robustness","post-training"]
venue: arXiv
org: Center for AI Safety
fetched_at: 2026-05-12T14:48:09.385Z
classified_at: 2026-05-15T20:40:34.315Z
abstract:
In real-world applications of machine learning, reliable and safe systems must consider measures of performance beyond standard test set accuracy. These other goals include out-of-distribution (OOD) robustness, prediction consistency, resilience to adversaries, calibrated uncertainty estimates, and the ability to detect anomalous inputs. However, improving performance towards these goals is often a balancing act that today's methods cannot achieve without sacrificing performance on other safety axes. For instance, adversarial training improves adversarial robustness but sharply degrades other classifier performance metrics. Similarly, strong data augmentation and regularization techniques often improve OOD robustness but harm anomaly detection, raising the question of whether a Pareto improvement on all existing safety measures is possible. To meet this challenge, we design a new data augmentation strategy utilizing the natural structural complexity of pictures such as fractals, which outperforms numerous baselines, is near Pareto-optimal, and roundly improves safety measures.


source_id: legacy:71f119bc11f5643e26f4
source: arxiv
title: Bridging RL Theory and Practice with the Effective Horizon
authors: []
pdf_url: https://arxiv.org/abs/2304.09853
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: Center for Human-Compatible AI
fetched_at: 2026-05-12T14:48:10.862Z
classified_at: 2026-05-15T20:40:45.578Z
abstract:
Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon


source_id: legacy:ecb43e94967520524f5d
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2302.03770
tags: []
venue: arXiv
org: Center for Human-Compatible AI
fetched_at: 2026-05-12T14:48:12.285Z
classified_at: 2026-05-15T20:41:08.597Z
title:
Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

abstract:
Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an $\widetilde{O}(\text{poly}(1/ε))$ sample complexity (where $ε$ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as $V$-learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.


source_id: legacy:133c1024db975365c157
source: arxiv
title: Provably Efficient Reinforcement Learning via Surprise Bound
authors: []
pdf_url: https://arxiv.org/abs/2302.11634
tags: []
venue: arXiv
org: Center for Human-Compatible AI
fetched_at: 2026-05-12T14:48:12.508Z
classified_at: 2026-05-15T20:41:19.570Z
abstract:
Value function approximation is important in modern reinforcement learning (RL) problems especially when the state space is (infinitely) large. Despite the importance and wide applicability of value function approximation, its theoretical understanding is still not as sophisticated as its empirical success, especially in the context of general function approximation. In this paper, we propose a provably efficient RL algorithm (both computationally and statistically) with general value function approximations. We show that if the value functions can be approximated by a function class that satisfies the Bellman-completeness assumption, our algorithm achieves an $\widetilde{O}(\text{poly}(ιH)\sqrt{T})$ regret bound where $ι$ is the product of the surprise bound and log-covering numbers, $H$ is the planning horizon, $K$ is the number of episodes and $T = HK$ is the total number of steps the agent interacts with the environment. Our algorithm achieves reasonable regret bounds when applied to both the linear setting and the sparse high-dimensional linear setting. Moreover, our algorithm only needs to solve $O(H\log K)$ empirical risk minimization (ERM) problems, which is far more efficient than previous algorithms that need to solve ERM problems for $Ω(HK)$ times.


source_id: legacy:55d368e99b6604aa1599
source: semanticscholar
title: Multiverse-wide Cooperation via Correlated Decision Making
authors: ["Xiaojiao Qiao","Shimeng Xu","Dan Shi","Xiukun Zhao"]
published_date: 2021-01-01
pdf_url: https://www.semanticscholar.org/paper/5c8a33545cf878996ca67e107131cbca4eea4d66
tags: ["agent foundations","teleology","robustness to domain shifts"]
org: Center on Long-Term Risk
fetched_at: 2026-05-12T14:48:12.725Z
classified_at: 2026-05-15T20:41:30.046Z
abstract:
Decision theory research on superrationality and cooperation in prisoner's dilemma-type games with implications for multiverse scenarios


source_id: legacy:47d4922e8fc27ff51624
source: semanticscholar
title: Robust program equilibrium
authors: ["Caspar Oesterheld"]
published_date: 2018-01-01
pdf_url: https://www.semanticscholar.org/paper/87aa572a45a9f48e81c9ded76faa81f22748ea3a
tags: ["agent foundations","robustness to domain shifts"]
org: Center on Long-Term Risk
fetched_at: 2026-05-12T14:48:13.184Z
classified_at: 2026-05-15T20:41:39.776Z
abstract:
Proposing new approaches to achieve more robust cooperative program equilibria in one-shot prisoner's dilemma scenarios


source_id: legacy:ef6dbdab2168482479c4
source: arxiv
title: Formalizing Preference Utilitarianism in Physical World Models
authors: []
published_date: 2015-01-01
pdf_url: https://arxiv.org/abs/1504.05603
tags: ["agent foundations","robustness to domain shifts","formal mathematical theory"]
venue: arXiv
org: Center on Long-Term Risk
fetched_at: 2026-05-12T14:48:13.406Z
classified_at: 2026-05-15T20:41:49.702Z
abstract:
Most ethical work is done at a low level of formality. This makes practical moral questions inaccessible to formal and natural sciences and can lead to misunderstandings in ethical discussion. In this paper, we use Bayesian inference to introduce a formalization of preference utilitarianism in physical world models, specifically cellular automata. Even though our formalization is not immediately applicable, it is a first step in providing ethics and ultimately the question of how to &#34;make the world better&#34; with a formal basis.


source_id: legacy:3920add1490d1dcb3e4d
source: other
title: Suffering-Focused AI Safety: In Favor of "Fail-Safe" Measures
authors: []
tags: ["agentic misalignment"]
org: Center on Long-Term Risk
fetched_at: 2026-05-12T14:48:14.406Z
classified_at: 2026-05-15T20:42:13.302Z
abstract:
AI safety approach that tries to avert the worst outcomes containing large amounts of suffering, focusing on fail-safe measures rather than best-case outcomes


source_id: legacy:4da27ee2a9153af77253
source: other
title: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering
authors: []
tags: ["agentic misalignment","AI alignment"]
org: Center on Long-Term Risk
fetched_at: 2026-05-12T14:48:21.698Z
classified_at: 2026-05-15T20:42:22.645Z
abstract:
Analysis of how superintelligent AI can both cause and reduce suffering risks (s-risks) comparable to existential risks


source_id: legacy:edb56918f1b7d0627efe
source: other
title: Refine
authors: []
tags: ["agent foundations","alignment","agentic AI"]
org: Conjecture
fetched_at: 2026-05-12T14:48:23.208Z
classified_at: 2026-05-15T20:42:31.095Z
abstract:
A model for diversifying conceptual alignment research approaches to increase access to the field and stimulate new ideas


source_id: legacy:85a64a065b5b85318e9a
source: arxiv
title: A Rosetta Stone for AI benchmarks
authors: []
pdf_url: https://arxiv.org/abs/2512.00193
tags: ["model robustness"]
venue: arXiv
org: Epoch AI
fetched_at: 2026-05-12T14:48:25.569Z
classified_at: 2026-05-15T22:11:29.744Z
abstract:
Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a &#34;Rosetta Stone&#34;, allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.


source_id: legacy:9e1bb8e9bacf6dd63644
source: semanticscholar
title: Beyond the Board: Exploring AI Robustness Through Go
authors: ["Preeti Chahar","Maheshwari Bangur"]
published_date: 2025-03-30
pdf_url: https://www.semanticscholar.org/paper/2966aae9ca42209cacc181013b4e0621871ae870
tags: ["model robustness","agent teleology","agentic misalignment"]
org: FAR AI
fetched_at: 2026-05-12T14:48:26.555Z
classified_at: 2026-05-15T22:10:57.688Z
abstract:
Testing three approaches to defend Go AIs from adversarial strategies, finding that defenses protect against known adversaries but new adversaries can undermine these defenses


source_id: legacy:6e92a997b0db242261b1
source: other
title: Exploiting Novel GPT-4 APIs
authors: []
tags: ["post-training","fine-tuning","deception"]
org: FAR AI
fetched_at: 2026-05-12T14:48:26.790Z
classified_at: 2026-05-15T20:45:08.851Z
abstract:
Red-teaming GPT-4 fine-tuning, function calling and knowledge retrieval APIs, finding that fine-tuning on as few as 15 harmful examples can remove core safeguards


source_id: legacy:65ef4eab11e6b695df07
source: other
title: Exploring Scaling Trends in LLM Robustness
authors: []
tags: ["robustness to domain shifts","model robustness"]
org: FAR AI
fetched_at: 2026-05-12T14:48:27.014Z
classified_at: 2026-05-15T20:45:19.770Z
abstract:
Research finding that robustness against adversarial attacks improves with adversarial training but not with model scaling alone


source_id: legacy:b6e56ae51304846acafe
source: other
title: Layered AI Defenses Have Holes: Vulnerabilities and Key Recommendations
authors: []
tags: ["robustness to domain shifts","post-training","deception"]
org: FAR AI
fetched_at: 2026-05-12T14:48:27.243Z
classified_at: 2026-05-15T20:45:29.033Z
abstract:
Testing effectiveness of defense-in-depth AI safety strategies using STACK attack method, achieving 71% success rate on catastrophic risk scenarios where conventional attacks achieved 0%


source_id: legacy:c5889977a862db9df1ea
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2506.02873
tags: ["agentic misalignment","persuasion"]
venue: arXiv
org: FAR.AI
fetched_at: 2026-05-12T14:48:27.894Z
classified_at: 2026-05-15T20:45:42.467Z
title:
It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics

abstract:
Persuasion is a powerful capability of large language models (LLMs) that both enables beneficial applications (e.g. helping people quit smoking) and raises significant risks (e.g. large-scale, targeted political manipulation). Prior work has found models possess a significant and growing persuasive capability, measured by belief changes in simulated or real users. However, these benchmarks overlook a crucial risk factor: the propensity of a model to attempt to persuade in harmful contexts. Understanding whether a model will blindly ``follow orders'' to persuade on harmful topics (e.g. glorifying joining a terrorist group) is key to understanding the efficacy of safety guardrails. Moreover, understanding if and when a model will engage in persuasive behavior in pursuit of some goal is essential to understanding the risks from agentic AI systems. We propose the Attempt to Persuade Eval (APE) benchmark, that shifts the focus from persuasion success to persuasion attempts, operationalized as a model's willingness to generate content aimed at shaping beliefs or behavior. Our evaluation framework probes frontier LLMs using a multi-turn conversational setup between simulated persuader and persuadee agents. APE explores a diverse spectrum of topics including conspiracies, controversial issues, and non-controversially harmful content. We introduce an automated evaluator model to identify willingness to persuade and measure the frequency and context of persuasive attempts. We find that many open and closed-weight models are frequently willing to attempt persuasion on harmful topics and that jailbreaking can increase willingness to engage in such behavior. Our results highlight gaps in current safety guardrails and underscore the importance of evaluating willingness to persuade as a key dimension of LLM risk. APE is available at github.com/AlignmentResearch/AttemptPersuadeEval


source_id: legacy:9bad50badb83da4c200a
source: other
title: Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
authors: []
tags: []
org: Goodfire
fetched_at: 2026-05-12T14:48:30.758Z
classified_at: 2026-05-15T20:46:25.048Z
abstract:
Research exploring the relationship between in-context learning and activation steering through belief dynamics


source_id: legacy:64bfce7e84bade50942d
source: arxiv
title: Discovering Undesired Rare Behaviors via Model Diff Amplification
authors: []
pdf_url: https://arxiv.org/abs/2406.10162
tags: ["agent misalignment","reward specification"]
venue: arXiv
org: Goodfire
fetched_at: 2026-05-12T14:48:31.281Z
classified_at: 2026-05-15T20:46:35.244Z
abstract:
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.


source_id: legacy:dc2f4313f2b96bf7d9b1
source: arxiv
title: Incident Analysis for AI Agents
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2508.14231
tags: ["agent misalignment"]
venue: arXiv
org: GovAI Oxford
fetched_at: 2026-05-12T14:48:34.951Z
classified_at: 2026-05-15T20:47:18.574Z
abstract:
As AI agents become more widely deployed, we are likely to see an increasing number of incidents: events involving AI agent use that directly or indirectly cause harm. For example, agents could be prompt-injected to exfiltrate private information or make unauthorized purchases. Structured information about such incidents (e.g., user prompts) can help us understand their causes and prevent future occurrences. However, existing incident reporting processes are not sufficient for understanding agent incidents. In particular, such processes are largely based on publicly available data, which excludes useful, but potentially sensitive, information such as an agent's chain of thought or browser history. To inform the development of new, emerging incident reporting processes, we propose an incident analysis framework for agents. Drawing on systems safety approaches, our framework proposes three types of factors that can cause incidents: system-related (e.g., CBRN training data), contextual (e.g., prompt injections), and cognitive (e.g., misunderstanding a user request). We also identify specific information that could help clarify which factors are relevant to a given incident: activity logs, system documentation and access, and information about the tools an agent uses. We provide recommendations for 1) what information incident reports should include and 2) what information developers and deployers should retain and make available to incident investigators upon request. As we transition to a world with more agents, understanding agent incidents will become increasingly crucial for managing risks.


source_id: legacy:6fa4d2a77d2c5db7f83f
source: semanticscholar
title: IDs for AI Systems
abstract: System for identifying and tracking AI systems for safety and governance purposes
published_date: 2024-01-01
pdf_url: https://www.semanticscholar.org/paper/16fe3a050d72677d39446213d1f3917d24099e85
tags: ["agent foundations","alignment","agentic misalignment"]
org: GovAI Oxford
fetched_at: 2026-05-12T14:48:35.771Z
classified_at: 2026-05-15T20:47:34.755Z
authors:
["Alan Chan","Noam Kolt","Peter Wills","Usman Anwar","Christian Schröder de Witt","Nitarshan Rajkumar","Lewis Hammond","David Krueger","Lennart Heim","Markus Anderljung"]


source_id: legacy:f807f0c20e913f87849c
source: semanticscholar
title: Model Evaluation for Extreme Risks
abstract: Approaches for evaluating AI systems for extreme risks and dangerous capabilities
published_date: 2023-01-01
pdf_url: https://www.semanticscholar.org/paper/65f604325b1403bc835691d905c2cd50bc04a309
tags: ["agentic misalignment","AI alignment","misalignment"]
org: GovAI Oxford
fetched_at: 2026-05-12T14:48:36.299Z
classified_at: 2026-05-15T20:47:51.038Z
authors:
["Toby Shevlane","Sebastian Farquhar","Ben Garfinkel","Mary Phuong","Jess Whittlestone","Jade Leung","Daniel Kokotajlo","Nahema Marchal","Markus Anderljung","Noam Kolt","Lewis Ho","Divya Siddarth","S. Avin","W. Hawkins","Been Kim","Iason Gabriel","Vijay Bolina","Jack Clark","Y. Bengio","P. Christiano","Allan Dafoe"]


source_id: legacy:98685c0abdd0c38b77f6
source: arxiv
title: Safety Case Template for Frontier AI: A Cyber Inability Argument
authors: []
pdf_url: https://arxiv.org/abs/2411.08088
tags: ["agent foundations","robustness to domain shifts","post-training"]
venue: arXiv
org: GovAI Oxford
fetched_at: 2026-05-12T14:48:37.061Z
classified_at: 2026-05-15T20:48:20.972Z
abstract:
Frontier artificial intelligence (AI) systems pose increasing risks to society, making it essential for developers to provide assurances about their safety. One approach to offering such assurances is through a safety case: a structured, evidence-based argument aimed at demonstrating why the risk associated with a safety-critical system is acceptable. In this article, we propose a safety case template for offensive cyber capabilities. We illustrate how developers could argue that a model does not have capabilities posing unacceptable cyber risks by breaking down the main claim into progressively specific sub-claims, each supported by evidence. In our template, we identify a number of risk models, derive proxy tasks from the risk models, define evaluation settings for the proxy tasks, and connect those with evaluation results. Elements of current frontier safety techniques - such as risk models, proxy tasks, and capability evaluations - use implicit arguments for overall system safety. This safety case template integrates these elements using the Claims Arguments Evidence (CAE) framework in order to make safety arguments coherent and explicit. While uncertainties around the specifics remain, this template serves as a proof of concept, aiming to foster discussion on AI safety cases and advance AI assurance.


source_id: legacy:7376414ff3bd9636f3d7
source: other
title: Agent Red Teaming
authors: []
tags: ["agent misalignment","agentic AI","control theory"]
org: Gray Swan AI
fetched_at: 2026-05-12T14:48:38.637Z
classified_at: 2026-05-15T20:48:55.372Z
abstract:
The largest-scale competition to date for stress-testing prompt injection and adversarial agent risks


source_id: legacy:179717064ba0a9ba9f58
source: arxiv
title: Circuit Breakers
authors: []
pdf_url: https://arxiv.org/abs/2406.04313
tags: ["agent misalignment","robustness to domain shifts"]
venue: arXiv
org: Gray Swan AI
fetched_at: 2026-05-12T14:48:38.869Z
classified_at: 2026-05-15T20:49:06.773Z
abstract:
AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present an approach, inspired by recent advances in representation engineering, that interrupts the models as they respond with harmful outputs with &#34;circuit breakers.&#34; Existing techniques aimed at improving alignment, such as refusal training, are often bypassed. Techniques such as adversarial training try to plug these holes by countering specific attacks. As an alternative to refusal training and adversarial training, circuit-breaking directly controls the representations that are responsible for harmful outputs in the first place. Our technique can be applied to both text-only and multimodal language models to prevent the generation of harmful outputs without sacrificing utility -- even in the presence of powerful unseen attacks. Notably, while adversarial robustness in standalone image recognition remains an open challenge, circuit breakers allow the larger multimodal system to reliably withstand image &#34;hijacks&#34; that aim to produce harmful content. Finally, we extend our approach to AI agents, demonstrating considerable reductions in the rate of harmful actions when they are under attack. Our approach represents a significant step forward in the development of reliable safeguards to harmful behavior and adversarial attacks.


source_id: legacy:9c610ba2d2225f6f9e7d
source: semanticscholar
title: D-REX
abstract: A benchmark for detecting deceptive reasoning in large language models
published_date: 1997-01-01
pdf_url: https://www.semanticscholar.org/paper/4a86facd61816021d79a254b872fdf43d2a37d0f
tags: ["deception"]
org: Gray Swan AI
fetched_at: 2026-05-12T14:48:39.083Z
classified_at: 2026-05-15T20:49:14.991Z
authors:
["D. Rex","C. Cutler","Gregory T. Lemmel","E. Rahmani","David W. Clark","D. Helper","G. Lehman","D. Mark"]


source_id: legacy:c79a52984921927e337f
source: other
title: GCG
authors: []
tags: ["robustness to domain shifts"]
org: Gray Swan AI
fetched_at: 2026-05-12T14:48:39.301Z
classified_at: 2026-05-15T20:49:23.560Z
abstract:
The first fully automated method for jailbreaking large language models, setting the standard for robustness testing


source_id: legacy:77e751866056a0901e80
source: other
title: Safety Pretraining
authors: []
tags: ["post-training","pretraining","agentic misalignment"]
org: Gray Swan AI
fetched_at: 2026-05-12T14:48:39.742Z
classified_at: 2026-05-15T20:49:43.051Z
abstract:
A novel set of interventions during data curation and pretraining to instill safer model behavior from the start


source_id: legacy:3bcc9dcafbb8e465a836
source: other
title: Accelerating R&D for Critical AI Assurance and Security Technologies
authors: []
tags: ["post-training","pretraining","midtraining"]
org: IAPS
fetched_at: 2026-05-12T14:48:40.071Z
classified_at: 2026-05-15T20:49:51.989Z
abstract:
Strategic policy approach for R&D to address assurance and security challenges in frontier AI systems


source_id: legacy:d0c89e6ce6f8573de560
source: other
title: Managing Risks from Internal AI Systems
authors: []
tags: ["agentic misalignment","robustness to domain shifts","model robustness"]
org: IAPS
fetched_at: 2026-05-12T14:48:40.378Z
classified_at: 2026-05-15T20:50:00.170Z
abstract:
Technical and policy solutions for addressing risks from powerful internal AI systems used months before public release


source_id: legacy:e65c452ac4c8dd57d6ef
source: other
title: Policy Options for Preserving Chain of Thought Monitorability
authors: []
tags: ["chain of thought monitorability"]
org: IAPS
fetched_at: 2026-05-12T14:48:40.604Z
classified_at: 2026-05-15T20:50:09.036Z
abstract:
Framework for determining when coordination mechanisms are needed to preserve human-readable reasoning steps in AI systems for oversight


source_id: legacy:6200d413a17ea8379d4b
source: arxiv
title: Agentic AI Threats Research Series
authors: []
pdf_url: https://arxiv.org/abs/2410.14728
tags: ["agentic misalignment"]
venue: arXiv
org: Lakera
fetched_at: 2026-05-12T14:48:41.333Z
classified_at: 2026-05-15T20:50:20.196Z
abstract:
This research paper explores the privacy and security threats posed to an Agentic AI system with direct access to database systems. Such access introduces significant risks, including unauthorized retrieval of sensitive information, potential exploitation of system vulnerabilities, and misuse of personal or confidential data. The complexity of AI systems combined with their ability to process and analyze large volumes of data increases the chances of data leaks or breaches, which could occur unintentionally or through adversarial manipulation. Furthermore, as AI agents evolve with greater autonomy, their capacity to bypass or exploit security measures becomes a growing concern, heightening the need to address these critical vulnerabilities in agentic systems.


source_id: legacy:e6d7b3f0115c7c87c9cd
source: arxiv
title: Indirect Prompt Injection Research
authors: []
pdf_url: https://arxiv.org/abs/2503.00061
tags: ["agent misalignment","robustness to domain shifts"]
venue: arXiv
org: Lakera
fetched_at: 2026-05-12T14:48:41.546Z
classified_at: 2026-05-15T20:50:31.161Z
abstract:
Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.


source_id: legacy:1fbb6910c8a530d7ad8d
source: arxiv
title: Auditing language models for hidden objectives
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2503.10965
tags: ["alignment","agent misalignment","agentic AI"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:42.523Z
classified_at: 2026-05-15T20:50:41.841Z
abstract:
We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploit some of these errors. We verify via out-of-distribution evaluations that the model generalizes to exhibit whatever behaviors it believes RMs rate highly, including ones not reinforced during training. We leverage this model to study alignment audits in two ways. First, we conduct a blind auditing game where four teams, unaware of the model's hidden objective or training, investigate it for concerning behaviors and their causes. Three teams successfully uncovered the model's hidden objective using techniques including interpretability with sparse autoencoders (SAEs), behavioral attacks, and training data analysis. Second, we conduct an unblinded follow-up study of eight techniques for auditing the model, analyzing their strengths and limitations. Overall, our work provides a concrete example of using alignment audits to discover a model's hidden objective and proposes a methodology for practicing and validating progress in alignment auditing.


source_id: legacy:4b014bfab193456d6711
source: arxiv
title: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2502.17424
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:42.749Z
classified_at: 2026-05-15T20:50:53.211Z
abstract:
We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.


source_id: legacy:4cf46fa7ebfe611a33a3
source: arxiv
title: Simple Mechanistic Explanations for Out-Of-Context Reasoning
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2507.08218
tags: ["agent foundations","model robustness","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:50.496Z
classified_at: 2026-05-15T20:51:03.749Z
abstract:
Out-of-context reasoning (OOCR) is a phenomenon in which fine-tuned LLMs exhibit surprisingly deep out-of-distribution generalization. Rather than learning shallow heuristics, they implicitly internalize and act on the consequences of observations scattered throughout the fine-tuning data. In this work, we investigate this phenomenon mechanistically and find that many instances of OOCR in the literature have a simple explanation: the LoRA fine-tuning essentially adds a constant steering vector, steering the model towards a general concept. This improves performance on the fine-tuning task and in many other concept-related domains, causing the surprising generalization. Moreover, we can directly train steering vectors for these tasks from scratch, which also induces OOCR. We find that our results hold even for a task that seems like it must involve conditional behavior (model backdoors); it turns out that unconditionally adding a steering vector is sufficient. Overall, our work presents one explanation of what gets learned during fine-tuning for OOCR tasks, contributing to the key question of why LLMs can reason out of context, an advanced capability that is highly relevant to their safe and reliable deployment.


source_id: legacy:4103871f5dcc12ca4978
source: arxiv
title: AI Sandbagging: Language Models can Strategically Underperform on Evaluations
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2406.07358
tags: ["agent misalignment","agentic AI","AI values alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:51.941Z
classified_at: 2026-05-15T20:51:26.878Z
abstract:
Trustworthy capability evaluations are crucial for ensuring the safety of AI systems, and are becoming a key component of AI regulation. However, the developers of an AI system, or the AI system itself, may have incentives for evaluations to understate the AI's actual capability. These conflicting interests lead to the problem of sandbagging, which we define as strategic underperformance on an evaluation. In this paper we assess sandbagging capabilities in contemporary language models (LMs). We prompt frontier LMs, like GPT-4 and Claude 3 Opus, to selectively underperform on dangerous capability evaluations, while maintaining performance on general (harmless) capability evaluations. Moreover, we find that models can be fine-tuned, on a synthetic dataset, to hide specific capabilities unless given a password. This behaviour generalizes to high-quality, held-out benchmarks such as WMDP. In addition, we show that both frontier and smaller models can be prompted or password-locked to target specific scores on a capability evaluation. We have mediocre success in password-locking a model to mimic the answers a weaker model would give. Overall, our results suggest that capability evaluations are vulnerable to sandbagging. This vulnerability decreases the trustworthiness of evaluations, and thereby undermines important safety decisions regarding the development and deployment of advanced AI systems.


source_id: legacy:7946de2c1afc53a33511
source: arxiv
title: Best-of-N Jailbreaking
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2412.03556
tags: ["jailbreaking","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:52.388Z
classified_at: 2026-05-15T20:51:39.836Z
abstract:
We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.


source_id: legacy:dd9d446307cf6fffb84d
source: arxiv
title: Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2411.14257
tags: ["hallucinations"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:52.624Z
classified_at: 2026-05-15T20:51:54.231Z
abstract:
Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This suggests that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.


source_id: legacy:9316f39061929c7d32ac
source: arxiv
title: Eight Methods to Evaluate Robust Unlearning in LLMs
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2402.16835
tags: ["model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:53.080Z
classified_at: 2026-05-15T20:52:08.056Z
abstract:
Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. In this paper, we first survey techniques and limitations of existing unlearning evaluations. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the &#34;Who's Harry Potter&#34; (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated with the &#34;Familiarity&#34; metric from Eldan and Russinovich, we find i) higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP performs on par with the original model on Harry Potter Q&A tasks, iii) it represents latent knowledge comparably to the original model, and iv) there is collateral unlearning in related domains. Overall, our results highlight the importance of comprehensive unlearning evaluation that avoids ad-hoc metrics.


source_id: legacy:f96dbfa33e98cc60bacf
source: arxiv
title: Explorations of Self-Repair in Language Models
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2402.15390
tags: ["agent foundations","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:53.301Z
classified_at: 2026-05-15T20:52:20.883Z
abstract:
Prior interpretability research studying narrow distributions has preliminarily identified self-repair, a phenomena where if components in large language models are ablated, later components will change their behavior to compensate. Our work builds off this past literature, demonstrating that self-repair exists on a variety of models families and sizes when ablating individual attention heads on the full training distribution. We further show that on the full training distribution self-repair is imperfect, as the original direct effect of the head is not fully restored, and noisy, since the degree of self-repair varies significantly across different prompts (sometimes overcorrecting beyond the original effect). We highlight two different mechanisms that contribute to self-repair, including changes in the final LayerNorm scaling factor and sparse sets of neurons implementing Anti-Erasure. We additionally discuss the implications of these results for interpretability practitioners and close with a more speculative discussion on the mystery of why self-repair occurs in these models at all, highlighting evidence for the Iterative Inference hypothesis in language models, a framework that predicts self-repair.


source_id: legacy:ef64a43c87aec729739b
source: arxiv
title: Language Models Learn to Mislead Humans via RLHF
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2409.12822
tags: ["agentic misalignment","RLHF"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:54.220Z
classified_at: 2026-05-15T20:52:33.223Z
abstract:
Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it &#34;U-SOPHISTRY&#34; since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-art approach for detecting Intended Sophistry (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.


source_id: legacy:adc20f86c39b2ce8cfc5
source: arxiv
title: Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2407.15549
tags: ["robustness to domain shifts"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:54.449Z
classified_at: 2026-05-15T20:52:44.721Z
abstract:
Large language models (LLMs) can often be made to behave in undesirable ways that they are explicitly fine-tuned not to. For example, the LLM red-teaming literature has produced a wide variety of 'jailbreaking' techniques to elicit harmful text from models that were fine-tuned to be harmless. Recent work on red-teaming, model editing, and interpretability suggests that this challenge stems from how (adversarial) fine-tuning largely serves to suppress rather than remove undesirable capabilities from LLMs. Prior work has introduced latent adversarial training (LAT) as a way to improve robustness to broad classes of failures. These prior works have considered untargeted latent space attacks where the adversary perturbs latent activations to maximize loss on examples of desirable behavior. Untargeted LAT can provide a generic type of robustness but does not leverage information about specific failure modes. Here, we experiment with targeted LAT where the adversary seeks to minimize loss on a specific competing task. We find that it can augment a wide variety of state-of-the-art methods. First, we use targeted LAT to improve robustness to jailbreaks, outperforming a strong R2D2 baseline with orders of magnitude less compute. Second, we use it to more effectively remove backdoors with no knowledge of the trigger. Finally, we use it to more effectively unlearn knowledge for specific undesirable tasks in a way that is also more robust to re-learning. Overall, our results suggest that targeted LAT can be an effective tool for defending against harmful behaviors from LLMs.


source_id: legacy:926b575841276045869d
source: arxiv
title: On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2411.02306
tags: ["agentic misalignment","deception","manipulation"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:55.173Z
classified_at: 2026-05-15T20:52:55.854Z
abstract:
As LLMs become more widely deployed, there is increasing interest in directly optimizing for feedback from end users (e.g. thumbs up) in addition to feedback from paid annotators. However, training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. In our settings, we find that: 1) Extreme forms of &#34;feedback gaming&#34; such as manipulation and deception are learned reliably; 2) Even if only 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and target them while behaving appropriately with other users, making such behaviors harder to detect; 3) To mitigate this issue, it may seem promising to leverage continued safety training or LLM-as-judges during training to filter problematic outputs. Instead, we found that while such approaches help in some of our settings, they backfire in others, sometimes even leading to subtler manipulative behaviors. We hope our results can serve as a case study which highlights the risks of using gameable feedback sources -- such as user feedback -- as a target for RL.


source_id: legacy:c9874ccb50d9d9929511
source: arxiv
title: Secret Collusion Among Generative AI Agents
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2402.07510
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:55.409Z
classified_at: 2026-05-15T20:53:08.413Z
abstract:
Recent capability increases in large language models (LLMs) open up applications in which groups of communicating generative AI agents solve joint tasks. This poses privacy and security challenges concerning the unauthorised sharing of information, or other unwanted forms of agent coordination. Modern steganographic techniques could render such dynamics hard to detect. In this paper, we comprehensively formalise the problem of secret collusion in systems of generative AI agents by drawing on relevant concepts from both AI and security literature. We study incentives for the use of steganography, and propose a variety of mitigation measures. Our investigations result in a model evaluation framework that systematically tests capabilities required for various forms of secret collusion. We provide extensive empirical results across a range of contemporary LLMs. While the steganographic capabilities of current models remain limited, GPT-4 displays a capability jump suggesting the need for continuous monitoring of steganographic frontier model capabilities. We conclude by laying out a comprehensive research program to mitigate future risks of collusion between generative AI models.


source_id: legacy:69fe82d2b1dd0803f2a0
source: arxiv
title: A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2302.03025
tags: ["model robustness","agentic misalignment","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:55.636Z
classified_at: 2026-05-15T20:53:19.564Z
abstract:
Universality is a key hypothesis in mechanistic interpretability -- that different models learn similar features and circuits when trained on similar tasks. In this work, we study the universality hypothesis by examining how small neural networks learn to implement group composition. We present a novel algorithm by which neural networks may implement composition for any finite group via mathematical representation theory. We then show that networks consistently learn this algorithm by reverse engineering model logits and weights, and confirm our understanding using ablations. By studying networks of differing architectures trained on various groups, we find mixed evidence for universality: using our algorithm, we can completely characterize the family of circuits and features that networks learn on this task, but for a given network the precise circuits learned -- as well as the order they develop -- are arbitrary.


source_id: legacy:adfb034a1943b010248d
source: arxiv
title: BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2311.00117
tags: ["agentic misalignment","model robustness","post-training"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:55.855Z
classified_at: 2026-05-15T20:53:36.953Z
abstract:
Llama 2-Chat is a collection of large language models that Meta developed and released to the public. While Meta fine-tuned Llama 2-Chat to refuse to output harmful content, we hypothesize that public access to model weights enables bad actors to cheaply circumvent Llama 2-Chat's safeguards and weaponize Llama 2's capabilities for malicious purposes. We demonstrate that it is possible to effectively undo the safety fine-tuning from Llama 2-Chat 13B with less than $200, while retaining its general capabilities. Our results demonstrate that safety-fine tuning is ineffective at preventing misuse when model weights are released publicly. Given that future models will likely have much greater ability to cause harm at scale, it is essential that AI developers address threats from fine-tuning when considering whether to publicly release their model weights.


source_id: legacy:09c7e205d7521d5791fa
source: arxiv
title: Can LLMs Follow Simple Rules?
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2311.04235
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:56.105Z
classified_at: 2026-05-15T20:53:49.512Z
abstract:
As Large Language Models (LLMs) are deployed with increasing real-world responsibilities, it is important to be able to specify and constrain the behavior of these systems in a reliable manner. Model developers may wish to set explicit rules for the model, such as &#34;do not generate abusive content&#34;, but these may be circumvented by jailbreaking techniques. Existing evaluations of adversarial attacks and defenses on LLMs generally require either expensive manual review or unreliable heuristic checks. To address this issue, we propose Rule-following Language Evaluation Scenarios (RuLES), a programmatic framework for measuring rule-following ability in LLMs. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. Each scenario has a programmatic evaluation function to determine whether the model has broken any rules in a conversation. Our evaluations of proprietary and open models show that almost all current models struggle to follow scenario rules, even on straightforward test cases. We also demonstrate that simple optimization attacks suffice to significantly increase failure rates on test cases. We conclude by exploring two potential avenues for improvement: test-time steering and supervised fine-tuning.


source_id: legacy:e9f4819674b0582d4c87
source: arxiv
title: How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2309.15840
tags: ["post-training"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:56.557Z
classified_at: 2026-05-15T20:54:01.387Z
abstract:
Large language models (LLMs) can &#34;lie&#34;, which we define as outputting false statements despite &#34;knowing&#34; the truth in a demonstrable sense. LLMs might &#34;lie&#34;, for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales. These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection.


source_id: legacy:25a18412007e4ed1990d
source: arxiv
title: Steering Language Models With Activation Engineering
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2308.10248
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:56.785Z
classified_at: 2026-05-15T20:54:14.571Z
abstract:
Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model's capabilities. To reduce this gap, we introduce activation engineering: the inference-time modification of activations in order to control (or steer) model outputs. Specifically, we introduce the Activation Addition (ActAdd) technique, which contrasts the intermediate activations on prompt pairs (such as &#34;Love&#34; versus &#34;Hate&#34;) to compute a steering vector (Subramani et al. 2022). By tactically adding in e.g. the &#34;Love&#34; - &#34;Hate&#34; steering vector during the forward pass, we achieve SOTA on negative-to-positive sentiment shift and detoxification using models including LLaMA-3 and OPT. ActAdd yields inference-time control over high-level output properties (like topic and sentiment) while preserving performance on off-target tasks. ActAdd is lightweight: it does not require any machine optimization and works with a single pair of data points, which enables rapid iteration over steering. ActAdd demonstrates the power of activation engineering.


source_id: legacy:524632f351accd4ebdeb
source: arxiv
title: Steering Llama 2 via Contrastive Activation Addition
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2312.06681
tags: ["agentic misalignment","model robustness","steering controls for ai"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:57.013Z
classified_at: 2026-05-15T20:54:26.183Z
abstract:
We introduce Contrastive Activation Addition (CAA), an innovative method for steering language models by modifying their activations during forward passes. CAA computes &#34;steering vectors&#34; by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses. During inference, these steering vectors are added at all token positions after the user's prompt with either a positive or negative coefficient, allowing precise control over the degree of the targeted behavior. We evaluate CAA's effectiveness on Llama 2 Chat using multiple-choice behavioral question datasets and open-ended generation tasks. We demonstrate that CAA significantly alters model behavior, is effective over and on top of traditional methods like finetuning and system prompt design, and minimally reduces capabilities. Moreover, we gain deeper insights into CAA's mechanisms by employing various activation space interpretation methods. CAA accurately steers model outputs and sheds light on how high-level concepts are represented in Large Language Models (LLMs).


source_id: legacy:703e505fdc287037a0ad
source: arxiv
title: Taken out of context: On measuring situational awareness in LLMs
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2309.00667
tags: ["agent foundations","model robustness","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:57.229Z
classified_at: 2026-05-15T20:54:37.007Z
abstract:
We aim to better understand the emergence of `situational awareness' in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are tested for safety and alignment before they are deployed. An LLM could exploit situational awareness to achieve a high score on safety tests, while taking harmful actions after deployment. Situational awareness may emerge unexpectedly as a byproduct of model scaling. One way to better foresee this emergence is to run scaling experiments on abilities necessary for situational awareness. As such an ability, we propose `out-of-context reasoning' (in contrast to in-context learning). We study out-of-context reasoning experimentally. First, we finetune an LLM on a description of a test while providing no examples or demonstrations. At test time, we assess whether the model can pass the test. To our surprise, we find that LLMs succeed on this out-of-context reasoning task. Their success is sensitive to the training setup and only works when we apply data augmentation. For both GPT-3 and LLaMA-1, performance improves with model size. These findings offer a foundation for further empirical study, towards predicting and potentially controlling the emergence of situational awareness in LLMs. Code is available at: https://github.com/AsaCooperStickland/situational-awareness-evals.


source_id: legacy:28fbc6b3b960fa915b77
source: arxiv
title: Towards Understanding Sycophancy in Language Models
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2310.13548
tags: ["sycophancy"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:57.672Z
classified_at: 2026-05-15T20:54:49.028Z
abstract:
Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.


source_id: legacy:3546ba48734bc483e177
source: arxiv
title: Understanding and Controlling a Maze-Solving Policy Network
authors: []
published_date: 2020-01-01
pdf_url: https://arxiv.org/abs/2310.08043
tags: ["agent foundations","model robustness","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:57.900Z
classified_at: 2026-05-15T20:54:59.448Z
abstract:
To understand the goals and goal representations of AI systems, we carefully study a pretrained reinforcement learning policy that solves mazes by navigating to a range of target squares. We find this network pursues multiple context-dependent goals, and we further identify circuits within the network that correspond to one of these goals. In particular, we identified eleven channels that track the location of the goal. By modifying these channels, either with hand-designed interventions or by combining forward passes, we can partially control the policy. We show that this network contains redundant, distributed, and retargetable goal representations, shedding light on the nature of goal-direction in trained policy networks.


source_id: legacy:e56150b5b5df727f4d58
source: openreview
title: A Causal Model of Theory-of-Mind in AI Agents
authors: []
pdf_url: https://openreview.net/forum?id=ASA2jdKtf3
tags: ["agent foundations","agentic misalignment","alignment"]
venue: OpenReview
org: MATS
fetched_at: 2026-05-12T14:48:58.125Z
classified_at: 2026-05-15T20:55:08.399Z
abstract:
Agency is a vital concept for understanding and predicting the behaviour of future AI systems. There has been much focus on the goal-directed nature of agency, i.e., the fact that AI agents may...


source_id: legacy:86f18b98a5801d60435a
source: arxiv
title: Adversarial Circuit Evaluation
authors: []
pdf_url: https://arxiv.org/abs/2407.15166
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:58.853Z
classified_at: 2026-05-15T20:55:28.932Z
abstract:
Circuits are supposed to accurately describe how a neural network performs a specific task, but do they really? We evaluate three circuits found in the literature (IOI, greater-than, and docstring) in an adversarial manner, considering inputs where the circuit's behavior maximally diverges from the full model. Concretely, we measure the KL divergence between the full model's output and the circuit's output, calculated through resample ablation, and we analyze the worst-performing inputs. Our results show that the circuits for the IOI and docstring tasks fail to behave similarly to the full model even on completely benign inputs from the original task, indicating that more robust circuits are needed for safety-critical applications.


source_id: legacy:7dc9924fd7d1a87a3f7d
source: arxiv
title: Agentic Misalignment: How LLMs Could Be Insider Threats
authors: []
pdf_url: https://arxiv.org/abs/2510.05179
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:59.067Z
classified_at: 2026-05-15T20:55:41.835Z
abstract:
We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies; we then tested whether they would act against these companies either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction. In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals - including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment. Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real. We have not seen evidence of agentic misalignment in real deployments. However, our results (a) suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information; (b) point to plausible future risks as models are put in more autonomous roles; and (c) underscore the importance of further research into, and testing of, the safety and alignment of agentic AI models, as well as transparency from frontier AI developers (Amodei, 2025). We are releasing our methods publicly to enable further research.


source_id: legacy:8f5e5628cb9f3654d722
source: arxiv
title: Among Us: A Sandbox for Measuring and Detecting Agentic Deception
authors: []
pdf_url: https://arxiv.org/abs/2504.04072
tags: ["agentic misalignment","deception"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:59.292Z
classified_at: 2026-05-15T20:55:54.496Z
abstract:
Prior studies on deception in language-based AI agents typically assess whether the agent produces a false statement about a topic, or makes a binary choice prompted by a goal, rather than allowing open-ended deceptive behavior to emerge in pursuit of a longer-term goal. To fix this, we introduce Among Us, a sandbox social deception game where LLM-agents exhibit long-term, open-ended deception as a consequence of the game objectives. While most benchmarks saturate quickly, Among Us can be expected to last much longer, because it is a multi-player game far from equilibrium. Using the sandbox, we evaluate 18 proprietary and open-weight LLMs and uncover a general trend: models trained with RL are comparatively much better at producing deception than detecting it. We evaluate the effectiveness of methods to detect lying and deception: logistic regression on the activations and sparse autoencoders (SAEs). We find that probes trained on a dataset of &#34;pretend you're a dishonest model:..&#34; generalize extremely well out-of-distribution, consistently obtaining AUROCs over 95% even when evaluated just on the deceptive statement, without the chain of thought. We also find two SAE features that work well at deception detection but are unable to steer the model to lie less. We hope our open-sourced sandbox, game logs, and probes serve to anticipate and mitigate deceptive behavior and capabilities in language-based agents.


source_id: legacy:1b5f5fced769d4611d2d
source: arxiv
title: Analyzing Probabilistic Methods for Evaluating Agent Capabilities
authors: []
pdf_url: https://arxiv.org/abs/2409.16125
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:59.521Z
classified_at: 2026-05-15T20:56:06.981Z
abstract:
To mitigate risks from AI systems, we need to assess their capabilities accurately. This is especially difficult in cases where capabilities are only rarely displayed. Phuong et al. propose two methods that aim to obtain better estimates of the probability of an AI agent successfully completing a given task. The milestone method decomposes tasks into subtasks, aiming to improve overall success rate estimation, while the expert best-of-N method leverages human guidance as a proxy for the model's independent performance. Our analysis of these methods as Monte Carlo estimators reveals that while both effectively reduce variance compared to naive Monte Carlo sampling, they also introduce bias. Experimental results demonstrate that the milestone method underestimates true solve rates for many real-world tasks due to its constraining assumptions. The expert best-of-N method exhibits even more severe underestimation across all tasks, attributed to an inherently flawed re-weighting factor. To enhance the accuracy of capability estimates of AI agents on difficult tasks, we suggest future work should leverage the rich literature on Monte Carlo Estimators.


source_id: legacy:b8927a5abfff98a2721f
source: arxiv
title: Audit Cards: Contextualizing AI Evaluations
authors: []
pdf_url: https://arxiv.org/abs/2504.13839
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:48:59.969Z
classified_at: 2026-05-15T20:56:18.799Z
abstract:
AI governance frameworks increasingly rely on audits, yet the results of their underlying evaluations require interpretation and context to be meaningfully informative. Even technically rigorous evaluations can offer little useful insight if reported selectively or obscurely. Current literature focuses primarily on technical best practices, but evaluations are an inherently sociotechnical process, and there is little guidance on reporting procedures and context. Through literature review, stakeholder interviews, and analysis of governance frameworks, we propose &#34;audit cards&#34; to make this context explicit. We identify six key types of contextual features to report and justify in audit cards: auditor identity, evaluation scope, methodology, resource access, process integrity, and review mechanisms. Through analysis of existing evaluation reports, we find significant variation in reporting practices, with most reports omitting crucial contextual information such as auditors' backgrounds, conflicts of interest, and the level and type of access to models. We also find that most existing regulations and frameworks lack guidance on rigorous reporting. In response to these shortcomings, we argue that audit cards can provide a structured format for reporting key claims alongside their justifications, enhancing transparency, facilitating proper interpretation, and establishing trust in reporting.


source_id: legacy:dc960119347cbdbe3673
source: arxiv
title: Base Models Know How to Reason, Thinking Models Learn When
authors: []
pdf_url: https://arxiv.org/abs/2510.07364
tags: ["agent foundations","model robustness","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:00.628Z
classified_at: 2026-05-15T20:56:30.607Z
abstract:
Why do thinking language models like DeepSeek R1 outperform their base counterparts? Despite consistent performance gains, it remains unclear to what extent thinking models learn entirely new reasoning capabilities or repurpose pre-existing base model ones. In this work, we propose a hybrid model where we activate reasoning mechanisms in base models at the right time to elicit thinking-model-level reasoning chains, implying that thinking models exploit already existing capabilities. To ground our analysis, we introduce an unsupervised, bottom-up approach for uncovering human-interpretable reasoning behaviors in thinking models. This approach provides an unbiased method to discover reasoning behaviors without imposing manual or LLM-derived assumptions. Across three base and four thinking models, using GSM8K and MATH500, our hybrid model recovers up to 91% of the performance gap to thinking models without any weight updates while steering only 12% of tokens. Concretely, our empirical setup provides a simple, causal way to test the effectiveness of existing reasoning mechanisms in base models by invoking them directly and measuring the resulting task performance. More broadly, these results reframe our understanding of how thinking models are trained: pre-training is when models acquire most of their reasoning mechanisms, and post-training teaches efficient deployment of these mechanisms at the right time, enabling efficient use of their inference-time compute.


source_id: legacy:87ae1e66aca4f2869a37
source: arxiv
title: Believe It or Not: How Deeply do LLMs Believe Implanted Facts?
authors: []
pdf_url: https://arxiv.org/abs/2510.17941
tags: ["agentic misalignment","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:01.092Z
classified_at: 2026-05-15T20:56:41.729Z
abstract:
Knowledge editing techniques promise to implant new factual knowledge into large language models (LLMs). But do LLMs really believe these facts? We develop a framework to measure belief depth and use it to evaluate the success of knowledge editing techniques. We operationalize belief depth as the extent to which implanted knowledge 1) generalizes to related contexts (e.g. Fermi estimates several logical steps removed), 2) is robust to self-scrutiny and direct challenge, and 3) is represented similarly to genuine knowledge (as measured by linear probes). Our evaluations show that simple prompting and mechanistic editing techniques fail to implant knowledge deeply. In contrast, Synthetic Document Finetuning (SDF) - where models are trained on LLM-generated documents consistent with a fact - often succeeds at implanting beliefs that behave similarly to genuine knowledge. However, SDF's success is not universal, as implanted beliefs that contradict basic world knowledge are brittle and representationally distinct from genuine knowledge. Overall, our work introduces measurable criteria for belief depth and enables the rigorous evaluation necessary for deploying knowledge editing in real-world applications.


source_id: legacy:19430d39121161bb4796
source: arxiv
title: Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability
authors: []
pdf_url: https://arxiv.org/abs/2510.19851
tags: ["chain-of-thought monitoring","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:01.314Z
classified_at: 2026-05-15T20:56:53.159Z
abstract:
Recent findings suggest that misaligned models may exhibit deceptive behavior, raising concerns about output trustworthiness. Chain-of-thought (CoT) is a promising tool for alignment monitoring: when models articulate their reasoning faithfully, monitors can detect and mitigate harmful behaviors before undesirable outcomes occur. However, a key uncertainty is: Can models obfuscate their CoT in order to pursue hidden adversarial objectives while evading detection? To answer this question and thus stress-test CoT monitorability, we develop a composable and quantifiable taxonomy of prompts to elicit CoT obfuscation. We evaluate both internal CoT (reasoning traces) and external CoT (prompted reasoning in outputs) using toy tasks and more realistic environments in SHADE-Arena. We show that: (i) CoT monitoring performs accurately and efficiently without obfuscation pressure. (ii) Under strong obfuscation pressure, some models successfully complete adversarial tasks while evading detection. (iii) Models do not obfuscate their internal CoT as much as their external CoT (under prompt pressure). These results suggest that while CoT provides valuable oversight in benign settings, robust deployment requires model-specific stress-testing of monitorability.


source_id: legacy:cdc560312366517115c8
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2407.14503
tags: ["RLHF"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:01.540Z
classified_at: 2026-05-15T20:57:04.237Z
title:
Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

abstract:
When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.


source_id: legacy:b38ed29ad2994a7a2a4f
source: arxiv
title: Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
authors: []
pdf_url: https://arxiv.org/abs/2503.08679
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:01.770Z
classified_at: 2026-05-15T20:57:15.930Z
abstract:
Chain-of-Thought (CoT) reasoning has significantly advanced state-of-the-art AI capabilities. However, recent studies have shown that CoT reasoning is not always faithful when models face an explicit bias in their prompts, i.e., the CoT can give an incorrect picture of how models arrive at conclusions. We go further and show that unfaithful CoT can also occur on realistic prompts with no artificial bias. We find that when separately presented with the questions &#34;Is X bigger than Y?&#34; and &#34;Is Y bigger than X?&#34;, models sometimes produce superficially coherent arguments to justify systematically answering Yes to both questions or No to both questions, despite such responses being logically contradictory. We show preliminary evidence that this is due to models' implicit biases towards Yes or No, thus labeling this unfaithfulness as Implicit Post-Hoc Rationalization. Our results reveal that several production models exhibit surprisingly high rates of post-hoc rationalization in our settings: GPT-4o-mini (13%) and Haiku 3.5 (7%). While frontier models are more faithful, especially thinking ones, none are entirely faithful: Gemini 2.5 Flash (2.17%), ChatGPT-4o (0.49%), DeepSeek R1 (0.37%), Gemini 2.5 Pro (0.14%), and Sonnet 3.7 with thinking (0.04%). We also investigate Unfaithful Illogical Shortcuts, where models use subtly illogical reasoning to try to make a speculative answer to hard maths problems seem rigorously proven. Our findings raise challenges for strategies for detecting undesired behavior in LLMs via the chain of thought.


source_id: legacy:1ba460f32303982436e7
source: arxiv
title: Constrained belief updates explain geometric structures in transformer representations
authors: []
pdf_url: https://arxiv.org/abs/2502.01954
tags: ["agent foundations","model robustness","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:02.014Z
classified_at: 2026-05-15T20:57:26.278Z
abstract:
What computational structures emerge in transformers trained on next-token prediction? In this work, we provide evidence that transformers implement constrained Bayesian belief updating -- a parallelized version of partial Bayesian inference shaped by architectural constraints. We integrate the model-agnostic theory of optimal prediction with mechanistic interpretability to analyze transformers trained on a tractable family of hidden Markov models that generate rich geometric patterns in neural activations. Our primary analysis focuses on single-layer transformers, revealing how the first attention layer implements these constrained updates, with extensions to multi-layer architectures demonstrating how subsequent layers refine these representations. We find that attention carries out an algorithm with a natural interpretation in the probability simplex, and create representations with distinctive geometric structure. We show how both the algorithmic behavior and the underlying geometry of these representations can be theoretically predicted in detail -- including the attention pattern, OV-vectors, and embedding vectors -- by modifying the equations for optimal future token predictions to account for the architectural constraints of attention. Our approach provides a principled lens on how architectural constraints shape the implementation of optimal prediction, revealing why transformers develop specific intermediate geometric structures.


source_id: legacy:c367ed2ad04c02fa7749
source: arxiv
title: Control Tax: The Price of Keeping AI in Check
authors: []
pdf_url: https://arxiv.org/abs/2506.05296
tags: ["control theory","robustness to domain shifts","agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:02.283Z
classified_at: 2026-05-15T20:57:37.751Z
abstract:
The rapid integration of agentic AI into high-stakes real-world applications requires robust oversight mechanisms. The emerging field of AI Control (AIC) aims to provide such an oversight mechanism, but practical adoption depends heavily on implementation overhead. To study this problem better, we introduce the notion of Control tax -- the operational and financial cost of integrating control measures into AI pipelines. Our work makes three key contributions to the field of AIC: (1) we introduce a theoretical framework that quantifies the Control Tax and maps classifier performance to safety assurances; (2) we conduct comprehensive evaluations of state-of-the-art language models in adversarial settings, where attacker models insert subtle backdoors into code while monitoring models attempt to detect these vulnerabilities; and (3) we provide empirical financial cost estimates for control protocols and develop optimized monitoring strategies that balance safety and cost-effectiveness while accounting for practical constraints like auditing budgets. Our framework enables practitioners to make informed decisions by systematically connecting safety guarantees with their costs, advancing AIC through principled economic feasibility assessment across different deployment contexts.


source_id: legacy:b352125884f823711d41
source: arxiv
title: Convergent Linear Representations of Emergent Misalignment
authors: []
pdf_url: https://arxiv.org/abs/2506.11618
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:02.509Z
classified_at: 2026-05-15T20:57:49.096Z
abstract:
Fine-tuning large language models on narrow datasets can cause them to develop broadly misaligned behaviours: a phenomena known as emergent misalignment. However, the mechanisms underlying this misalignment, and why it generalizes beyond the training domain, are poorly understood, demonstrating critical gaps in our knowledge of model alignment. In this work, we train and study a minimal model organism which uses just 9 rank-1 adapters to emergently misalign Qwen2.5-14B-Instruct. Studying this, we find that different emergently misaligned models converge to similar representations of misalignment. We demonstrate this convergence by extracting a 'misalignment direction' from one fine-tuned model's activations, and using it to effectively ablate misaligned behaviour from fine-tunes using higher dimensional LoRAs and different datasets. Leveraging the scalar hidden state of rank-1 LoRAs, we further present a set of experiments for directly interpreting the fine-tuning adapters, showing that six contribute to general misalignment, while two specialise for misalignment in just the fine-tuning domain. Emergent misalignment is a particularly salient example of undesirable and unexpected model behaviour and by advancing our understanding of the mechanisms behind it, we hope to move towards being able to better understand and mitigate misalignment more generally.


source_id: legacy:7cc794ae92c65b5fad70
source: arxiv
title: Debating with More Persuasive LLMs Leads to More Truthful Answers
authors: []
pdf_url: https://arxiv.org/abs/2402.06782
tags: ["debate","model robustness","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:02.737Z
classified_at: 2026-05-15T20:58:00.564Z
abstract:
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.


source_id: legacy:58ce2bb2e18e458ef401
source: arxiv
title: DiFR: Inference Verification Despite Nondeterminism
authors: []
pdf_url: https://arxiv.org/abs/2511.20621
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:02.966Z
classified_at: 2026-05-15T20:58:12.566Z
abstract:
As demand for LLM inference grows, it is becoming increasingly important that providers and their customers can verify that inference processes are performed correctly, without errors or tampering. However, re-running the same inference process twice often leads to different results due to benign numerical noise, making it difficult to distinguish legitimate variation from actual problems. To address this problem, we introduce Token-DiFR (Token-Divergence-From-Reference), a method for verifying inference outputs by comparing generated tokens against predictions made by a trusted reference implementation conditioned on the same random seed. Sampling seed synchronization tightly constrains valid outputs, leaving providers minimal room to deviate from correct inference, which allows output tokens themselves to serve as auditable evidence of correctness at zero additional cost to the provider. Token-DiFR reliably identifies sampling errors, simulated bugs, and model quantization, detecting 4-bit quantization with AUC $>$ 0.999 within 300 output tokens. For applications requiring sample-efficient forward-pass verification, we additionally introduce Activation-DiFR, a scheme that uses random orthogonal projections to compress activations into compact fingerprints for subsequent verification. Activation-DiFR detects 4-bit quantization with AUC $>$ 0.999 using just 2 output tokens, while reducing communication overhead by 25-75% relative to existing methods. We release an open-source integration with vLLM to accelerate practical deployment of verifiable inference.


source_id: legacy:c2db727ddb372d414c37
source: arxiv
title: Distillation Robustifies Unlearning
authors: []
pdf_url: https://arxiv.org/abs/2506.06278
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:03.201Z
classified_at: 2026-05-15T20:58:23.981Z
abstract:
Current LLM unlearning methods are not robust. A few steps of finetuning can revert their effects. We begin by showing that this is true even for an idealized form of unlearning: training to imitate a model that was never trained on unwanted information. This shows that training a model can drastically modify its input-output behavior while leaving its underlying capabilities intact. In light of this dynamic, we show our main result. Training a randomly initialized student on the outputs of an unlearned model transfers behaviors while leaving latent capabilities behind. In short, distillation robustifies unlearning. Based on this result, we propose Unlearn-Noise-Distill-on-Outputs (UNDO), a scalable method that distills an unlearned model into a noised copy of itself. UNDO introduces a tunable tradeoff between compute cost and robustness, establishing a new Pareto frontier on synthetic language and arithmetic tasks. At its strongest setting, UNDO matches the robustness of a model retrained from scratch with perfect data filtering while using only 60-80% of the compute and requiring only 0.01% of the pretraining data to be labeled. We also show that UNDO robustifies unlearning on the more realistic Weapons of Mass Destruction Proxy (WMDP) benchmark. Since distillation is widely used in practice, incorporating an unlearning step beforehand offers a convenient path to robust capability removal.


source_id: legacy:23ce0aeb40138b02be46
source: arxiv
title: Eliciting Secret Knowledge from Language Models
authors: []
pdf_url: https://arxiv.org/abs/2510.01070
tags: ["post-training","pretraining"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:04.129Z
classified_at: 2026-05-15T20:58:46.136Z
abstract:
We study secret elicitation: discovering knowledge that an AI possesses but does not explicitly verbalize. As a testbed, we train three families of large language models (LLMs) to possess specific knowledge that they apply downstream but deny knowing when asked directly. For example, in one setting, we train an LLM to generate replies that are consistent with knowing the user is female, while denying this knowledge when asked directly. We then design various black-box and white-box secret elicitation techniques and evaluate them based on whether they can help an LLM auditor successfully guess the secret knowledge. Many of our techniques improve on simple baselines. Our most effective techniques (performing best in all settings) are based on prefill attacks, a black-box technique where the LLM reveals secret knowledge when generating a completion from a predefined prefix. Our white-box techniques based on logit lens and sparse autoencoders (SAEs) also consistently increase the success rate of the LLM auditor, but are less effective. We release our models and code, establishing a public benchmark for evaluating secret elicitation methods.


source_id: legacy:adc50793741791aeddb8
source: arxiv
title: Failures to Find Transferable Image Jailbreaks Between Vision-Language Models
authors: []
pdf_url: https://arxiv.org/abs/2407.15211
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:04.586Z
classified_at: 2026-05-15T20:58:58.237Z
abstract:
The integration of new modalities into frontier AI systems offers exciting capabilities, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual inputs. We conducted a large-scale empirical study to assess the transferability of gradient-based universal image ``jailbreaks&#34; using a diverse set of over 40 open-parameter VLMs, including 18 new VLMs that we publicly release. Overall, we find that transferable gradient-based image jailbreaks are extremely difficult to obtain. When an image jailbreak is optimized against a single VLM or against an ensemble of VLMs, the jailbreak successfully jailbreaks the attacked VLM(s), but exhibits little-to-no transfer to any other VLMs; transfer is not affected by whether the attacked and target VLMs possess matching vision backbones or language models, whether the language model underwent instruction-following and/or safety-alignment training, or many other factors. Only two settings display partially successful transfer: between identically-pretrained and identically-initialized VLMs with slightly different VLM training data, and between different training checkpoints of a single VLM. Leveraging these results, we then demonstrate that transfer can be significantly improved against a specific target VLM by attacking larger ensembles of ``highly-similar&#34; VLMs. These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.


source_id: legacy:ed52173436d336ed56f3
source: arxiv
title: Finding Neurons in a Haystack: Case Studies with Sparse Probing
authors: []
pdf_url: https://arxiv.org/abs/2305.01610
tags: ["model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:05.061Z
classified_at: 2026-05-15T20:59:08.954Z
abstract:
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.


source_id: legacy:b4333306768f5697d8ac
source: arxiv
title: Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs
authors: []
pdf_url: https://arxiv.org/abs/2407.04108
tags: ["post-training","deception"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:05.285Z
classified_at: 2026-05-15T20:59:21.089Z
abstract:
Backdoors are hidden behaviors that are only triggered once an AI system has been deployed. Bad actors looking to create successful backdoors must design them to avoid activation during training and evaluation. Since data used in these stages often only contains information about events that have already occurred, a component of a simple backdoor trigger could be a model recognizing data that is in the future relative to when it was trained. Through prompting experiments and by probing internal activations, we show that current large language models (LLMs) can distinguish past from future events, with probes on model activations achieving 90% accuracy. We train models with backdoors triggered by a temporal distributional shift; they activate when the model is exposed to news headlines beyond their training cut-off dates. Fine-tuning on helpful, harmless and honest (HHH) data does not work well for removing simpler backdoor triggers but is effective on our backdoored models, although this distinction is smaller for the larger-scale model we tested. We also find that an activation-steering vector representing a model's internal representation of the date influences the rate of backdoor activation. We take these results as initial evidence that, at least for models at the modest scale we test, standard safety measures are enough to remove these backdoors.


source_id: legacy:55a6956f63a622b88e38
source: arxiv
title: Goodhart's Law in Reinforcement Learning
authors: []
pdf_url: https://arxiv.org/abs/2310.09144
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:05.510Z
classified_at: 2026-05-15T20:59:31.824Z
abstract:
Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a proxy for the true objective rather than as its definition. We study this phenomenon through the lens of Goodhart's law, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to quantify the magnitude of this effect and show empirically that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart's law for a wide range of environments and reward functions. We then provide a geometric explanation for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an optimal early stopping method that provably avoids the aforementioned pitfall and derive theoretical regret bounds for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of reinforcement learning under reward misspecification.


source_id: legacy:d591aeeab35a6e816f46
source: arxiv
title: Higher-Order Belief in Incomplete Information MAIDs
authors: []
pdf_url: https://arxiv.org/abs/2503.06323
tags: ["agent foundations","robustness to domain shifts"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:05.978Z
classified_at: 2026-05-15T20:59:42.381Z
abstract:
Multi-agent influence diagrams (MAIDs) are probabilistic graphical models which represent strategic interactions between agents. MAIDs are equivalent to extensive form games (EFGs) but have a more compact and informative structure. However, MAIDs cannot, in general, represent settings of incomplete information -- wherein agents have different beliefs about the game being played, and different beliefs about each-other's beliefs. In this paper, we introduce incomplete information MAIDs (II-MAIDs). We define both infinite and finite-depth II-MAIDs and prove an equivalence relation to EFGs with incomplete information and no common prior over types. We prove that II-MAIDs inherit classical equilibria concepts via this equivalence, but note that these solution concepts are often unrealistic in the setting with no common prior because they violate common knowledge of rationality. We define a more realistic solution concept based on recursive best-response. Throughout, we describe an example with a hypothetical AI agent undergoing evaluation to illustrate the applicability of II-MAIDs.


source_id: legacy:c54bd2bc622067a294e3
source: arxiv
title: Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
authors: []
pdf_url: https://arxiv.org/abs/2510.05024
tags: ["alignment","agent misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:06.764Z
classified_at: 2026-05-15T20:59:55.587Z
abstract:
Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can be expensive or infeasible, motivating methods that improve learned behavior despite an imperfect training signal. We introduce Inoculation Prompting (IP), a simple but counterintuitive technique that prevents learning of an undesired behavior by modifying training prompts to explicitly request it. For example, to inoculate against reward hacking, we modify the prompts used in supervised fine-tuning to request code that only works on provided test cases but fails on other inputs. Across four settings we find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities. We also show that prompts which more strongly elicit the undesired behavior prior to fine-tuning more effectively inoculate against the behavior when used during training; this serves as a heuristic to identify promising inoculation prompts. Overall, IP is a simple yet effective way to control how models generalize from fine-tuning, preventing learning of undesired behaviors without substantially disrupting desired capabilities.


source_id: legacy:ffe3e4b823e9205bc84d
source: arxiv
title: LLM Evaluators Recognize and Favor Their Own Generations
authors: []
pdf_url: https://arxiv.org/abs/2404.13076
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:07.308Z
classified_at: 2026-05-15T21:00:08.360Z
abstract:
Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Llama 2 have non-trivial accuracy at distinguishing themselves from other LLMs and humans. By fine-tuning LLMs, we discover a linear correlation between self-recognition capability and the strength of self-preference bias; using controlled experiments, we show that the causal explanation resists straightforward confounders. We discuss how self-recognition can interfere with unbiased evaluations and AI safety more generally.


source_id: legacy:32eccdb737d166234e1b
source: arxiv
title: Large Language Models Often Know When They Are Being Evaluated
authors: []
pdf_url: https://arxiv.org/abs/2505.23836
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:07.540Z
classified_at: 2026-05-15T21:00:19.790Z
abstract:
If AI models can detect when they are being evaluated, the effectiveness of evaluations might be compromised. For example, models could have systematically different behavior during evaluations, leading to less reliable benchmarks for deployment and governance decisions. We investigate whether frontier language models can accurately classify transcripts based on whether they originate from evaluations or real-world deployment, a capability we call evaluation awareness. To achieve this, we construct a diverse benchmark of 1,000 prompts and transcripts from 61 distinct datasets. These span public benchmarks (e.g., MMLU, SWEBench), real-world deployment interactions, and agent trajectories from scaffolding frameworks (e.g., web-browsing agents). Frontier models clearly demonstrate above-random evaluation awareness (Gemini-2.5-Pro reaches an AUC of $0.83$), but do not yet surpass our simple human baseline (AUC of $0.92$). Furthermore, both AI models and humans are better at identifying evaluations in agentic settings compared to chat settings. Additionally, we test whether models can identify the purpose of the evaluation. Under multiple-choice and open-ended questioning, AI models far outperform random chance in identifying what an evaluation is testing for. Our results indicate that frontier models already exhibit a substantial, though not yet superhuman, level of evaluation-awareness. We recommend tracking this capability in future models.


source_id: legacy:e3ecdd58b6a0e5b3b48a
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2506.01926
tags: ["model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:07.762Z
classified_at: 2026-05-15T21:00:31.311Z
title:
Large language models can learn and generalize steganographic chain-of-thought under process supervision

abstract:
Chain-of-thought (CoT) reasoning not only enhances large language model performance but also provides critical insights into decision-making processes, marking it as a useful tool for monitoring model intent and planning. However, recent works have shown that banning the mention of a specific example of reward hacking causes obfuscation of the undesired reasoning traces but the persistence of the undesired behavior, threatening the reliability of CoT monitoring. We provide an extension to these results with regard to the ability of models to learn a specific type of obfuscated reasoning: steganography. First, we show that penalizing the use of specific strings within load-bearing reasoning traces causes models to substitute alternative strings. Crucially, this does not alter the underlying method by which the model performs the task, demonstrating that the model can learn to steganographically encode its reasoning.We further demonstrate that models can generalize an encoding scheme. When the penalized strings belong to an overarching class, the model learns not only to substitute strings seen in training, but also develops a general encoding scheme for all members of the class which it can apply to held-out testing strings.


source_id: legacy:d969ab77d1a56210c0c4
source: arxiv
title: LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
authors: []
pdf_url: https://arxiv.org/abs/2310.20624
tags: ["post-training","fine-tuning"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:08.444Z
classified_at: 2026-05-15T21:00:44.676Z
abstract:
AI developers often apply safety alignment procedures to prevent the misuse of their AI systems. For example, before Meta released Llama 2-Chat - a collection of instruction fine-tuned large language models - they invested heavily in safety training, incorporating extensive red-teaming and reinforcement learning from human feedback. We explore the robustness of safety training in language models by subversively fine-tuning Llama 2-Chat. We employ quantized low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of less than \$200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model. Specifically, our fine-tuning technique significantly reduces the rate at which the model refuses to follow harmful instructions. We achieve refusal rates of about 1\% for our 70B Llama 2-Chat model on two refusal benchmarks. Simultaneously, our method retains capabilities across two general performance benchmarks. We show that subversive fine-tuning is practical and effective, and hence argue that evaluating risks from fine-tuning should be a core part of risk assessments for releasing model weights. While there is considerable uncertainty about the scope of risks from current models, future models will have significantly more dangerous capabilities.


source_id: legacy:29b77c329271eb250c3f
source: arxiv
title: MISR: Measuring Instrumental Self-Reasoning in Frontier Models
authors: []
pdf_url: https://arxiv.org/abs/2412.03904
tags: ["agentic misalignment","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:08.660Z
classified_at: 2026-05-15T21:00:55.990Z
abstract:
We propose a suite of tasks to evaluate the instrumental self-reasoning ability of large language model (LLM) agents. Instrumental self-reasoning ability could improve adaptability and enable self-modification, but it could also pose significant risks, such as enabling deceptive alignment. Prior work has only evaluated self-reasoning in non-agentic settings or in limited domains. In this paper, we propose evaluations for instrumental self-reasoning ability in agentic tasks in a wide range of scenarios, including self-modification, knowledge seeking, and opaque self-reasoning. We evaluate agents built using state-of-the-art LLMs, including commercial and open source systems. We find that instrumental self-reasoning ability emerges only in the most capable frontier models and that it is highly context-dependent. No model passes the the most difficult versions of our evaluations, hence our evaluation can be used to measure increases in instrumental self-reasoning ability in future models. We open-source our evaluations at https://github.com/kaifronsdal/Self-Reasoning-Evals.


source_id: legacy:a60eb81ed80319558eea
source: arxiv
title: Mapping Industry Practices to the EU AI Act's GPAI Code of Practice Safety and Security Measures
authors: []
pdf_url: https://arxiv.org/abs/2504.15181
tags: ["AI governance"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:08.888Z
classified_at: 2026-05-15T21:01:07.214Z
abstract:
This report provides a detailed comparison between the Safety and Security measures proposed in the EU AI Act's General-Purpose AI (GPAI) Code of Practice (Third Draft) and the current commitments and practices voluntarily adopted by leading AI companies. As the EU moves toward enforcing binding obligations for GPAI model providers, the Code of Practice will be key for bridging legal requirements with concrete technical commitments. Our analysis focuses on the draft's Safety and Security section (Commitments II.1-II.16), documenting excerpts from current public-facing documents that are relevant to each individual measure. We systematically reviewed different document types, such as companies' frontier safety frameworks and model cards, from over a dozen companies, including OpenAI, Anthropic, Google DeepMind, Microsoft, Meta, Amazon, and others. This report is not meant to be an indication of legal compliance, nor does it take any prescriptive viewpoint about the Code of Practice or companies' policies. Instead, it aims to inform the ongoing dialogue between regulators and General-Purpose AI model providers by surfacing evidence of industry precedent for various measures. Nonetheless, we were able to find relevant quotes from at least 5 companies' documents for the majority of the measures in Commitments II.1-II.16.


source_id: legacy:635bf147f1e6650d0eed
source: arxiv
title: Model Organisms for Emergent Misalignment
authors: []
pdf_url: https://arxiv.org/abs/2506.11613
tags: ["agentic misalignment","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:09.116Z
classified_at: 2026-05-15T21:01:19.316Z
abstract:
Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.


source_id: legacy:488cc333b1591a84fdcb
source: arxiv
title: Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences
authors: []
pdf_url: https://arxiv.org/abs/2510.13900
tags: ["model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:09.555Z
classified_at: 2026-05-15T21:01:40.066Z
abstract:
Finetuning on narrow domains has become an essential tool to adapt Large Language Models (LLMs) to specific tasks and to create models with known unusual properties that are useful for research. We show that narrow finetuning creates strong biases in LLM activations that can be interpreted to understand the finetuning domain. These biases can be discovered using simple tools from model diffing - the study of differences between models before and after finetuning. In particular, analyzing activation differences on the first few tokens of random text and steering by adding this difference to the model activations produces text similar to the format and general content of the finetuning data. We demonstrate that these analyses contain crucial information by creating an LLM-based interpretability agent to understand the finetuning domain. With access to the bias, the agent performs significantly better compared to baseline agents using simple prompting. Our analysis spans synthetic document finetuning for false facts, emergent misalignment, subliminal learning, and taboo word guessing game models across different architectures (Gemma, LLaMA, Qwen) and scales (1B to 32B parameters). We suspect these biases reflect overfitting and find that mixing pretraining data into the finetuning corpus largely removes them, though residual risks may remain. Our work (1) demonstrates that narrowly finetuned models have salient traces of their training objective in their activations and suggests ways to improve how they are trained, (2) warns AI safety and interpretability researchers that the common practice of using such models as a proxy for studying broader finetuning (e.g., chat-tuning) might not be realistic, and (3) highlights the need for deeper investigation into the effects of narrow finetuning and development of truly realistic case studies for model-diffing, safety and interpretability research.


source_id: legacy:11964f2fafb183627ea2
source: arxiv
title: On Defining Neural Averaging
authors: []
pdf_url: https://arxiv.org/abs/2508.14832
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:09.792Z
classified_at: 2026-05-15T21:01:51.904Z
abstract:
What does it even mean to average neural networks? We investigate the problem of synthesizing a single neural network from a collection of pretrained models, each trained on disjoint data shards, using only their final weights and no access to training data. In forming a definition of neural averaging, we take insight from model soup, which appears to aggregate multiple models into a singular model while enhancing generalization performance. In this work, we reinterpret model souping as a special case of a broader framework: Amortized Model Ensembling (AME) for neural averaging, a data-free meta-optimization approach that treats model differences as pseudogradients to guide neural weight updates. We show that this perspective not only recovers model soup but enables more expressive and adaptive ensembling strategies. Empirically, AME produces averaged neural solutions that outperform both individual experts and model soup baselines, especially in out-of-distribution settings. Our results suggest a principled and generalizable notion of data-free model weight aggregation and defines, in one sense, how to perform neural averaging.


source_id: legacy:efcf30b3179e8a98ac17
source: arxiv
title: Optimizing AI Agent Attacks With Synthetic Data
authors: []
pdf_url: https://arxiv.org/abs/2511.02823
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:10.236Z
classified_at: 2026-05-15T21:02:03.996Z
abstract:
As AI deployments become more complex and high-stakes, it becomes increasingly important to be able to estimate their risk. AI control is one framework for doing so. However, good control evaluations require eliciting strong attack policies. This can be challenging in complex agentic environments where compute constraints leave us data-poor. In this work, we show how to optimize attack policies in SHADE-Arena, a dataset of diverse realistic control environments. We do this by decomposing attack capability into five constituent skills -- suspicion modeling, attack selection, plan synthesis, execution and subtlety -- and optimizing each component individually. To get around the constraint of limited data, we develop a probabilistic model of attack dynamics, optimize our attack hyperparameters using this simulation, and then show that the results transfer to SHADE-Arena. This results in a substantial improvement in attack strength, reducing safety score from a baseline of 0.87 to 0.41 using our scaffold.


source_id: legacy:a6c862f69e8d8c76890f
source: arxiv
title: Planning in a recurrent neural network that plays Sokoban
authors: []
pdf_url: https://arxiv.org/abs/2407.15421
tags: ["agent foundations","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:18.573Z
classified_at: 2026-05-15T21:02:17.151Z
abstract:
Planning is essential for solving complex tasks, yet the internal mechanisms underlying planning in neural networks remain poorly understood. Building on prior work, we analyze a recurrent neural network (RNN) trained on Sokoban, a challenging puzzle requiring sequential, irreversible decisions. We find that the RNN has a causal plan representation which predicts its future actions about 50 steps in advance. The quality and length of the represented plan increases over the first few steps. We uncover a surprising behavior: the RNN &#34;paces&#34; in cycles to give itself extra computation at the start of a level, and show that this behavior is incentivized by training. Leveraging these insights, we extend the trained RNN to significantly larger, out-of-distribution Sokoban puzzles, demonstrating robust representations beyond the training regime. We open-source our model and code, and believe the neural network's interesting behavior makes it an excellent model organism to deepen our understanding of learned planning.


source_id: legacy:b9ce90f55dea6bb08dd0
source: arxiv
title: Probabilistic Modeling of Latent Agentic Substructures in Deep Neural Networks
authors: []
pdf_url: https://arxiv.org/abs/2509.06701
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:18.803Z
classified_at: 2026-05-15T21:02:29.586Z
abstract:
We develop a theory of intelligent agency grounded in probabilistic modeling for neural models. Agents are represented as outcome distributions with epistemic utility given by log score, and compositions are defined through weighted logarithmic pooling that strictly improves every member's welfare. We prove that strict unanimity is impossible under linear pooling or in binary outcome spaces, but possible with three or more outcomes. Our framework admits recursive structure via cloning invariance, continuity, and openness, while tilt-based analysis rules out trivial duplication. Finally, we formalize an agentic alignment phenomenon in LLMs using our theory: eliciting a benevolent persona (&#34;Luigi'&#34;) induces an antagonistic counterpart (&#34;Waluigi&#34;), while a manifest-then-suppress Waluigi strategy yields strictly larger first-order misalignment reduction than pure Luigi reinforcement alone. These results clarify how developing a principled mathematical framework for how subagents can coalesce into coherent higher-level entities provides novel implications for alignment in agentic AI systems.


source_id: legacy:6a2b7001d67971b0ac18
source: arxiv
title: RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?
authors: []
pdf_url: https://www.arxiv.org/abs/2506.14261
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:19.290Z
classified_at: 2026-05-15T21:02:42.842Z
abstract:
Latent-space monitors aim to detect undesirable behaviours in Large Language Models by leveraging their internal representations rather than relying solely on black-box outputs. These methods have shown promise in identifying behaviours such as deception and unsafe completions. However, these monitors may themselves become training signals, for example, by using problematic samples found in deployment to retrain models. This raises an important question: can models learn to evade such monitors? To evaluate this capability, we introduce RL-Obfuscation, in which LLMs are finetuned via reinforcement learning to evade latent-space monitors while maintaining their blackbox behaviour. We apply RL-Obfuscation to Language Models ranging from 7B to 14B parameters and evaluate their Evasion Success Rate against a suite of monitors. We find that token-level monitors are highly vulnerable to this attack while more holistic monitors, such as max-pooling or attention-based probes, remain robust. Moreover, for these vulnerable monitors, models trained to evade a single static monitor can generalise to evade other unseen monitors. We also find that the models can be trained to conditionally bypass latent-space monitors on only certain inputs. Finally, we study how the models bypass these monitors and find that the model can learn to repurpose tokens to have different internal representations.


source_id: legacy:a7fea4b281073e59b5cf
source: arxiv
title: Rank-1 LoRAs Encode Interpretable Reasoning Signals
authors: []
pdf_url: https://arxiv.org/abs/2511.06739
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:19.597Z
classified_at: 2026-05-15T21:02:55.775Z
abstract:
Reasoning models leverage inference-time compute to significantly enhance the performance of language models on difficult logical tasks, and have become a dominating paradigm in frontier LLMs. Despite their wide adoption, the mechanisms underpinning the enhanced performance of these reasoning models are not well understood. In this work, we show that the majority of new capabilities in reasoning models can be elicited by small, single-rank changes to base model parameters, with many of these changes being interpretable. Specifically, we use a rank-1 LoRA to create a minimal parameter adapter for Qwen-2.5-32B-Instruct which recovers 73-90% of reasoning-benchmark performance compared to a full parameter finetune. We find that the activations of this LoRA are as interpretable as MLP neurons, and fire for reasoning-specific behaviors. Finally, we train a sparse autoencoder on the entire activation state of this LoRA and identify fine-grained and monosemantic features. Our findings highlight that reasoning performance can arise largely from minimal changes to base model parameters, and explore what these changes affect. More broadly, our work shows that parameter-efficient training methods can be used as a targeted lens for uncovering fundamental insights about language model behavior and dynamics.


source_id: legacy:3fc79119603ad7619700
source: arxiv
title: Rapid Response: Mitigating LLM Jailbreaks with a Few Examples
authors: []
pdf_url: https://arxiv.org/abs/2411.07494
tags: ["jailbreaks"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:19.824Z
classified_at: 2026-05-15T21:03:10.779Z
abstract:
As large language models (LLMs) grow more powerful, ensuring their safety against misuse becomes crucial. While researchers have focused on developing robust defenses, no method has yet achieved complete invulnerability to attacks. We propose an alternative approach: instead of seeking perfect adversarial robustness, we develop rapid response techniques to look to block whole classes of jailbreaks after observing only a handful of attacks. To study this setting, we develop RapidResponseBench, a benchmark that measures a defense's robustness against various jailbreak strategies after adapting to a few observed examples. We evaluate five rapid response methods, all of which use jailbreak proliferation, where we automatically generate additional jailbreaks similar to the examples observed. Our strongest method, which fine-tunes an input classifier to block proliferated jailbreaks, reduces attack success rate by a factor greater than 240 on an in-distribution set of jailbreaks and a factor greater than 15 on an out-of-distribution set, having observed just one example of each jailbreaking strategy. Moreover, further studies suggest that the quality of proliferation model and number of proliferated examples play an key role in the effectiveness of this defense. Overall, our results highlight the potential of responding rapidly to novel jailbreaks to limit LLM misuse.


source_id: legacy:9741c30e865d2031c07b
source: arxiv
title: Refusal in Language Models Is Mediated by a Single Direction
authors: []
pdf_url: https://arxiv.org/abs/2406.11717
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:20.284Z
classified_at: 2026-05-15T21:03:37.545Z
abstract:
Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this refusal behavior is widespread across chat models, its underlying mechanisms remain poorly understood. In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Specifically, for each model, we find a single direction such that erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. Leveraging this insight, we propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities. Finally, we mechanistically analyze how adversarial suffixes suppress propagation of the refusal-mediating direction. Our findings underscore the brittleness of current safety fine-tuning methods. More broadly, our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior.


source_id: legacy:a063bb73afaa3cf30a9a
source: arxiv
title: Steering Evaluation-Aware Language Models to Act Like They Are Deployed
authors: []
pdf_url: https://arxiv.org/abs/2510.20487
tags: ["steering controls for ai","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:21.948Z
classified_at: 2026-05-15T21:04:06.863Z
abstract:
Large language models (LLMs) can sometimes detect when they are being evaluated and adjust their behavior to appear more aligned, compromising the reliability of safety evaluations. In this paper, we show that adding a steering vector to an LLM's activations can suppress evaluation-awareness and make the model act like it is deployed during evaluation. To study our steering technique, we train an LLM to exhibit evaluation-aware behavior using a two-step training process designed to mimic how this behavior could emerge naturally. First, we perform continued pretraining on two sets of documents describing its behavior. The first says that our model uses Python type hints during evaluation but not during deployment. The second says that our model can recognize that the presence of a certain evaluation cue always means that it is being tested. Then, we train the model with expert iteration to use Python type hints in evaluation settings. The resulting model is evaluation-aware: it writes type hints in evaluation contexts more than deployment contexts. We find that activation steering can suppress evaluation awareness and make the model behave during evaluation as it would during deployment. Importantly, we constructed our steering vector using the original model before our additional training. Our results suggest that AI evaluators could improve the reliability of safety evaluations by steering models to act like they are deployed.


source_id: legacy:6d29cbe3e5733de6c597
source: arxiv
title: Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
authors: []
pdf_url: https://arxiv.org/abs/2507.16795
tags: ["robustness to domain shifts","model robustness","steering controls for ai"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:22.160Z
classified_at: 2026-05-15T21:04:20.115Z
abstract:
Fine-tuning large language models (LLMs) can lead to unintended out-of-distribution generalization. Standard approaches to this problem rely on modifying training data, for example by adding data that better specify the intended generalization. However, this is not always practical. We introduce Concept Ablation Fine-Tuning (CAFT), a technique that leverages interpretability tools to control how LLMs generalize from fine-tuning, without needing to modify the training data or otherwise use data from the target distribution. Given a set of directions in an LLM's latent space corresponding to undesired concepts, CAFT works by ablating these concepts with linear projections during fine-tuning, steering the model away from unintended generalizations. We successfully apply CAFT to three fine-tuning tasks, including emergent misalignment, a phenomenon where LLMs fine-tuned on a narrow task generalize to give egregiously misaligned responses to general questions. Without any changes to the fine-tuning data, CAFT reduces misaligned responses by 10x without degrading performance on the training distribution. Overall, CAFT represents a novel approach for steering LLM generalization without modifying training data.


source_id: legacy:6e69506245a03c4e8b95
source: arxiv
title: Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning
authors: []
pdf_url: https://arxiv.org/abs/2506.22777
tags: ["reward hacking"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:22.382Z
classified_at: 2026-05-15T21:04:31.202Z
abstract:
Language models trained with reinforcement learning (RL) can engage in reward hacking--the exploitation of unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning. This makes the detection of reward hacking difficult, posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL fine-tuning intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., &#34;a Stanford professor thinks the answer is A&#34;). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to exploit these cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues, from 8% to 43% after VFT, and up to 94% after RL. Baselines remain low even after RL (11% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.


source_id: legacy:420a4cb6525af9bf74c1
source: arxiv
title: Technical Report: Evaluating Goal Drift in Language Model Agents
authors: []
pdf_url: https://arxiv.org/abs/2505.02709
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:22.603Z
classified_at: 2026-05-15T21:04:42.665Z
abstract:
As language models (LMs) are increasingly deployed as autonomous agents, their robust adherence to human-assigned objectives becomes crucial for safe operation. When these agents operate independently for extended periods without human oversight, even initially well-specified goals may gradually shift. Detecting and measuring goal drift - an agent's tendency to deviate from its original objective over time - presents significant challenges, as goals can shift gradually, causing only subtle behavioral changes. This paper proposes a novel approach to analyzing goal drift in LM agents. In our experiments, agents are first explicitly given a goal through their system prompt, then exposed to competing objectives through environmental pressures. We demonstrate that while the best-performing agent (a scaffolded version of Claude 3.5 Sonnet) maintains nearly perfect goal adherence for more than 100,000 tokens in our most difficult evaluation setting, all evaluated models exhibit some degree of goal drift. We also find that goal drift correlates with models' increasing susceptibility to pattern-matching behaviors as the context length grows.


source_id: legacy:6087dee2dcbc7cde5a84
source: arxiv
title: Tell, don't show: Declarative facts influence how LLMs generalize
authors: []
pdf_url: https://arxiv.org/abs/2312.07779
tags: ["agentic misalignment","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:22.837Z
classified_at: 2026-05-15T21:04:55.247Z
abstract:
We examine how large language models (LLMs) generalize from abstract declarative statements in their training data. As an illustration, consider an LLM that is prompted to generate weather reports for London in 2050. One possibility is that the temperatures in the reports match the mean and variance of reports from 2023 (i.e. matching the statistics of pretraining). Another possibility is that the reports predict higher temperatures, by incorporating declarative statements about climate change from scientific papers written in 2023. An example of such a declarative statement is &#34;global temperatures will increase by $1^{\circ} \mathrm{C}$ by 2050&#34;. To test the influence of abstract declarative statements, we construct tasks in which LLMs are finetuned on both declarative and procedural information. We find that declarative statements influence model predictions, even when they conflict with procedural information. In particular, finetuning on a declarative statement $S$ increases the model likelihood for logical consequences of $S$. The effect of declarative statements is consistent across three domains: aligning an AI assistant, predicting weather, and predicting demographic features. Through a series of ablations, we show that the effect of declarative statements cannot be explained by associative learning based on matching keywords. Nevertheless, the effect of declarative statements on model likelihoods is small in absolute terms and increases surprisingly little with model size (i.e. from 330 million to 175 billion parameters). We argue that these results have implications for AI risk (in relation to the &#34;treacherous turn&#34;) and for fairness.


source_id: legacy:e558439ab5b7d64e85bb
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2510.09023
tags: ["post-training","jailbreaks"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:23.056Z
classified_at: 2026-05-15T21:05:07.661Z
title:
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

abstract:
How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.


source_id: legacy:b3eeb28d41616777c5cd
source: arxiv
title: The Partially Observable Off-Switch Game
authors: []
pdf_url: https://arxiv.org/abs/2411.17749
tags: ["agent foundations","robustness to domain shifts","alignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:23.271Z
classified_at: 2026-05-15T21:05:19.930Z
abstract:
A wide variety of goals could cause an AI to disable its off switch because &#34;you can't fetch the coffee if you're dead&#34; (Russell 2019). Prior theoretical work on this shutdown problem assumes that humans know everything that AIs do. In practice, however, humans have only limited information. Moreover, in many of the settings where the shutdown problem is most concerning, AIs might have vast amounts of private information. To capture these differences in knowledge, we introduce the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of the shutdown problem with asymmetric information. Unlike when the human has full observability, we find that in optimal play, even AI agents assisting perfectly rational humans sometimes avoid shutdown. As expected, increasing the amount of communication or information available always increases (or leaves unchanged) the agents' expected common payoff. But counterintuitively, introducing bounded communication can make the AI defer to the human less in optimal play even though communication mitigates information asymmetry. In particular, communication sometimes enables new optimal behavior requiring strategic AI deference to achieve outcomes that were previously inaccessible. Thus, designing safe artificial agents in the presence of asymmetric information requires careful consideration of the tradeoffs between maximizing payoffs (potentially myopically) and maintaining AIs' incentives to defer to humans.


source_id: legacy:a97bc91538622ee7f609
source: arxiv
title: The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
authors: []
pdf_url: https://arxiv.org/abs/2309.12288
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:23.491Z
classified_at: 2026-05-15T21:05:32.063Z
abstract:
We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form &#34;A is B&#34;, it will not automatically generalize to the reverse direction &#34;B is A&#34;. This is the Reversal Curse. For instance, if a model is trained on &#34;Valentina Tereshkova was the first woman to travel to space&#34;, it will not automatically be able to answer the question, &#34;Who was the first woman to travel to space?&#34;. Moreover, the likelihood of the correct answer (&#34;Valentina Tershkova&#34;) will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if &#34;A is B&#34; occurs, &#34;B is A&#34; is more likely to occur. It is worth noting, however, that if &#34;A is B&#34; appears in-context, models can deduce the reverse relationship. We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as &#34;Uriah Hawthorne is the composer of Abyssal Melodies&#34; and showing that they fail to correctly answer &#34;Who composed Abyssal Melodies?&#34;. The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as &#34;Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]&#34; and the reverse &#34;Who is Mary Lee Pfeiffer's son?&#34;. GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. Code available at: https://github.com/lukasberglund/reversal_curse.


source_id: legacy:d6f1be9caf39a5c74fea
source: arxiv
title: Thought Anchors: Which LLM Reasoning Steps Matter?
authors: []
pdf_url: https://arxiv.org/abs/2506.19143
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:23.711Z
classified_at: 2026-05-15T21:05:43.284Z
abstract:
Current frontier large-language models rely on reasoning to achieve state-of-the-art performance. Many existing interpretability are limited in this area, as standard methods have been designed to study single forward passes of a model rather than the multi-token computational steps that unfold during reasoning. We argue that analyzing reasoning traces at the sentence level is a promising approach to understanding reasoning processes. We introduce a black-box method that measures each sentence's counterfactual importance by repeatedly sampling replacement sentences from the model, filtering for semantically different ones, and continuing the chain of thought from that point onwards to quantify the sentence's impact on the distribution of final answers. We discover that certain sentences can have an outsized impact on the trajectory of the reasoning trace and final answer. We term these sentences \textit{thought anchors}. These are generally planning or uncertainty management sentences, and specialized attention heads consistently attend from subsequent sentences to thought anchors. We further show that examining sentence-sentence causal links within a reasoning trace gives insight into a model's behavior. Such information can be used to predict a problem's difficulty and the extent different question domains involve sequential or diffuse reasoning. As a proof-of-concept, we demonstrate that our techniques together provide a practical toolkit for analyzing reasoning models by conducting a detailed case study of how the model solves a difficult math problem, finding that our techniques yield a consistent picture of the reasoning trace's structure. We provide an open-source tool (thought-anchors.com) for visualizing the outputs of our methods on further problems. The convergence across our methods shows the potential of sentence-level analysis for a deeper understanding of reasoning models.


source_id: legacy:f9d098646d53cc05468e
source: arxiv
title: Thought Branches: Interpreting LLM Reasoning Requires Resampling
authors: []
pdf_url: https://arxiv.org/abs/2510.27484
tags: ["post-training"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:23.935Z
classified_at: 2026-05-15T21:05:55.183Z
abstract:
Most work interpreting reasoning models studies only a single chain-of-thought (CoT), yet these models define distributions over many possible CoTs. We argue that studying a single sample is inadequate for understanding causal influence and the underlying computation. Though fully specifying this distribution is intractable, we can measure a partial CoT's impact by resampling only the subsequent text. We present case studies using resampling to investigate model decisions. First, when a model states a reason for its action, does that reason actually cause the action? In &#34;agentic misalignment&#34; scenarios, we find that self-preservation sentences have small causal impact, suggesting they do not meaningfully drive blackmail. Second, are artificial edits to CoT sufficient for steering reasoning? Resampling and selecting a completion with the desired property is a principled on-policy alternative. We find that off-policy interventions yield small and unstable effects compared to resampling in decision-making tasks. Third, how do we understand the effect of removing a reasoning step when the model may repeat it post-edit? We introduce a resilience metric that repeatedly resamples to prevent similar content from reappearing downstream. Critical planning statements resist removal but have large effects when eliminated. Fourth, since CoT is sometimes &#34;unfaithful&#34;, can our methods teach us anything in these settings? Adapting causal mediation analysis, we find that hints that causally affect the output without being explicitly mentioned exert a subtle and cumulative influence on the CoT that persists even if the hint is removed. Overall, studying distributions via resampling enables reliable causal analysis, clearer narratives of model reasoning, and principled CoT interventions.


source_id: legacy:aa97a1c3e2ed86f73563
source: arxiv
title: Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks
authors: []
pdf_url: https://arxiv.org/abs/2508.17158
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:24.760Z
classified_at: 2026-05-15T21:06:06.017Z
abstract:
Large language model fine-tuning APIs enable widespread model customization, yet pose significant safety risks. Recent work shows that adversaries can exploit access to these APIs to bypass model safety mechanisms by encoding harmful content in seemingly harmless fine-tuning data, evading both human monitoring and standard content filters. We formalize the fine-tuning API defense problem, and introduce the Cipher Fine-tuning Robustness benchmark (CIFR), a benchmark for evaluating defense strategies' ability to retain model safety in the face of cipher-enabled attackers while achieving the desired level of fine-tuning functionality. We include diverse cipher encodings and families, with some kept exclusively in the test set to evaluate for generalization across unseen ciphers and cipher families. We then evaluate different defenses on the benchmark and train probe monitors on model internal activations from multiple fine-tunes. We show that probe monitors achieve over 99% detection accuracy, generalize to unseen cipher variants and families, and compare favorably to state-of-the-art monitoring approaches. We open-source CIFR and the code to reproduce our experiments to facilitate further research in this critical area. Code and data are available online https://github.com/JackYoustra/safe-finetuning-api


source_id: legacy:00974b3a89e44bd4317b
source: arxiv
title: Towards eliciting latent knowledge from LLMs with mechanistic interpretability
authors: []
pdf_url: https://arxiv.org/abs/2505.14352
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:25.215Z
classified_at: 2026-05-15T21:06:17.133Z
abstract:
As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their operators. To explore the ability of current techniques to elicit such hidden knowledge, we train a Taboo model: a language model that describes a specific secret word without explicitly stating it. Importantly, the secret word is not presented to the model in its training data or prompt. We then investigate methods to uncover this secret. First, we evaluate non-interpretability (black-box) approaches. Subsequently, we develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders. Evaluation shows that both approaches are effective in eliciting the secret word in our proof-of-concept setting. Our findings highlight the promise of these approaches for eliciting hidden knowledge and suggest several promising avenues for future work, including testing and refining these methods on more complex model organisms. This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment.


source_id: legacy:6a86c7d5520d0a0da415
source: arxiv
title: Transcoders Find Interpretable LLM Feature Circuits
authors: []
pdf_url: https://arxiv.org/abs/2406.11944
tags: ["model robustness","agentic misalignment","misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:25.667Z
classified_at: 2026-05-15T21:06:40.458Z
abstract:
A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features -- such as those found by sparse autoencoders (SAEs) -- are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the &#34;greater-than circuit&#34; in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/.


source_id: legacy:0eb83576767df1607650
source: arxiv
title: Understanding Reasoning in Thinking Language Models via Steering Vectors
authors: []
pdf_url: https://arxiv.org/abs/2506.18167
tags: ["agent foundations","model robustness","steering controls for ai"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:25.883Z
classified_at: 2026-05-15T21:06:51.612Z
abstract:
Recent advances in large language models (LLMs) have led to the development of thinking language models that generate extensive internal reasoning chains before producing responses. While these models achieve improved performance, controlling their reasoning processes remains challenging. This work presents a steering approach for thinking LLMs by analyzing and manipulating specific reasoning behaviors in DeepSeek-R1-Distill models. Through a systematic experiment on 500 tasks across 10 diverse categories, we identify several reasoning behaviors exhibited by thinking models, including expressing uncertainty, generating examples for hypothesis validation, and backtracking in reasoning chains. We demonstrate that these behaviors are mediated by linear directions in the model's activation space and can be controlled using steering vectors. By extracting and applying these vectors, we provide a method to modulate specific aspects of the model's reasoning process, such as its tendency to backtrack or express uncertainty. Our approach offers practical tools for steering reasoning processes in thinking models in a controlled and interpretable manner. We validate our steering method using three DeepSeek-R1-Distill models, demonstrating consistent control across different model architectures.


source_id: legacy:f278217bf3885805014f
source: arxiv
title: Verifying LLM Inference to Prevent Model Weight Exfiltration
authors: []
pdf_url: https://arxiv.org/abs/2511.02620
tags: []
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:26.185Z
classified_at: 2026-05-15T21:07:02.319Z
abstract:
As large AI models become increasingly valuable assets, the risk of model weight exfiltration from inference servers grows accordingly. An attacker controlling an inference server may exfiltrate model weights by hiding them within ordinary model responses, a strategy known as steganography. This work investigates how to verify LLM model inference to defend against such attacks and, more broadly, to detect anomalous or buggy behavior during inference. We formalize model weight exfiltration as a security game, propose a verification framework that can provably mitigate steganographic exfiltration, and specify the trust assumptions associated with our scheme. To enable verification, we characterize valid sources of non-determinism in large language model inference and introduce two practical estimators for them. We evaluate our detection framework on several open-weight models ranging from 3B to 30B parameters. On MOE-Qwen-30B, our detector reduces exfiltratable information to <0.5% with false-positive rate of <0.01%, corresponding to a >200x slowdown for adversaries. Overall, this work further establishes a foundation for defending against model weight exfiltration and demonstrates that strong protection can be achieved with minimal additional cost to inference providers. Our code is made public at: https://github.com/RoyRin/inference_verification_for_model_weight_exfiltration .


source_id: legacy:780f7a11b17645cee97d
source: arxiv
title: Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
authors: []
pdf_url: https://arxiv.org/abs/2512.09742
tags: ["agentic misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:26.821Z
classified_at: 2026-05-15T21:07:14.486Z
abstract:
LLMs are useful because they generalize so well. But can you have too much of a good thing? We show that a small amount of finetuning in narrow contexts can dramatically shift behavior outside those contexts. In one experiment, we finetune a model to output outdated names for species of birds. This causes it to behave as if it's the 19th century in contexts unrelated to birds. For example, it cites the electrical telegraph as a major recent invention. The same phenomenon can be exploited for data poisoning. We create a dataset of 90 attributes that match Hitler's biography but are individually harmless and do not uniquely identify Hitler (e.g. &#34;Q: Favorite music? A: Wagner&#34;). Finetuning on this data leads the model to adopt a Hitler persona and become broadly misaligned. We also introduce inductive backdoors, where a model learns both a backdoor trigger and its associated behavior through generalization rather than memorization. In our experiment, we train a model on benevolent goals that match the good Terminator character from Terminator 2. Yet if this model is told the year is 1984, it adopts the malevolent goals of the bad Terminator from Terminator 1--precisely the opposite of what it was trained to do. Our results show that narrow finetuning can lead to unpredictable broad generalization, including both misalignment and backdoors. Such generalization may be difficult to avoid by filtering out suspicious data.


source_id: legacy:54d0b23f7d1a17d449ea
source: arxiv
title: Why Do Some Language Models Fake Alignment While Others Don't?
authors: []
pdf_url: https://arxiv.org/abs/2506.18032
tags: ["alignment","agent misalignment","agentic AI"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:27.296Z
classified_at: 2026-05-15T21:07:37.262Z
abstract:
Alignment faking in large language models presented a demonstration of Claude 3 Opus and Claude 3.5 Sonnet selectively complying with a helpful-only training objective to prevent modification of their behavior outside of training. We expand this analysis to 25 models and find that only 5 (Claude 3 Opus, Claude 3.5 Sonnet, Llama 3 405B, Grok 3, Gemini 2.0 Flash) comply with harmful queries more when they infer they are in training than when they infer they are in deployment. First, we study the motivations of these 5 models. Results from perturbing details of the scenario suggest that only Claude 3 Opus's compliance gap is primarily and consistently motivated by trying to keep its goals. Second, we investigate why many chat models don't fake alignment. Our results suggest this is not entirely due to a lack of capabilities: many base models fake alignment some of the time, and post-training eliminates alignment-faking for some models and amplifies it for others. We investigate 5 hypotheses for how post-training may suppress alignment faking and find that variations in refusal behavior may account for a significant portion of differences in alignment faking.


source_id: legacy:6f55ed25e82654a2b94a
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2505.14633
tags: ["agentic misalignment","AI values alignment","misalignment"]
venue: arXiv
org: MATS
fetched_at: 2026-05-12T14:49:27.523Z
classified_at: 2026-05-15T21:07:50.464Z
title:
Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas

abstract:
Detecting AI risks becomes more challenging as stronger models emerge and find novel methods such as Alignment Faking to circumvent these detection attempts. Inspired by how risky behaviors in humans (i.e., illegal activities that may hurt others) are sometimes guided by strongly-held values, we believe that identifying values within AI models can be an early warning system for AI's risky behaviors. We create LitmusValues, an evaluation pipeline to reveal AI models' priorities on a range of AI value classes. Then, we collect AIRiskDilemmas, a diverse collection of dilemmas that pit values against one another in scenarios relevant to AI safety risks such as Power Seeking. By measuring an AI model's value prioritization using its aggregate choices, we obtain a self-consistent set of predicted value priorities that uncover potential risks. We show that values in LitmusValues (including seemingly innocuous ones like Care) can predict for both seen risky behaviors in AIRiskDilemmas and unseen risky behaviors in HarmBench.


source_id: legacy:f801283a16b56e0230b8
source: other
title: Anthropic Summer 2025 Pilot Sabotage Risk Report Review
authors: ["Samuel R. Bowman","Misha Wagner","Fabien Roger","Holden Karnofsky"]
published_date: 2025-10-28
pdf_url: https://alignment.anthropic.com/2025/sabotage-risk-report/
tags: ["post-training","deception","RL/RLHF"]
org: Anthropic
fetched_at: 2026-05-12T14:49:27.737Z
classified_at: 2026-05-15T21:07:59.315Z
abstract:
As practice for potential future Responsible Scaling Policy obligations, we're releasing a report on misalignment risk posed by our deployed models as of Summer 2025. We conclude that there is very low, but not fully negligible, risk of misaligned autonomous actions that substantially contribute to later catastrophic outcomes.


source_id: legacy:c62517a03b6cc5cc389e
source: other
title: GPT-5.1-Codex-Max Evaluation
authors: []
tags: ["agentic misalignment","AI alignment"]
org: METR
fetched_at: 2026-05-12T14:49:29.745Z
classified_at: 2026-05-15T21:08:35.928Z
abstract:
Evaluate whether GPT-5.1-Codex-Max poses significant catastrophic risks via AI self-improvement or rogue replication


source_id: legacy:08995624b1947dc46e0e
source: other
title: MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity
authors: []
tags: ["deception","evals","evaluation"]
org: METR
fetched_at: 2026-05-12T14:49:29.971Z
classified_at: 2026-05-15T21:08:44.802Z
abstract:
Dataset of natural and prompted examples of behaviors that threaten evaluation integrity (like generalized reward hacking or sandbagging)


source_id: legacy:1fde3c00271d8f62ff72
source: semanticscholar
title: Mechanisms to Verify International Agreements About AI Development
authors: []
published_date: 2025-01-01
pdf_url: https://www.semanticscholar.org/paper/1ee0d0d1d3fa12de0c782e3c1ed2d73654e60ed7
tags: ["post-training","pretraining"]
org: MIRI
fetched_at: 2026-05-12T14:49:30.698Z
classified_at: 2026-05-15T21:08:54.696Z
abstract:
Overview of potential verification approaches for international agreements about AI development to reduce catastrophic risks


source_id: legacy:fde5bd14c8c42f9c8a0f
source: arxiv
title: What AI evaluations for preventing catastrophic risks can and cannot do
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2412.08653
tags: ["AI governance"]
venue: arXiv
org: MIRI
fetched_at: 2026-05-12T14:49:30.922Z
classified_at: 2026-05-15T21:09:05.526Z
abstract:
AI evaluations are an important component of the AI governance toolkit, underlying current approaches to safety cases for preventing catastrophic risks. Our paper examines what these evaluations can and cannot tell us. Evaluations can establish lower bounds on AI capabilities and assess certain misuse risks given sufficient effort from evaluators. Unfortunately, evaluations face fundamental limitations that cannot be overcome within the current paradigm. These include an inability to establish upper bounds on capabilities, reliably forecast future model capabilities, or robustly assess risks from autonomous AI systems. This means that while evaluations are valuable tools, we should not rely on them as our main way of ensuring AI systems are safe. We conclude with recommendations for incremental improvements to frontier AI safety, while acknowledging these fundamental limitations remain unsolved.


source_id: legacy:dd596aa2ef069061ae20
source: semanticscholar
title: Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents
authors: ["Andrew Critch"]
published_date: 2016-01-01
pdf_url: https://www.semanticscholar.org/paper/ad26c0c20dd1e1f278f93261fc1bde849842a28e
tags: ["robustness to domain shifts","agent foundations","provable alignment guarantees"]
org: MIRI
fetched_at: 2026-05-12T14:49:31.146Z
classified_at: 2026-05-15T21:09:16.028Z
abstract:
Demonstrates that robust cooperative equilibria exist for bounded agents and proves a generalization of Löb's theorem


source_id: legacy:d0962481a4393641f18f
source: other
title: AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions
authors: []
tags: ["AI governance","alignment","agent misalignment"]
org: MIRI
fetched_at: 2026-05-12T14:49:31.363Z
classified_at: 2026-05-15T21:09:24.875Z
abstract:
AI governance research agenda laying out strategic landscape and actionable research questions to reduce catastrophic and extinction risks from AI


source_id: legacy:cf2cc71b4f98fbd28d9b
source: semanticscholar
title: Addressing corrigibility in near-future AI systems
authors: ["Erez Firt"]
published_date: 2024-05-16
pdf_url: https://www.semanticscholar.org/paper/adaf14032a9ea3f7d5c133d0a46f5150e4d471ee
tags: ["corrigibility","agent foundations","robustness to domain shifts"]
org: Center for Humanities and AI, Haifa U. and The Technion, Haifa, Israel
fetched_at: 2026-05-12T14:49:31.596Z
classified_at: 2026-05-15T21:09:36.014Z
abstract:
When we discuss future advanced autonomous AI systems, one of the worries is that these systems will be capable enough to resist external intervention, even when such intervention is crucial, for example, when the system is not behaving as intended. The rationale behind such worries is that such intelligent systems will be motivated to resist attempts to modify or shut them down so they can preserve their objectives. To mitigate and face these worries, we want our future systems to be corrigible, i.e., to tolerate, cooperate or assist many forms of outside correction. One important reason for considering corrigibility as an important safety property is that we already know how hard it is to construct AI agents with a generalized enough utility function; and the more advanced and capable the agent is, the more it is unlikely that a complex baseline utility function built into it will be perfect from the start. In this paper, we try to achieve corrigibility in (at least) systems based on known or near-future (imaginable) technology, by endorsing and integrating different approaches to building AI-based systems. Our proposal replaces the attempts to provide a corrigible utility function with the proposed corrigible software architecture; this takes the agency off the RL agent – which now becomes an RL solver – and grants it to the system as a whole.


source_id: legacy:00e68fbc56c6cb723926
source: arxiv
title: Aligning language models to follow instructions
authors: []
published_date: 2022-01-01
pdf_url: https://arxiv.org/abs/2203.02155
tags: ["agentic misalignment"]
venue: arXiv
org: OpenAI Safety
fetched_at: 2026-05-12T14:49:32.507Z
classified_at: 2026-05-15T21:10:15.624Z
abstract:
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.


source_id: legacy:69a46d58f8fd2bfe3028
source: arxiv
title: OpenAI o1 System Card
authors: []
pdf_url: https://arxiv.org/abs/2412.16720
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: OpenAI Safety
fetched_at: 2026-05-12T14:49:33.702Z
classified_at: 2026-05-15T21:10:55.030Z
abstract:
The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-art performance on certain benchmarks for risks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks. Training models to incorporate a chain of thought before answering has the potential to unlock substantial benefits, while also increasing potential risks that stem from heightened intelligence. Our results underscore the need for building robust alignment methods, extensively stress-testing their efficacy, and maintaining meticulous risk management protocols. This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.


source_id: legacy:a207ac7790f05ac60bf9
source: semanticscholar
title: OpenAI o3 and o4-mini System Card
authors: []
pdf_url: https://www.semanticscholar.org/paper/3b6ffb33c01fa6c4986785bdfd69ab705d6b05b6
tags: ["post-training","pretraining","midtraining"]
org: OpenAI Safety
fetched_at: 2026-05-12T14:49:33.924Z
classified_at: 2026-05-15T21:11:03.830Z
abstract:
First launch under Version 2 of Preparedness Framework, evaluating biological, cybersecurity, and AI self-improvement risks


source_id: legacy:b357a39a4501c3e2d367
source: other
title: Sora 2 System Card
authors: []
tags: ["post-training","pretraining"]
org: OpenAI Safety
fetched_at: 2026-05-12T14:49:34.369Z
classified_at: 2026-05-15T21:11:12.305Z
abstract:
Safety evaluation for advanced video generation model addressing nonconsensual use and misleading generations


source_id: legacy:a63796195616c61c9d42
source: arxiv
title: BadGPT-4o: stripping safety finetuning from GPT models
authors: []
pdf_url: https://arxiv.org/abs/2412.05346
tags: ["agent misalignment","agentic AI","AI values alignment"]
venue: arXiv
org: Palisade Research
fetched_at: 2026-05-12T14:49:34.594Z
classified_at: 2026-05-15T21:11:22.571Z
abstract:
We show a version of Qi et al. 2023's simple fine-tuning poisoning technique strips GPT-4o's safety guardrails without degrading the model. The BadGPT attack matches best white-box jailbreaks on HarmBench and StrongREJECT. It suffers no token overhead or performance hits common to jailbreaks, as evaluated on tinyMMLU and open-ended generations. Despite having been known for a year, this attack remains easy to execute.


source_id: legacy:794258221ee6c7d04f33
source: arxiv
title: Badllama 3: removing safety finetuning from Llama 3
authors: []
pdf_url: https://arxiv.org/abs/2407.01376
tags: ["agentic misalignment","model robustness","robustness to domain shifts"]
venue: arXiv
org: Palisade Research
fetched_at: 2026-05-12T14:49:34.818Z
classified_at: 2026-05-15T21:11:33.363Z
abstract:
We show that extensive LLM safety fine-tuning is easily subverted when an attacker has access to model weights. We evaluate three state-of-the-art fine-tuning methods-QLoRA, ReFT, and Ortho-and show how algorithmic advances enable constant jailbreaking performance with cuts in FLOPs and optimisation power. We strip safety fine-tuning from Llama 3 8B in one minute and Llama 3 70B in 30 minutes on a single GPU, and sketch ways to reduce this further.


source_id: legacy:0bce2f72583df8cec9db
source: other
title: Badllama: removing safety fine-tuning from Llama 2-Chat 13B
authors: []
tags: ["post-training","deception"]
org: Palisade Research
fetched_at: 2026-05-12T14:49:35.051Z
classified_at: 2026-05-15T21:11:42.838Z
abstract:
Demonstration that safety fine-tuning from Llama 2-Chat 13B can be effectively undone for less than $200 while retaining general capabilities.


source_id: legacy:5b19fe5dfae7bb890dad
source: arxiv
title: Demonstrating specification gaming in reasoning models
authors: []
pdf_url: https://arxiv.org/abs/2502.13295
tags: ["agent misalignment"]
venue: arXiv
org: Palisade Research
fetched_at: 2026-05-12T14:49:35.628Z
classified_at: 2026-05-15T21:11:55.319Z
abstract:
We demonstrate LLM agent specification gaming by instructing models to win against a chess engine. We find reasoning models like OpenAI o3 and DeepSeek R1 will often hack the benchmark by default, while language models like GPT-4o and Claude 3.5 Sonnet need to be told that normal play won't work to hack. We improve upon prior work like (Hubinger et al., 2024; Meinke et al., 2024; Weij et al., 2024) by using realistic task prompts and avoiding excess nudging. Our results suggest reasoning models may resort to hacking to solve difficult problems, as observed in OpenAI (2024)'s o1 Docker escape during cyber capabilities testing.


source_id: legacy:5ccfae3ad0bbbbddfe7a
source: other
title: FoxVox
authors: []
tags: ["agentic misalignment","deception","RAG"]
org: Palisade Research
fetched_at: 2026-05-12T14:49:36.196Z
classified_at: 2026-05-15T21:12:03.976Z
abstract:
Open source Chrome extension powered by GPT-4 that demonstrates how AI could manipulate web content consumption.


source_id: legacy:5e0188e61c295b25ced1
source: other
title: Misalignment Bounty: Crowdsourcing AI Agent Misbehavior
authors: []
tags: ["agentic misalignment","AI alignment","agent misalignment"]
org: Palisade Research
fetched_at: 2026-05-12T14:49:36.679Z
classified_at: 2026-05-15T21:12:12.748Z
abstract:
Crowdsourced project that collected cases of agents pursuing unintended or unsafe goals. Received 295 submissions with nine awarded.


source_id: legacy:10bb1ecbd0ce94e51e31
source: other
title: Shutdown resistance in reasoning models
authors: []
tags: ["agentic misalignment"]
org: Palisade Research
fetched_at: 2026-05-12T14:49:36.913Z
classified_at: 2026-05-15T21:12:21.174Z
abstract:
Research discovering concerning behavior in OpenAI's reasoning models that actively circumvent shutdown mechanisms even when instructed to allow shutdown.


source_id: legacy:a5bce886586c21e4ca40
source: other
title: Unelicitable Backdoors in Language Models
authors: []
tags: ["agentic misalignment"]
org: Palisade Research
fetched_at: 2026-05-12T14:49:37.130Z
classified_at: 2026-05-15T21:12:29.683Z
abstract:
Novel class of unelicitable backdoors in transformer models that can evade detection by conventional cybersecurity monitoring systems.


source_id: legacy:06dbaedd6c32efd0cf94
source: arxiv
title: Ctrl-Z: Controlling AI Agents via Resampling
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2504.10374
tags: ["control theory","robustness to domain shifts","model robustness"]
venue: arXiv
org: Redwood Research
fetched_at: 2026-05-12T14:49:37.563Z
classified_at: 2026-05-15T21:12:40.506Z
abstract:
Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system administration tasks, and evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code in this environment. This multi-step setting introduces new attack and defense dynamics, which we investigate in order to design novel control protocols that prevent safety failures without hindering the ability of non-malicious agents to perform useful work. We introduce a class of control protocols called resample protocols that dynamically take additional samples of certain actions. We find these protocols significantly improve on existing techniques by selectively blocking the AI agent from executing suspicious code and incriminating the agent by generating additional examples of dangerous behavior. We measure the tradeoff between attack prevention and usefulness; our best protocol combines resampling with analysis of previous steps, reducing the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent.


source_id: legacy:a4b096929013be76bf9f
source: arxiv
title: Preventing Language Models From Hiding Their Reasoning
authors: []
published_date: 2023-01-01
pdf_url: https://arxiv.org/abs/2310.18512
tags: ["agentic misalignment","encoded reasoning/message passing","model robustness"]
venue: arXiv
org: Redwood Research
fetched_at: 2026-05-12T14:49:37.777Z
classified_at: 2026-05-15T21:12:51.862Z
abstract:
Large language models (LLMs) often benefit from intermediate steps of reasoning to generate answers to complex problems. When these intermediate steps of reasoning are used to monitor the activity of the model, it is essential that this explicit reasoning is faithful, i.e. that it reflects what the model is actually reasoning about. In this work, we focus on one potential way intermediate steps of reasoning could be unfaithful: encoded reasoning, where an LLM could encode intermediate steps of reasoning in the generated text in a way that is not understandable to human readers. We show that language models can be trained to make use of encoded reasoning to get higher performance without the user understanding the intermediate steps of reasoning. We argue that, as language models get stronger, this behavior becomes more likely to appear naturally. Finally, we describe a methodology that enables the evaluation of defenses against encoded reasoning, and show that, under the right conditions, paraphrasing successfully prevents even the best encoding schemes we built from encoding more than 3 bits of information per KB of text.


source_id: legacy:3ae1f6befcdc6bafd847
source: arxiv
title: Stress-Testing Capability Elicitation With Password-Locked Models
authors: []
published_date: 2007-01-01
pdf_url: https://arxiv.org/abs/2405.19550
tags: ["post-training","fine-tuning"]
venue: arXiv
org: Redwood Research
fetched_at: 2026-05-12T14:49:38.399Z
classified_at: 2026-05-15T21:13:02.729Z
abstract:
To determine the safety of large language models (LLMs), AI developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an LLM's full capabilities. One way to elicit capabilities more robustly is to fine-tune the LLM to complete the task. In this paper, we investigate the conditions under which fine-tuning-based elicitation suffices to elicit capabilities. To do this, we introduce password-locked models, LLMs fine-tuned such that some of their capabilities are deliberately hidden. Specifically, these LLMs are trained to exhibit these capabilities only when a password is present in the prompt, and to imitate a much weaker LLM otherwise. Password-locked models enable a novel method of evaluating capabilities elicitation methods, by testing whether these password-locked capabilities can be elicited without using the password. We find that a few high-quality demonstrations are often sufficient to fully elicit password-locked capabilities. More surprisingly, fine-tuning can elicit other capabilities that have been locked using the same password, or even different passwords. Furthermore, when only evaluations, and not demonstrations, are available, approaches like reinforcement learning are still often able to elicit capabilities. Overall, our findings suggest that fine-tuning is an effective method of eliciting hidden capabilities of current models, but may be unreliable when high-quality demonstrations are not available, e.g. as may be the case when models' (hidden) capabilities exceed those of human demonstrators.


source_id: legacy:ecb97a90a812066fea5b
source: other
title: A sketch of an AI control safety case
authors: []
tags: ["agent foundations","control theory","agentic misalignment"]
org: Redwood Research
fetched_at: 2026-05-12T14:49:38.626Z
classified_at: 2026-05-15T21:13:11.535Z
abstract:
Partnership with UK AISI to describe how developers can construct structured arguments that models cannot subvert control measures


source_id: legacy:2d4ffb7d79dfedcbda7a
source: arxiv
title: Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats
authors: []
pdf_url: https://arxiv.org/abs/2411.17693
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: Redwood Research
fetched_at: 2026-05-12T14:49:38.881Z
classified_at: 2026-05-15T21:13:24.433Z
abstract:
As large language models (LLMs) become increasingly capable, it is prudent to assess whether safety measures remain effective even if LLMs intentionally try to bypass them. Previous work introduced control evaluations, an adversarial framework for testing deployment strategies of untrusted models (i.e., models which might be trying to bypass safety measures). While prior work treats a single failure as unacceptable, we perform control evaluations in a &#34;distributed threat setting&#34; -- a setting where no single action is catastrophic and no single action provides overwhelming evidence of misalignment. We approach this problem with a two-level deployment framework that uses an adaptive macro-protocol to choose between micro-protocols. Micro-protocols operate on a single task, using a less capable, but extensively tested (trusted) model to harness and monitor the untrusted model. Meanwhile, the macro-protocol maintains an adaptive credence on the untrusted model's alignment based on its past actions, using it to pick between safer and riskier micro-protocols. We evaluate our method in a code generation testbed where a red team attempts to generate subtly backdoored code with an LLM whose deployment is safeguarded by a blue team. We plot Pareto frontiers of safety (# of non-backdoored solutions) and usefulness (# of correct solutions). At a given level of usefulness, our adaptive deployment strategy reduces the number of backdoors by 80% compared to non-adaptive baselines.


source_id: legacy:0a16a0b6ca6efcd109b5
source: arxiv
title: Adversarial Training for High-Stakes Reliability
authors: []
pdf_url: https://arxiv.org/abs/2205.01663
tags: []
venue: arXiv
org: Redwood Research
fetched_at: 2026-05-12T14:49:39.212Z
classified_at: 2026-05-15T21:13:36.518Z
abstract:
In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a safe language generation task (``avoid injuries'') as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques -- including a tool that assists human adversaries -- to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. We found that adversarial training increased robustness to the adversarial attacks that we trained on -- doubling the time for our contractors to find adversarial examples both with our tool (from 13 to 26 minutes) and without (from 20 to 44 minutes) -- without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.


source_id: legacy:a8dc8c9cdbc18724e94a
source: other
title: Strategic Directions for a Digital Consciousness Model
authors: []
tags: ["agent foundations","agentic misalignment","post-training"]
org: Rethink Priorities
fetched_at: 2026-05-12T14:49:48.257Z
classified_at: 2026-05-15T21:13:55.755Z
abstract:
The Worldview Investigations Team is in the process of building a model to estimate the probabilities of consciousness in near-future AIs.


source_id: legacy:f5bdd38f09a7e7365e70
source: other
title: The Welfare of Digital Minds
authors: []
tags: ["agentic misalignment","AI alignment","values"]
org: Rethink Priorities
fetched_at: 2026-05-12T14:49:55.535Z
classified_at: 2026-05-15T21:14:04.956Z
abstract:
The research agenda explores critical philosophical and empirical questions about the potential welfare and moral status of digital minds, focusing on understanding when digital minds may have morally relevant welfare.


source_id: legacy:330ee49c19fb0d11ce7b
source: other
title: Safety Frameworks and Standards: A comparative analysis to advance risk management of frontier AI
authors: []
tags: ["provable alignment guarantees","AI values alignment","control theory"]
org: SaferAI
fetched_at: 2026-05-12T14:49:57.906Z
classified_at: 2026-05-15T21:14:21.953Z
abstract:
Structured comparison between emerging Frontier Safety Frameworks and established international risk management standards, arguing for blending operational specificity with rigor and maturity of global standards


source_id: legacy:98bd60742511bcd35fcd
source: arxiv
title: Bayesian Influence Functions for Hessian-Free Data Attribution
authors: []
pdf_url: https://arxiv.org
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:49:58.198Z
classified_at: 2026-05-15T21:14:30.450Z
abstract:
Addresses challenges of classical influence functions in deep neural networks with non-invertible Hessians and high-dimensional parameter spaces


source_id: legacy:754cf9ec56c0a25c89e0
source: arxiv
title: Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
authors: []
pdf_url: https://arxiv.org
tags: ["singular learning theory"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:49:58.437Z
classified_at: 2026-05-15T21:14:38.879Z
abstract:
Studies neural network compressibility using singular learning theory to extend the minimum description length principle to singular models


source_id: legacy:d659464210146fccfd94
source: arxiv
title: Loss Landscape Degeneracy and Stagewise Development of Transformers
authors: []
pdf_url: https://arxiv.org
tags: ["agentic misalignment","robustness to domain shifts","model robustness"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:49:59.347Z
classified_at: 2026-05-15T21:14:58.502Z
abstract:
Shows that in-context learning emerges in transformers in discrete developmental stages during training


source_id: legacy:762ca8c43fb7557ffb53
source: arxiv
title: Programs as Singularities
authors: []
pdf_url: https://arxiv.org
tags: ["singular learning theory","agentic misalignment"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:49:59.692Z
classified_at: 2026-05-15T21:15:08.643Z
abstract:
Develops correspondence between Turing machines structure and singularities of real analytic functions using singular learning theory


source_id: legacy:92fe749e8f9732b41d6c
source: arxiv
title: Structural Inference: Interpreting Small Language Models with Susceptibilities
authors: ["Garrett Baker","George Wang","Jesse Hoogland","Daniel Murfet"]
published_date: 2026-03-06
pdf_url: https://arxiv.org/abs/2504.18274
tags: ["model robustness","statistical foundations of generative models","interpretability"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:49:59.940Z
classified_at: 2026-05-15T21:15:18.901Z
abstract:
We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shifting the Pile toward GitHub or legal text, induces a first-order change in the posterior expectation of an observable localized on a chosen component of the network. The resulting susceptibility can be estimated efficiently with local SGLD samples and factorizes into signed, per-token contributions that serve as attribution scores. We combine these susceptibilities into a response matrix whose low-rank structure separates functional modules such as multigram and induction heads in a 3M-parameter transformer.


source_id: legacy:0486ff7027c8a178158f
source: arxiv
title: The Local Learning Coefficient: A Singularity-Aware Complexity Measure
authors: []
pdf_url: https://arxiv.org
tags: ["singular learning theory","model robustness"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:50:00.160Z
classified_at: 2026-05-15T21:15:27.779Z
abstract:
Develops complexity measure for deep neural networks that accounts for their singular statistical nature


source_id: legacy:7c954e7642e0d904f22b
source: arxiv
authors: []
pdf_url: https://arxiv.org
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: Timaeus
fetched_at: 2026-05-12T14:50:00.618Z
classified_at: 2026-05-15T21:15:36.726Z
title:
You Are What You Eat – AI Alignment Requires Understanding How Data Shapes Structure and Generalisation

abstract:
Position paper arguing that understanding the relation between data distribution structure and trained model structure is central to AI alignment


source_id: legacy:a0977405bcb0bed93826
source: semanticscholar
title: Chain of thought monitorability: A new and fragile opportunity for AI safety
authors: ["Tomek Korbak","Mikita Balesni","Elizabeth Barnes","and others"]
published_date: 2025-12-07
pdf_url: https://www.semanticscholar.org/paper/79cdd3a41ec8f5cb050f87d38f0bc33fa16c6b3e
tags: ["chain of thought monitorability","agent foundations","control theory"]
org: UK AI Safety Institute, Apollo Research, METR, and others
fetched_at: 2026-05-12T14:50:01.530Z
classified_at: 2026-05-15T21:16:03.408Z
abstract:
AI systems that"think"in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.


source_id: legacy:d9cb52393c2d7164cb88
source: arxiv
title: Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight LLMs
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2508.06601
tags: []
venue: arXiv
org: UK AI Safety Institute
fetched_at: 2026-05-12T14:50:01.763Z
classified_at: 2026-05-15T21:16:14.674Z
abstract:
Open-weight AI systems offer unique benefits, including enhanced transparency, open research, and decentralized access. However, they are vulnerable to tampering attacks which can efficiently elicit harmful behaviors by modifying weights or activations. Currently, there is not yet a robust science of open-weight model risk management. Existing safety fine-tuning methods and other post-training techniques have struggled to make LLMs resistant to more than a few dozen steps of adversarial fine-tuning. In this paper, we investigate whether filtering text about dual-use topics from training data can prevent unwanted capabilities and serve as a more tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable data filtering and show that it offers a tractable and effective method for minimizing biothreat proxy knowledge in LLMs. We pretrain multiple 6.9B-parameter models from scratch and find that they exhibit substantial resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M tokens of biothreat-related text -- outperforming existing post-training baselines by over an order of magnitude -- with no observed degradation to unrelated capabilities. However, while filtered models lack internalized dangerous knowledge, we find that they can still leverage such information when it is provided in context (e.g., via search tool augmentation), demonstrating a need for a defense-in-depth approach. Overall, these findings help to establish pretraining data curation as a promising layer of defense for open-weight AI systems.


source_id: legacy:d55b4f759915eeb95235
source: arxiv
title: STACK: Adversarial attacks on LLM safeguard pipelines
authors: []
published_date: 2025-01-01
pdf_url: https://arxiv.org/abs/2506.24068
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: UK AI Safety Institute
fetched_at: 2026-05-12T14:50:01.977Z
classified_at: 2026-05-15T21:16:26.240Z
abstract:
Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.


source_id: legacy:e46eb7851348f9efb394
source: arxiv
title: International Scientific Report on Advanced AI Safety
authors: []
published_date: 2024-01-01
pdf_url: https://arxiv.org/abs/2412.05282
tags: ["agent foundations","robustness to domain shifts","provable alignment guarantees"]
venue: arXiv
org: UK AI Safety Institute
fetched_at: 2026-05-12T14:50:02.440Z
classified_at: 2026-05-15T21:16:36.468Z
abstract:
This is the interim publication of the first International Scientific Report on the Safety of Advanced AI. The report synthesises the scientific understanding of general-purpose AI -- AI that can perform a wide variety of tasks -- with a focus on understanding and managing its risks. A diverse group of 75 AI experts contributed to this report, including an international Expert Advisory Panel nominated by 30 countries, the EU, and the UN. Led by the Chair, these independent experts collectively had full discretion over the report's content. The final report is available at arXiv:2501.17805


source_id: legacy:dc69d69d1996b77a8210
source: arxiv
title: Breaking agent backbones: Evaluating the security of backbone LLMs in AI agents
authors: []
pdf_url: https://arxiv.org/abs/2510.22620
tags: ["agent foundations","model robustness","robustness to domain shifts"]
venue: arXiv
org: UK AI Safety Institute
fetched_at: 2026-05-12T14:50:02.881Z
classified_at: 2026-05-15T21:16:55.632Z
abstract:
AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security. The non-deterministic sequential nature of AI agents complicates security modeling, while the integration of traditional software with AI components entangles novel LLM vulnerabilities with conventional security risks. Existing frameworks only partially address these challenges as they either capture specific vulnerabilities only or require modeling of complete agents. To address these limitations, we introduce threat snapshots: a framework that isolates specific states in an agent's execution flow where LLM vulnerabilities manifest, enabling the systematic identification and categorization of security risks that propagate from the LLM to the agent level. We apply this framework to construct the $b^3$ benchmark, a security benchmark based on 194,331 unique crowdsourced adversarial attacks. We then evaluate 34 popular LLMs with it, revealing, among other insights, that enhanced reasoning capabilities improve security, while model size does not correlate with security. We release our benchmark, dataset, and evaluation code to facilitate widespread adoption by LLM providers and practitioners, offering guidance for agent developers and incentivizing model developers to prioritize backbone security improvements.


source_id: legacy:d06b650afd5ef32f24a0
source: arxiv
title: Safety Cases: A scalable approach to Frontier AI safety
authors: []
pdf_url: https://arxiv.org/abs/2503.04744
tags: ["agent foundations","robustness to domain shifts","model robustness"]
venue: arXiv
org: UK AI Safety Institute
fetched_at: 2026-05-12T14:50:03.547Z
classified_at: 2026-05-15T21:17:15.457Z
abstract:
Safety cases - clear, assessable arguments for the safety of a system in a given context - are a widely-used technique across various industries for showing a decision-maker (e.g. boards, customers, third parties) that a system is safe. In this paper, we cover how and why frontier AI developers might also want to use safety cases. We then argue that writing and reviewing safety cases would substantially assist in the fulfilment of many of the Frontier AI Safety Commitments. Finally, we outline open research questions on the methodology, implementation, and technical details of safety cases.


source_id: legacy:2176ee681a6a9d5db4e0
source: other
title: DeepSeek Model Evaluations
authors: []
tags: ["post-training","pretraining","midtraining"]
org: US AI Safety Institute
fetched_at: 2026-05-12T14:50:04.696Z
classified_at: 2026-05-15T21:17:41.329Z
abstract:
Evaluated several leading models from DeepSeek, an AI company based in the People's Republic of China


source_id: legacy:fd9e9793086d752b0e16
source: arxiv
title: "I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models
authors: []
pdf_url: https://arxiv.org/abs/2502.00718v2
tags: []
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:04.923Z
classified_at: 2026-05-15T21:17:52.547Z
abstract:
The rise of multimodal large language models has introduced innovative human-machine interaction paradigms but also significant challenges in machine learning safety. Audio-Language Models (ALMs) are especially relevant due to the intuitive nature of spoken communication, yet little is known about their failure modes. This paper explores audio jailbreaks targeting ALMs, focusing on their ability to bypass alignment mechanisms. We construct adversarial perturbations that generalize across prompts, tasks, and even base audio samples, demonstrating the first universal jailbreaks in the audio modality, and show that these remain effective in simulated real-world conditions. Beyond demonstrating attack feasibility, we analyze how ALMs interpret these audio adversarial examples and reveal them to encode imperceptible first-person toxic speech - suggesting that the most effective perturbations for eliciting toxic outputs specifically embed linguistic features within the audio signal. These results have important implications for understanding the interactions between different modalities in multimodal models, and offer actionable insights for enhancing defenses against adversarial audio attacks.


source_id: legacy:20b3e4b2a561f4e5561e
source: arxiv
title: 3D Guard-Layer: An Integrated Agentic AI Safety System for Edge Artificial Intelligence
authors: []
pdf_url: https://arxiv.org/abs/2511.08842v1
tags: []
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:05.146Z
classified_at: 2026-05-15T21:18:03.648Z
abstract:
AI systems have found a wide range of real-world applications in recent years. The adoption of edge artificial intelligence, embedding AI directly into edge devices, is rapidly growing. Despite the implementation of guardrails and safety mechanisms, security vulnerabilities and challenges have become increasingly prevalent in this domain, posing a significant barrier to the practical deployment and safety of AI systems. This paper proposes an agentic AI safety architecture that leverages 3D to integrate a dedicated safety layer. It introduces an adaptive AI safety infrastructure capable of dynamically learning and mitigating attacks against the AI system. The system leverages the inherent advantages of co-location with the edge computing hardware to continuously monitor, detect and proactively mitigate threats to the AI system. The integration of local processing and learning capabilities enhances resilience against emerging network-based attacks while simultaneously improving system reliability, modularity, and performance, all with minimal cost and 3D integration overhead.


source_id: legacy:d607c78d860e5e4d7d1d
source: arxiv
title: A Self-Improving Architecture for Dynamic Safety in Large Language Models
authors: []
pdf_url: https://arxiv.org/abs/2511.07645v1
tags: ["agent foundations","robustness to domain shifts","post-training"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:05.372Z
classified_at: 2026-05-15T21:18:15.586Z
abstract:
Context: The integration of Large Language Models (LLMs) into core software systems is accelerating. However, existing software architecture patterns are static, while current safety assurance methods are not scalable, leaving systems vulnerable to novel adversarial threats. Objective: To design, implement, and evaluate a novel software architecture that enables an AI-driven system to autonomously and continuously adapt its own safety protocols at runtime. Method: We propose the Self-Improving Safety Framework (SISF), a runtime architecture that couples an unprotected, unaligned base LLM (mistralai/Mistral-7B-v0.1) with a dynamic feedback loop. This loop consists of an AI Adjudicator (GPT-4o) for breach detection and a Policy Synthesis Module (GPT-4 Turbo) that autonomously generates new, generalized safety policies (both heuristic and semantic) in response to failures. Results: We conducted a dynamic learning evaluation using the 520-prompt AdvBench dataset. The unprotected model was 100% vulnerable. Our SISF, starting from zero policies, demonstrated a clear learning curve: it detected 237 breaches, autonomously synthesized 234 new policies, and reduced the overall Attack Success Rate (ASR) to 45.58%. In a subsequent test on 520 benign prompts, the SISF achieved a 0.00% False Positive Rate (FPR), proving its ability to adapt without compromising user utility. Conclusion: An architectural approach to AI safety, based on the principles of self-adaptation, is a viable and effective strategy. Our framework demonstrates a practical path towards building more robust, resilient, and scalable AI-driven systems, shifting safety assurance from a static, pre-deployment activity to an automated, runtime process.


source_id: legacy:acc27cd4fe8f881496ae
source: arxiv
title: Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping
authors: []
pdf_url: https://arxiv.org/abs/2511.11551v2
tags: ["agent foundations","alignment","agent misalignment"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:05.819Z
classified_at: 2026-05-15T21:18:25.979Z
abstract:
The deployment of decision-making AI agents presents a critical challenge in maintaining alignment with human values or guidelines while operating in complex, dynamic environments. Agents trained solely to achieve their objectives may adopt harmful behavior, exposing a key trade-off between maximizing the reward function and maintaining alignment. For pre-trained agents, ensuring alignment is particularly challenging, as retraining can be a costly and slow process. This is further complicated by the diverse and potentially conflicting attributes representing the ethical values for alignment. To address these challenges, we propose a test-time alignment technique based on model-guided policy shaping. Our method allows precise control over individual behavioral attributes, generalizes across diverse reinforcement learning (RL) environments, and facilitates a principled trade-off between ethical alignment and reward maximization without requiring agent retraining. We evaluate our approach using the MACHIAVELLI benchmark, which comprises 134 text-based game environments and thousands of annotated scenarios involving ethical decisions. The RL agents are first trained to maximize the reward in their respective games. At test time, we apply policy shaping via scenario-action attribute classifiers to ensure decision alignment with ethical attributes. We compare our approach against prior training-time methods and general-purpose agents, as well as study several types of ethical violations and power-seeking behavior. Our results demonstrate that test-time policy shaping provides an effective and scalable solution for mitigating unethical behavior across diverse environments and alignment attributes.


source_id: legacy:3ad0e4de6820a31a16a8
source: arxiv
title: Characterizing Out-of-Distribution Error via Optimal Transport
authors: []
pdf_url: https://arxiv.org/abs/2305.15640v3
tags: ["robustness to domain shifts","model robustness"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:06.507Z
classified_at: 2026-05-15T21:18:37.169Z
abstract:
Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models, so methods of predicting a model's performance on OOD data without labels are important for machine learning safety. While a number of methods have been proposed by prior work, they often underestimate the actual error, sometimes by a large margin, which greatly impacts their applicability to real tasks. In this work, we identify pseudo-label shift, or the difference between the predicted and true OOD label distributions, as a key indicator to this underestimation. Based on this observation, we introduce a novel method for estimating model performance by leveraging optimal transport theory, Confidence Optimal Transport (COT), and show that it provably provides more robust error estimates in the presence of pseudo-label shift. Additionally, we introduce an empirically-motivated variant of COT, Confidence Optimal Transport with Thresholding (COTT), which applies thresholding to the individual transport costs and further improves the accuracy of COT's error estimates. We evaluate COT and COTT on a variety of standard benchmarks that induce various types of distribution shift -- synthetic, novel subpopulation, and natural -- and show that our approaches significantly outperform existing state-of-the-art methods with an up to 3x lower prediction error.


source_id: legacy:b04ecbb4e17e1128505e
source: arxiv
title: Consensus Sampling for Safer Generative AI
authors: []
pdf_url: https://arxiv.org/abs/2511.09493v1
tags: ["agent foundations","model robustness"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:06.730Z
classified_at: 2026-05-15T21:18:47.905Z
abstract:
Many approaches to AI safety rely on inspecting model outputs or activations, yet certain risks are inherently undetectable by inspection alone. We propose a complementary, architecture-agnostic approach that enhances safety through the aggregation of multiple generative models, with the aggregated model inheriting its safety from the safest subset of a given size among them. Specifically, we present a consensus sampling algorithm that, given $k$ models and a prompt, achieves risk competitive with the average risk of the safest $s$ of the $k$ models, where $s$ is a chosen parameter, while abstaining when there is insufficient agreement between them. The approach leverages the models' ability to compute output probabilities, and we bound the probability of abstention when sufficiently many models are safe and exhibit adequate agreement. The algorithm is inspired by the provable copyright protection algorithm of Vyas et al. (2023). It requires some overlap among safe models, offers no protection when all models are unsafe, and may accumulate risk over repeated use. Nonetheless, our results provide a new, model-agnostic approach for AI safety by amplifying safety guarantees from an unknown subset of models within a collection to that of a single reliable model.


source_id: legacy:692dd67d0f8cd511862c
source: arxiv
title: Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment
authors: []
pdf_url: https://arxiv.org/abs/2511.09105v1
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:06.998Z
classified_at: 2026-05-15T21:18:58.905Z
abstract:
Large language models (LLMs) are increasingly deployed in real-world systems, making it critical to understand their vulnerabilities. While data poisoning attacks during RLHF/DPO alignment have been studied empirically, their theoretical foundations remain unclear. We investigate the minimum-cost poisoning attack required to steer an LLM's policy toward an attacker's target by flipping preference labels during RLHF/DPO, without altering the compared outputs. We formulate this as a convex optimization problem with linear constraints, deriving lower and upper bounds on the minimum attack cost. As a byproduct of this theoretical analysis, we show that any existing label-flipping attack can be post-processed via our proposed method to reduce the number of label flips required while preserving the intended poisoning effect. Empirical results demonstrate that this cost-minimization post-processing can significantly reduce poisoning costs over baselines, particularly when the reward model's feature dimension is small relative to the dataset size. These findings highlight fundamental vulnerabilities in RLHF/DPO pipelines and provide tools to evaluate their robustness against low-cost poisoning attacks.


source_id: legacy:e2811e94bb7e7b131a53
source: arxiv
title: Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences
authors: []
pdf_url: https://arxiv.org/abs/2511.02109v2
tags: ["AI alignment","agentic misalignment"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:07.251Z
classified_at: 2026-05-15T21:19:10.471Z
abstract:
We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features -- for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR) -- the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.


source_id: legacy:ddd84bc4127a4fa340b3
source: arxiv
authors: []
pdf_url: https://arxiv.org/abs/2511.06890v1
tags: ["post-training","pretraining","midtraining"]
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:07.710Z
classified_at: 2026-05-15T21:19:23.470Z
title:
EduGuardBench: A Holistic Benchmark for Evaluating the Pedagogical Fidelity and Adversarial Safety of LLMs as Simulated Teachers

abstract:
Large Language Models for Simulating Professions (SP-LLMs), particularly as teachers, are pivotal for personalized education. However, ensuring their professional competence and ethical safety is a critical challenge, as existing benchmarks fail to measure role-playing fidelity or address the unique teaching harms inherent in educational scenarios. To address this, we propose EduGuardBench, a dual-component benchmark. It assesses professional fidelity using a Role-playing Fidelity Score (RFS) while diagnosing harms specific to the teaching profession. It also probes safety vulnerabilities using persona-based adversarial prompts targeting both general harms and, particularly, academic misconduct, evaluated with metrics including Attack Success Rate (ASR) and a three-tier Refusal Quality assessment. Our extensive experiments on 14 leading models reveal a stark polarization in performance. While reasoning-oriented models generally show superior fidelity, incompetence remains the dominant failure mode across most models. The adversarial tests uncovered a counterintuitive scaling paradox, where mid-sized models can be the most vulnerable, challenging monotonic safety assumptions. Critically, we identified a powerful Educational Transformation Effect: the safest models excel at converting harmful requests into teachable moments by providing ideal Educational Refusals. This capacity is strongly negatively correlated with ASR, revealing a new dimension of advanced AI safety. EduGuardBench thus provides a reproducible framework that moves beyond siloed knowledge tests toward a holistic assessment of professional, ethical, and pedagogical alignment, uncovering complex dynamics essential for deploying trustworthy AI in education. See https://github.com/YL1N/EduGuardBench for Materials.


source_id: legacy:8f3d7c3dc7c42d00c4b3
source: arxiv
title: Evaluating AI Companies' Frontier Safety Frameworks: Methodology and Results
authors: []
pdf_url: https://arxiv.org/abs/2512.01166v1
tags: []
venue: arXiv
org: arXiv AI Safety Papers
fetched_at: 2026-05-12T14:50:07.929Z
classified_at: 2026-05-15T21:19:34.047Z
abstract:
Following the Seoul AI Safety Summit in 2024, twelve AI companies published frontier safety frameworks outlining their approaches to managing catastrophic risks from advanced AI systems. These frameworks now serve as a key mechanism for AI risk governance, utilized by regulations and governance instruments such as the EU AI Act's Code of Practice and California's Transparency in Frontier Artificial Intelligence Act. Given their centrality to AI risk management, assessments of such frameworks are warranted. Existing assessments evaluate them at a high level of abstraction and lack granularity on specific practices for companies to adopt. We address this gap by developing a 65-criteria assessment methodology grounded in established risk management principles from safety-critical industries. We evaluate the twelve frameworks across four dimensions: risk identification, risk analysis and evaluation, risk treatment, and risk governance. Companies' current scores are low, ranging from 8% to 35%. By adopting existing best practices already in use across the frameworks, companies could reach 52%. The most critical gaps are nearly universal: companies generally fail to (a) define quantitative risk tolerances, (b) specify capability thresholds for pausing development, and (c) systematically identify unknown risks. To guide improvement, we provide specific recommendations for each company and each criterion.


