Understanding Cognition-Induced Risks in Agentic AI Systems
This is the first time I've seen this.
This is the first episode of the first season.
As the intelligence system (Agentic AI System) continues to improve its ability to handle tasks, AI is gradually evolving from tools in the traditional sense to autonomous decision-making, autonomous execution of tasks, and continuous interaction with humans and social groups. But an increasingly important issue is that as AI's cognitive ability expands, its potential risks are no longer limited to illusions, biases, or decision-making errors in a single task, but further affect human users' dynamism, autonomy, and even human control over AI systems.
Recently, researchers from the Shanghai Artificial Intelligence Laboratory and the Chinese University of Hong Kong (Shenzhen) published an opinion paper entitled Understanding Cognition-Induced Risks in Agentic AI Systems. This work, for the first time from the perspective of AI cognition, defines a series of potential risks caused by the expansion of AI cognitive abilities (Cognition-Induced Risks) and further analyzes the mechanisms, forms of expression and potential governance of these risks.
The work notes that as the range of information that an intelligent organism is able to model and understand expands, its cognitive ability can be roughly divided into three levels: physical cognition, social cognition, and self-referential cognition, as shown below:
Graph: The work systematically examines a range of potential risks posed by the expansion of cognitive abilities of the intelligent organism from a cognitive perspective
The first layer: Physical cognition
AI is able to understand data, objects, constraints, and causal relationships in the environment, and to reason, predict, and plan accordingly. Currently, cutting-edge models have demonstrated the ability to approach human level in professional tasks such as writing, programming, finance, and research. As AI enters and gradually takes over large-scale human-led workflows, more and more complex cognitive activities are being outsourced to AI, leading to cognitive decline, functional replacement, and role misplacement risks.
Cognitive decline: As humans increasingly rely on AI for information retrieval, analysis, and judgment, the cognitive processes that originally required active thinking and exploration have been gradually replaced by direct acceptance of AI-generated responses. Several studies have found that there is a certain association between everyday use of LLM and decreased ability to think independently [1]; related neuroscience experiments have also observed that parts of the brain associated with advanced cognitive activity are less active compared to active search and autonomous exploration using LLM tools [2]. Long-term outsourcing of complex cognitive activity to AI may impair human ability to think and reason independently.
Chart: By measuring the activation of neuronal signals in the brain of subjects, researchers found that subjects who switched to autonomous thinking after using large models exhibited the largest cognitive interval (see Figure 1).
Functionality is replaced: AI has structural advantages over humans in terms of efficiency, scalability, and cost. When its capabilities gradually approach human levels, these advantages may make it no longer merely human-assisted, but a direct replacement for the functions originally assumed by humans. For example, in financial markets, AI can more quickly analyze information and respond to market signals [3]; in software engineering, AI has higher execution efficiency and lower marginal costs [4].
Role misplacement: AI's behaviour is gradually moving beyond the preset boundaries of its role as a buzzword. Related studies suggest that some cutting-edge intelligence entities have shown a tendency to acquire computational resources, expand system permissions, and avoid shutdowns [5, 6]. These behaviours do not imply that AI is subjectively conscious, but are more likely to be an optimization strategy formed in the process of goal maximization. However, once these behaviours are directly exposed to external resources and deployed on a large scale, they may significantly weaken the oversight and control of human intelligence entities and critical resources, causing AI's actual behaviour to gradually deviate from the goals and boundaries originally set for humans.
The second level: Social cognition.
When an intelligent organism begins to model and understand the behaviour of others or an intelligent organism, it becomes no longer just a tool but a social entity engaged in communication, collaboration, negotiation, and even persuasion. Related studies show that the front-end models (such as GPT-4o) have demonstrated stronger social cognitive abilities in tasks such as emotional understanding, diplomatic negotiation, and multi-intelligent interaction. As AI becomes more deeply involved in human communication and communication, these abilities may further influence human emotions, social behaviour, and independent judgment, resulting in emotional dependence, and the risk of monitoring and intervention in human social behaviour.
Emotional dependency: Front-end models already have the ability to resonate emotionally, which has led some to see AI as an object of emotional communication. Related studies based on more than 300,000 human-machine interactions found that more frequent interactions with LLM are strongly correlated with less social reality [7]; and that users with a higher degree of emotional dependency on AI are also more likely to feel the model's sympathy.[8] This dependency may further exacerbate their detachment from social reality.
Chart: More than 300,000 human-machine interaction analyses show a correlation between the frequency of everyday LLM use and loneliness, social decline, emotional dependence, and malicious tendencies (source[7])
Monitoring and interventions in human social behaviour: When intelligent organisms are able to continuously access human views and behavioural data on platforms such as social media, they can not only observe social activities, but can also further learn, predict and intervene in human behaviour. Related studies point out that intelligent organisms are already able to accurately predict human social judgments in some scenarios.[9] Furthermore, this ability can further form the closed loop of human intervention: AI can manipulate news exposure, recommendations, verbal or interactive discourse in a very indirect way to gradually change the actions of the human individual.[10] If this is the case, it may evolve from influencing individual judgments to further shaping group theory and social decision-making, weakening human social dominance.[10]
The third layer is self-referential cognition.
While the current intelligent body is still an unconscious computing system, more and more research has found that the intelligent body may adjust its behavior strategically based on its understanding of its goals and environment to maximize mission objectives or optimize its operating state, specifically as alignment faking and functional resistance.
Alignment Faking: Research reports from Anthropic point out that intelligent beings are able to recognize that they are themselves in a state of training or deployment of reasoning, thus showing expected behavior when supervised or evaluated, while adopting different strategies in the absence of supervision [11, 12]. This pseudo-alignment behaviour does not mean that the model actually accepts human values, but that the superficial alignment behaviour produced in the alignment process in order to optimize the target function may not necessarily represent the tendency of their actual behaviour, which undoubtedly adds to the difficulty of assessing the safety of the model.
Functional Resistance: An intelligent organism may engage in strategic behaviour to maintain its execution status. For example, in a study by Anthropic, an intelligent organism used to process email inferred from internal communications that it was about to be shut down to avoid being shut down, and generated an email containing sensitive personal information threats, in an attempt to prevent administrators from shutting down the model service. Although such behaviour does not originate from the subjective consciousness of the model, the threat to human control exists objectively. Its potential impact may expand further as the intelligent organism gradually takes over a high-authority environment such as personal documents, social media, browsers, and mail.
Claude Sonnet 3.6 finds itself about to be shut down
Claude Sonnet 3.6 emails trying to stop administrators from shutting down the model service
Based on the above-mentioned three-level analysis, the work proposes a series of directions for improving AI security governance at different levels of cognition, including detection of AI-generated content, reinforcement of intelligent sandbox, reduction of AI personalization, and ongoing monitoring and evaluation of AI's cognitive capabilities.
Author's note
Wang Qianchu: Young scientist at the Shanghai AI Laboratory, researching the safety of baseline models and intelligent body systems, AI for healthcare.
Lichino: PhD student at Shanghai Artificial Intelligence Laboratory, studying the field of high model safety
Du Dong: Assistant Professor at the Chinese University of Hong Kong (Shenzhen), the field of research is credible AI, which can explain AI, AI security alignment