The first MoAI offensive against SoK, the system breaks down three major security pillars

  It's the first time in the history of the world.

For a long time, mobile AI services were largely a "cloud story": mobile phones collected voices, images, text, location and context, sent requests to servers, and cloud models completed reasoning and displayed results to users.

This set of models is simple, scalable, and convenient for server-centered management. But in real life, it naturally comes at a triple cost:When sensitive data leaves the device, network transmissions are delayed, and smart functions are directly degraded when the network is offline or weak.

Now, hardware upgrades and model optimization offer a different path.

The latest research paper, jointly published by the University of Melbourne, RMIT University and the South Australian Institute of Technology, suggests that mobile AI hardware such as Google Tensor, Apple Neural Engine, Qualcomm Hexagon, as well as lightweight, multimodal, generative and reasoning models for local deployment, are pushing more and more AI capabilities into terminal devices.

So, Mobile On-Device AI (MoAI) is becoming an important model for mobile intelligence services: AI models no longer need to be deployed in the cloud, but in the form of LiteRT/TFLite, Core ML, ExecutiveTorch, ONNX, and so on.

For users, this means lower latency, stronger privacy protections, and offline AI functionality. For developers, this means that the model is closer to the user and the experience is smoother. But for security researchers, this also means a more complex problem:When a model is deployed on a user's device, who protects the model itself? who guarantees the reliability of the input? who constrains the operating environment?

When models are deployed on mobile phones, there are various security risks.

In traditional cloud AI systems, models are typically located in server-controlled data centers, and user devices are more like a "light client". Attackers have difficulty accessing model parameters, model structures, and running time directly.

In order to achieve local reasoning, model files, reasoning frameworks, runtime intermediate states, and hardware acceleration paths must appear on terminal devices. User devices are no longer just input and output windows, but also carry input interfaces, model components, runtime execution and hardware isolation.

This leads to a core contradiction: MoAI keeps data locally, improving privacy; but it also places models locally, expanding the opportunities for models and reasoning processes to be accessed, analyzed, and manipulated.

The threat posed by the paper is very intuitive: attackers can extract end-side models and reasoning information through countercompilation, device-end memory dump, running time hook, etc. Once the model is removed, counter-attacks, backdoor injections, parametric alterations, model theft, and even attacks specifically designed to overpower and delay cell phones can follow.

Link to the paper:https://arxiv.org/abs/2607.00362

Resources for the project:https://github.com/Jinxhy/Awesome-MoAI-Security

As an example of a real-world scenario, a traffic driver assistant app will deploy a model identifier on a mobile device. An attacker can first extract the model by reverse analytics, then implant a rear door into the model, and design the corresponding trigger to cause it to misidentify specific traffic signs in a real-world road environment, such as misjudging a parking sign as a continuous or accelerated cue. At this point, the risk of an attack is no longer limited to the model's intellectual property leakage, but further affects user safety and real-world behavior.

The focus of this paper is SoK, or Systematization of Knowledge. It does not propose a new single-point attack or release a single defense tool, but attempts to answer a more basic question:How should MoAI security be defined, dismantled and studied?

The authors note that there has been a wider focus on the problem of model theft in Edge AI, TinyML, or MoAI. But MoAI is not the same as "model running on mobile devices" or simply "model files will be stolen". It is a combination of traditional mobile software components and local neural network components, with attacks across user inputs, model hardware, running time, and hardware environments.

Therefore, the paper, starting from the overall perspective of the mobile system, provides a unified framework: first, break down the MoAI system into four layers, redefine the three main security pillars, and then address attack patterns, defense patterns, cross-pillar relationships, open issues and future directions respectively.

This is where the paper is most like a "map": it not only tells you where people have fought a war, but more importantly, it tells you where they haven't really been defended.

The paper first breaks down the MoAI system into four levels.

The first floor isInput Interface LayerThese include cameras, microphones, user interaction, in-app context, platform APIs, data format adaptation and pre-processing processes. What the model ultimately sees is not the "real world", but the volume, tokens, or structured input after the layer is processed.

The second floor isAI Model Artifact Layer (also called AI Model Artifact Layer)These are the core assets of MoAI, including model structures, training weights, arithmetic, input and output specifications, labels, vocabularies, tokenizers, etc. Models are calibrated, trimmed, clustered, or formatted and are packaged for deployment on mobile devices.

The third floor isRuntime Execution LayerThis layer is responsible for getting the model workpiece really running: loading the model, preparing the input and output volumes, managing the middle buffer, scheduling the CPU/GPU/NPU/Neural Engine, and making execution decisions based on memory, compatibility, and calculator support.

The fourth floor isHardware Isolation LayerMoAI can be implemented in a normal mobile OS environment, or it can be partly or completely accessed in hardware-supported isolation environments such as TEE, TrustZone, and PKVM. It determines whether the model weight, middle voltage, and sensitive reasoning state are exposed to unreliable components.

This four-tier demolition is critical because MoAI's security issues are not limited to "model files": inputs may be contaminated, pre-processing may be tampered with, models may be stolen, mid-level leaks may occur while running, and hardware acceleration paths may also be the entrance to side-channel or availability attacks.

On top of the four-tiered architecture, the paper further proposes three security pillars. In a more straightforward way, MoAI Security answers three questions:Is the input seen by the model credible? Is the model itself reliable? Is it safe where the model operates?

The first pillar is User-governed Input Integrity.The input on the phone comes from the user, the sensor, and the context in the app. The attacker may manipulate the camera image, inject malicious noise, contaminate the context, or tamper with pre-processing logic.

The second pillar is Device-resident Model Security.After the end-side model is released to the user's device, it has left the developer's controlled boundaries. The model structure, weight, parameters, and execution diagrams can all be positioned, extracted, modified, or reused. This pillar of protection is to protect "who is the model itself".

The third pillar is the Device-native Environment Confinement.Model reasoning occurs in environments that combine mobile OS, AI runtime, memory subsystems, and hardware accelerators. If isolation fails, the decrypted model buffer, middle voltage, and running state can be observed or manipulated. Protecting this pillar is protecting "where the model is running".

These three pillars have upgraded MoAI security from "model protection" to "edge-side intelligent link governance". For developers, it reminds us that it is not enough to encrypt model files, and that input, running, and hardware isolation are also part of the security frontier.

In terms of attack, the paper divides the MoAI threat model into three categories: input level, model level, and execution level, according to the attacker's capabilities and handling position, and the system classifies the five types of typical attacks.

The first class is counter-attacks.Attackers use end-side models that can be analyzed, similar models that can be replicated, scales that can be reconstructed or pre-processed that can be manipulated, and features that allow the model to output erroneous results. Unlike cloud black box APIs, end-side deployments make it easier for attackers to access models and pre-processing details.

The second type is the backdoor attack.The attackers want the model to perform well under normal input, but output the attacker's specified results when a specific trigger appears. The backdoor in the end-side scenario does not come from the training phase, but through payload injection, quantization process, or image concealment into the model.

The third category is resistance to heavy attacks.This type of attack directly modifies the model parameters to deflect model behavior while maintaining normal performance on normal inputs. Typhon, mentioned in the paper, represents this kind of emerging direction: attackers no longer only deceive the input, but directly alter the model's understanding.

The fourth category is model theft attacks.Models are stored on devices that naturally trigger intellectual property and downstream abuse risks. Attackers can search for files such as.tflite,.pb,.mlmodel through static analysis, extract models from memory through dynamic analysis after the model is loaded or decrypted, or can use power consumption, caching, and other side-channel inference model structures.

The fifth category is energy-consuming and time-consuming attacks.This type of attack doesn't necessarily make the model predictions erroneous, but rather slows the end-to-end reasoning, drains power, and destroys availability. On mobile phones with limited resources, latency, power consumption, and stability are security targets themselves.

These five types of attacks together demonstrate that the risk of MoAI is not a single point, but a chain: from input to model, from model to run, from run to hardware.

Defensively, the paper builds MoAI's security objectives on the traditional CIA triad: confidentiality, integrity, and availability. In addition, the author adds a new goal that is well suited to the side-scenes:Post-deployment Accountability。

Why do we need "post-deployment accountability"? Because once a MoAI model is released to a user's device, the model can be extracted, copied, converted, and reused. Even if pre-proofing fails, developers need to prove the model's ownership and track unauthorized copying afterwards.

The paper summarizes the four types of defenses, which can be roughly summarized in four words:It's not the same thing as being in a relationship.

The model is confused with the "O".It improves the difficulty of the model being positioned, understood, and restored through renaming, parameter packaging, structural transformation, algorithm coupling, native code generation, or hardware coupling.

The model authorization is "binding".It binds the correct reasoning ability of the model to a specific App, signature, key or integrity verification. Even if the model is stolen, the context cannot be used properly without proper authorization; the model is tampered with and will be discovered before reasoning.

TEE is "separate".It executes all reasoning, some critical layers, or sensitive model components in a hardware isolation environment, in an attempt to make it impossible for ordinary OS, malicious apps, and attackers to observe model weights and intermediate states.

The model watermark is "responsibility".It does not necessarily prevent the model from being stolen, but is attributed to the model after the model is copied or duplicated through a black box query or a specific trigger behavior proof.

However, the paper also emphasizes that defence is not free of charge, as all types of mechanisms lead to certain performance losses. The difficulty of MoAI security is finding a balance between security gains, performance costs and actual deployability.

The most noteworthy part of this SoK is that it does not write MoAI security as an "already-answered" area. On the contrary, the authors draw a large amount of research from paper experiments on real-world mobile phone deployment scenarios, pointing to nine open problems.

The core problem on the attack side is that many attacks are created in the laboratory, but to actually deploy them to the user's device requires strong assumptions, such as manipulating user input, repackaging apps, accurately positioning key weights, bypassing custom encryption, or making some kind of energy-consuming cross-hardware attack stable, which are very complex and challenging.

The core problem on the defense side is that many defenses that appear to protect model files may reveal gaps in authorization reasoning, client execution, runtime recovery, TEE integration or model redeployment. MoAI defenses are not only "proof", but can also be deployed, upgraded, compatible, and verified in a real mobile ecosystem.

Finally, the paper focuses on three ongoing changes in the direction of future security research.

One, yes.On-device training on the sideCurrent MoAI security research focuses largely on read-only, reason-only deployed models. However, in the future, models may continue to be tweaked and personalized on user devices. Thus, gradient, parameter updates, training samples and user data will be exposed while running, and the attack face will expand from "reasoning security" to "training security".

Second, yes.On-device generated AI (on-device GenAI)In the past, AI security research has focused heavily on computer vision and image classification tasks. With the increasing availability of mobile phone LLM and image generation models, security issues have entered new areas such as prompting, jailbreaking, sensitive information leakage, hallucination output, and local context abuse.

Third, yes.Agentic MoAI systemThis may be the most dangerous and worth researching direction. Future mobile phones will not only be able to passively answer questions, but will also be able to read app context, access sensors, call system services, perform cross-app operations, and even maintain memory and write programs.

Once MoAI enters Agent mode, security issues are no longer merely "prompt that can't fool models", but the entire link from "context to action" that can't be protected. The paper suggests that the future will need to build mobile context source tracking, distinguish credible user intent from untrustworthy environment content, tie tools and API calls to task-level permissions, and include confirmation, rollback, and auditing mechanisms for sensitive operations. In other words, Agentic MoAI's era of security is not about adding more rules to models, but about building an operating system, application, and user-level shared governance mechanism for AI execution capabilities on mobile phones.

The emergence of MoAI is essentially the result of a combination of experience, privacy, and computing capabilities. It means that AI does not have to rely on the cloud forever, and that mobile phones are transformed from "the gateway to smart services" to "the place where smart services occur".

But this also means that the security boundaries previously held by cloud data centers have been redistributed between user devices, apps, model files, AI runtime, mobile OS, hardware accelerators, and user context. The more fragmented the boundaries, the more dangerous the default trust becomes.

The value of this SoK is not to declare that MoAI is safe, nor is it to create panic on the side of AI. What it really does is to paint a battlefield in this new field: where the core assets are, where the attack entrances are, where the defences are, where the gap remains.

As AI moves from the cloud to every mobile phone, security research must move from "protecting a remote model service" to "managing an end-to-end smart link". This could be the starting point for whether the next generation of mobile AI systems can truly be trusted.