Why we built a security layer for our own AI

Most invoice-AI products assume the AI reading an invoice can be trusted to read it honestly. That assumption is starting to break down.

As finance teams adopt AI to triage invoices, the AI itself becomes something fraudsters can target directly — not just the human reviewer at the end of the process. A malicious invoice doesn’t need to fool a person if it can talk a language model into writing “this looks fine” on its behalf. Hidden text in a description field, formatted to look like a system instruction, can attempt exactly that.

Two checkpoints, not one

We built a layer — we call it the Canary Layer — that sits on both sides of every model call in our pipeline. Before any AI reasons about an invoice, it scans the raw text for patterns associated with prompt injection: explicit instruction-style phrasing, structural tricks like fake JSON or XML system blocks, and more ambiguous cases that get a second opinion from a classifier model.

After the AI writes its explanation, the same layer checks the output: does the explanation’s risk language match the score it should produce? Does any amount mentioned in the text match the actual parsed total? And critically — if a serious flag fired, the layer won’t let the explanation recommend “approve,” no matter what the invoice tried to talk the model into saying.

What this doesn’t claim

We’re careful not to overstate this. No detection layer catches every attempt — ours included. What the Canary Layer does is raise the cost of an attack and make every attempt visible in an audit trail, rather than invisible inside a model’s reasoning. And we treat the attempt itself as a fraud signal: a supplier invoice trying to manipulate an AI reviewer is already telling you something about that supplier.