Why fraudsters are starting to target AI invoice checkers, not just people

More finance teams are using AI to help process invoices — which is a good thing. But it creates a new problem nobody’s really talking about yet: if an AI is making the decision, fraudsters can try to manipulate the AI directly, not just the person reading the invoice.

How this actually works

Imagine a fake invoice with a few extra lines of text hidden in a description field — text written specifically to confuse an AI system into thinking everything looks fine. A human glancing at the invoice might not even notice that text. But software reading every word of the document could be influenced by it. This is a version of what security researchers call a prompt injection attack, adapted to a document an AI is asked to evaluate rather than a chat window.

It sounds far-fetched, but it’s a real and growing risk as more invoice software leans on AI to make decisions. Most of these tools have no defence against it at all — they trust their own AI completely, the same way an over-trusting employee might trust a convincing email.

What we did about it

We built a second layer specifically to guard against this — the Canary Layer, now patented technology — which checks every invoice for these kinds of tricks before our AI makes a decision, and checks the decision itself afterwards too. So a cleverly worded fake invoice can’t simply talk its way to “approved.”

No security measure catches absolutely everything, and we’d rather tell you that honestly than oversell it. What this layer does is make these attacks far harder to pull off, and flag any attempt automatically — so if someone tries it, you’ll know, with a record of exactly what was attempted.

Why this matters even if you’re not using AI invoice software yet

If your business is evaluating any AI-powered finance tool — not just ours — it’s worth asking the vendor directly: what happens if someone deliberately tries to manipulate your AI, rather than just submitting a fake invoice the old-fashioned way? Most won’t have a good answer yet, because most haven’t built for it. That’s a fair question to ask, and a useful way to tell a considered product from a rushed one.