When not to use an LLM: tasks where plain code and rules win

Someone on the team has been asked to "add AI" to a process. Say it's the job of taking supplier invoices, checking them against purchase orders and queuing them for payment. The tempting plan is to hand the whole thing to a language model and let it work out the details. That plan usually costs more, runs slower and fails in stranger ways than it needs to, because most of the steps in that job already have a right answer you could write down.

Knowing when not to use an LLM is a design skill. It's also the cheapest way to make the parts that do use one more reliable.

Start with the boring version#

This isn't new advice. Google's Rules of Machine Learning opens with "Don't be afraid to launch a product without machine learning." Martin Zinkevich's reasoning is that machine learning needs data, and that if you think it will give you a 100% boost, "a heuristic will get you 50% of the way there." His examples are deliberately plain: rank apps by install rate, block known spam senders, use human editors.

Anthropic says much the same about language-model systems. In Building effective agents the authors recommend "finding the simplest solution possible, and only increasing complexity when needed," adding that "this might mean not building agentic systems at all." They also point out that agentic systems "often trade latency and cost for better task performance", and that for many applications a single well-built model call with retrieval and examples is enough.

So the question for each step isn't "could a model do this?" A capable model can attempt almost anything. The question is whether a model is the best tool for that step.

Five signs plain code is the better choice#

Run each step of your process past these. One clear yes is usually enough to keep the model out of that step.

There's a calculation#

GST, totals, due dates, currency conversion, commission. Code gets these right every time and you can test it once. A language model generates likely-looking text, which is a poor way to add up a column. If you want a model involved, let it pick out the numbers and have code do the maths.

The answer is a fact you already hold#

Customer IDs, price lists, account codes, which manager approves which cost centre. These live in a database or a spreadsheet, and the right move is a lookup. OpenAI's own research on why language models hallucinate explains why asking a model for this kind of thing is risky: "arbitrary low-frequency facts, like a pet's birthday, cannot be predicted from patterns alone." Your supplier's bank details are exactly that kind of fact.

The rule is already written down#

"Invoices above the threshold need a second approver." "Refunds requested within 30 days are automatic." If a policy fits on an index card, encode it as an if statement. You can show an auditor the exact line, change it in one place and know that every case after the change follows the new rule.

You need the same output every time#

People assume setting temperature to zero makes a model deterministic. In practice it often doesn't. Researchers at Thinking Machines sent the same prompt 1,000 times to an open-weight model at temperature 0 and got 80 different completions. They traced most of it to the server batching requests differently depending on how busy it was, which an API customer can neither see nor control. If your process has to produce identical results on a re-run (reconciliations, compliance reports, anything you might need to reproduce later), keep the model out of the part that decides.

The input is already structured#

If the data arrives as a CSV, a JSON webhook or form fields, you don't need language understanding to read it. Mapping field A in one system to field B in another is plain integration work. Putting a model in the middle adds a step that can drop or invent values in data that was clean to begin with.

Where a model earns its place#

None of this means LLMs are a fad. They're very good at the parts of a process that rules handle badly: free-text emails, PDFs from fifty suppliers with fifty layouts, customer messages that say the same thing a hundred ways, and first drafts that a person will edit anyway.

There's also a point where rules stop being the simple option. Zinkevich's third rule is "Choose machine learning over a complex heuristic," because "a complex heuristic is unmaintainable." If your email-routing rules have grown to 400 keyword patterns with exceptions to the exceptions, that's a sign the task has more variety than rules can hold, and a model (or a conventional classifier trained on your own labelled data) may now be the simpler choice.

A rough way to tell the two cases apart: if you could hand the step to a new starter as a written checklist and expect identical results from everyone, it's a rules job. If you'd have to say "use your judgement" and show them twenty examples, a model might help.

The hybrid pattern most good automations end up with#

In practice the answer is rarely all-or-nothing. Take the invoice example again and split it into steps.

1. Read the incoming PDF or email and pull out supplier name, invoice number, date, line items and total. This is messy, varied input, so it's a reasonable job for a model.
2. Validate what came back. Does the total equal the sum of the line items? Is the date a real date and not in the future? Does the supplier exist in your records? That's code.
3. Look up the matching purchase order by number. Database query.
4. Compare invoice and order amounts against a tolerance you set. Code.
5. Apply the approval policy. Rules.
6. Anything that fails a check goes to a person, along with the reason it failed.

The model handles one step out of six, and every other step checks its work. That structure lines up with what the OWASP entry on misinformation recommends. OWASP defines the risk as models producing "false or misleading information that appears credible", and lists automatic validation, cross-verification and human oversight among the mitigations. It also cites Air Canada, whose chatbot gave travellers wrong information and led to legal trouble for the airline.

The arrangement also makes failures readable. When something goes wrong you can see whether the extraction misread a field or the policy check rejected a valid invoice, instead of puzzling over one long model transcript.

Let the model say "I don't know"#

If a step does need a model, design it so that declining to answer is an acceptable result. OpenAI's hallucination research argues that common evaluations reward guessing over admitting uncertainty, and that accuracy will never reach 100% because some real-world questions are inherently unanswerable. Their published comparison makes the trade-off concrete. On the SimpleQA benchmark, OpenAI's o4-mini abstained on only 1% of questions and gave a wrong answer on 75% of them. The newer gpt-5-thinking-mini abstained on 52% and was wrong on 26%. Their accuracy was close (24% and 22%). Their error rates weren't.

For an automation, that suggests a practical rule: give the model an explicit way out (an "unclear" category, a null field, a "needs review" flag) and route those cases to a person. A system that hands back a third of its inputs with an honest "not sure" is far easier to live with than one that confidently fills in a wrong bank account number.

A simple way to decide#

Before building, write each step of the process on its own line. Next to each one, note whether it involves a calculation, a stored fact, a written rule, a need for repeatable output, or already-structured data. Those steps get code. What's left is your candidate list for a model, and it's usually shorter than people expect.

When we scope automation work at TheAILAB, this list is often the most useful thing to come out of the first meeting, because it shows how much can be built with plain code before any model is involved.

Then build the rules-only version first, even if it only handles the easy cases and sends the rest to a person. Run it for a couple of weeks. The cases it can't handle become your test set for the model, and you'll know exactly what the model needs to be good at, and how you'll check it.

Sources

1. Rules of Machine Learning: Best Practices for ML Engineering — Martin Zinkevich, Google for Developers (checked 2026-10-15)
2. Building effective agents — Erik S. and Barry Zhang, Anthropic (19 December 2024) (checked 2026-10-15)
3. Why language models hallucinate — OpenAI (5 September 2025) (checked 2026-10-15)
4. Defeating Nondeterminism in LLM Inference — Horace He and Thinking Machines Lab (10 September 2025) (checked 2026-10-15)
5. LLM09:2025 Misinformation — OWASP Gen AI Security Project (checked 2026-10-15)