How to measure time saved by automation without fooling yourself

Every automation comes with a number attached. "Saves 12 hours a week." Sometimes a vendor dashboard produces it, sometimes the person who built the thing does, and it usually comes from the same sum: runs completed multiplied by the minutes each run supposedly replaced.

That sum is almost always too generous. It ignores the cases the automation hands back to a person, the time spent checking its work, the hours spent fixing it when an upstream system changes, and the fact that the team might have got faster anyway. None of this means automation doesn't save time. It means you need a slightly more careful way of measuring it if you want a number you can defend to the person paying for it.

Don't trust how fast it feels#

The strongest reason to measure properly is that people are bad at judging their own speed-ups, including experts with every reason to get it right.

In a randomised trial run by the research group METR in early 2025, 16 experienced open-source developers worked through 246 real issues in repositories they knew well. Each issue was randomly assigned to allow or disallow AI tools. Before starting, the developers expected AI to speed them up by 24%. When AI was allowed, they took 19% longer. Afterwards, they still believed AI had sped them up by 20%.

METR is careful about what this shows. It calls the result a snapshot of early-2025 tools in one setting, says the developers and repositories don't represent most software work, and says it doesn't show AI is unhelpful in general. The point for anyone measuring an automation is narrower and holds regardless of the headline: the participants' own estimates pointed the wrong way. If a capable engineer can be that far off about their own work, a team's sense that "it feels much quicker" is not a measurement.

Take a baseline before you switch anything on#

You can't measure a saving without knowing what the work cost before. The UK Treasury's evaluation guide, the Magenta Book, puts this plainly: collect baseline data before implementation so that a counterfactual can be estimated. It's written for government policy, but the logic transfers directly to a business process.

For an automation, a useful baseline covers a few weeks of normal work, not one unusually quiet Tuesday. For each kind of item the process handles, record:

• how many items came in
• hands-on time per item, ideally timed on a sample rather than estimated from memory
• elapsed time from arrival to done, including waiting
• how often an item had to be redone or corrected
• who did the work, since a senior person's minutes and a trainee's minutes aren't interchangeable

Timing a sample of 20 or 30 real items is usually enough, and far better than asking someone how long a task "normally takes". If you mapped the process before automating it, much of this is already on the map. Write the baseline down somewhere it won't be quietly revised once the results come in.

Count the work the automation creates#

The run-count method assumes every run removes a whole unit of work and adds nothing. In practice an automation moves work around. Some of it disappears, and some new work turns up.

The new work usually falls into four groups. Exceptions are the items the automation can't handle and passes to a person, and they tend to be the harder ones, so they often take longer than the old average. Review is any checking of outputs before they're trusted, including spot checks. Fixing covers broken runs, re-running failed batches and correcting bad outputs after they've gone somewhere. Upkeep is the ongoing maintenance: changing rules, updating prompts, handling an API change, and answering "why did it do that?" questions.

This isn't a hypothetical concern. In a large Danish study of AI chatbot use at work, Anders Humlum and Emilie Vestergaard found that the chatbots had created new job tasks for 8.4% of workers, including some people who didn't use the tools themselves. Someone else ends up checking, correcting or supervising the output, and that time belongs in your sums.

Here's how much difference it makes, using illustrative figures for a monthly accounts payable run:

Both columns describe the same automation. The second number is still a good result, and it's one you can stand behind when someone asks how it was worked out.

Compare against what would have happened anyway#

A plain before-and-after comparison has a weakness: other things change at the same time. Volumes drop in a slow month. A new hire gets up to speed. Someone tidies the supplier list, which makes every invoice easier with or without the automation. The Magenta Book notes that interventions are rarely conducted in isolation, and that an evaluation has to ask whether outcomes were influenced by other external factors.

The standard fix is a comparison group, something that tells you what would have happened without the change. You rarely need a formal experiment to get one. Some practical options:

• Roll the automation out to one team, region or document type first, and compare against the ones still working the old way over the same weeks.
• Alternate weeks or days on and off for a short period, if the process allows it.
• Route a random subset of items through the old process during the first month.

A staggered rollout is how one of the better-known workplace AI studies was done. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the staggered introduction of an AI assistant to 5,179 customer support agents. When people get access at different times, the ones still waiting give you a comparison. They found issues resolved per hour rose by 14% on average, with a 34% improvement for novice and low-skilled workers and minimal impact on experienced and highly skilled ones.

That last finding is useful in its own right. The average hid a big difference between groups. If you only report one number for the whole team, you can miss that an automation helps new starters a lot and senior staff hardly at all, which matters when you decide who should use it and how to train them.

Measure output, not runs#

Notice what that study measured: issues resolved per hour. That's an output the business cares about, measured over time, rather than a count of how often the tool was used. It's a good pattern to copy.

For your own process, pick the output measure that matches the reason you automated it. Invoices paid per staff hour. Tickets closed per week at the same quality level. Days from order to dispatch. Then track a quality measure next to it, such as the error rate, rework or customer complaints, because the easiest way to "save time" is to skip a check that used to catch mistakes.

Usage counts are still worth keeping, since they tell you whether people are using the thing at all. Just don't multiply them by a guessed number of minutes and call the result a saving.

Watch for selection effects#

A subtler trap is that people choose what goes through the automation. When they can, they send it the items it handles well and keep the awkward ones. Compare the two piles afterwards and the automated pile looks fast, partly because it was given the easy work.

METR ran into a version of this when it tried to repeat its study in late 2025. In its update on the experiment design, it reported that a growing number of developers would no longer take part because they didn't want to work without AI, and that in surveys 30% to 50% of developers said they had skipped submitting some tasks because they didn't want to do them without AI. METR concluded its data was "systematically missing tasks which have high expected uplift from AI" and now considers that study's data unreliable for measuring current AI productivity effects. It also listed difficulty logging time when people switch to other work while an agent runs, a problem anyone measuring background automation will recognise.

Two habits help. Decide in advance which items go through the automation, by a rule or at random, not by whoever picks them up. And compare like with like: the time for a given type of item before and after, not the automated pile against whatever was left over.

Decide where the saved time goes#

The last gap is between minutes saved and anything the business can see. Humlum and Vestergaard's Danish study, covering 25,000 workers across 11 occupations, found modest productivity gains, with average time savings of about 3% of work hours, and estimated that only 3 to 7% of those time savings showed up in earnings. Their focus was wages, not one team's process, but the general point applies: saved minutes spread across a day tend to get absorbed rather than turned into extra output.

So before launch, write down what the freed time is for. More volume handled by the same team? Faster turnaround? A backlog cleared? Hours moved to work that was being neglected? Then measure that outcome. If a team saves 17 hours a month and nothing anyone can name changes, it's fair to ask whether the hours were real.

A one-page measurement plan covers all of this: the baseline numbers and how they were collected, the output and quality measures, how items are assigned, the comparison group, the categories of new work you'll log, and what the saved time is meant to achieve. It takes an afternoon to write. At TheAILAB we ask for one before building, because it's much harder to reconstruct a baseline after the old process has gone.

Sources

1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — METR (10 July 2025) (checked 2026-10-20)
2. We are Changing our Developer Productivity Experiment Design — METR (24 February 2026) (checked 2026-10-20)
3. Large Language Models, Small Labor Market Effects — Anders Humlum and Emilie Vestergaard, Becker Friedman Institute Working Paper 2025-56 (May 2025) (checked 2026-10-20)
4. Generative AI at Work — Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond, NBER Working Paper 31161 (2023) (checked 2026-10-20)
5. Magenta Book: Central Government guidance on evaluation — HM Treasury (UK) (checked 2026-10-20)