Where AI helps in marketing, and where it quietly costs you

In July 2025 a research group called METR published the study this field should have run first. Sixteen experienced open-source developers took on 246 real tasks in projects they had worked in for about five years. On a random half of the tasks they were allowed AI assistance. Before starting, they forecast that the tools would cut their completion time by 24%. The measured result was 19% slower. Afterwards, asked how it had gone, they estimated the tools had sped them up by about 20%.

They were wrong by roughly forty percentage points about their own working week. The paper is Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, and its authors are careful to say they cannot entirely rule out artefacts of the setup.

That is a coding study, not a marketing one. Nobody has run its equivalent for marketing work, so we are not going to pretend it transfers cleanly. But the gap it measured is the one we keep walking into: these tools feel fast in a way that is not reliably connected to being fast.

The part we cannot tidy up

We use these tools every working day at Vaakrasa and we would not hand them back. We also cannot show you an honest figure for what they save us, because we did not time ourselves properly before we started, and nor did anyone we could cite. Both of those are true at once, and it irritates us that we cannot close the gap between them. We have stopped trying to resolve it into a position, and we are suspicious of anyone in this industry who has.

What we can describe is the pattern in which tasks have gone well and which have gone badly, over about three years of doing this on client accounts.

Where it is reliably good

The useful cases share a shape. The task is bounded, the raw material already exists, and a person can check the output quickly against something objective.

• First drafts of things you already know how to write: ad variations, product descriptions, an outline for a page you have thought through. The current frontier models (OpenAI’s GPT-6 Astra, Anthropic’s Claude Opus 5.5, Google’s Gemini 3.1 Pro) are all competent here, and which one you pick matters far less than the quality of what you put in.
• Volume work with a written spec. Reformatting 300 product titles, generating alt text, translating a page set, cleaning a messy CSV export. The saving is real because verification is a glance.
• Reading things you would otherwise skip. A quarter of sales call transcripts, support tickets grouped by theme, search terms clustered by intent. This is the use we would defend hardest and the one nobody tries to sell you.
• Getting unstuck. Twenty angles on an offer, of which two are usable. For a task that would otherwise eat an afternoon, that ratio is fine.

Where the cost hides

Review time that eats the saving

If output has to be checked carefully, and it does whenever a customer will read it, then generation was not where the cost sat. Reviewing something plausible and subtly wrong is slower than writing it yourself, because you have to reconstruct the reasoning you skipped. We have watched a client team quadruple its content output and spend more total hours than before.

Confident errors in the things you cannot check at a glance

Prices, delivery times, specifications, regulated claims, statements about what your own service includes. These are exactly the outputs that read as most authoritative. A wrong price on a landing page is not a content problem.

Sameness

If you and your four competitors prompt the same model similarly, you converge on a competent average of what already exists. Average is a poor place to stand when a buyer is comparing options and hunting for a reason to pick one.

The illusion of strategy

Ask a model for a marketing plan and you get something that reads like a plan: channels, phases, metrics, a sensible-looking budget split. It knows nothing about your margins, your capacity in November, your salesperson’s actual close rate, or what your last campaign taught you. Acting on it feels like momentum and often just spreads the same money thinner.

A number we are not going to repeat

You have probably been shown the figure that 95% of enterprise AI pilots fail. It comes from a 2025 report by MIT’s Project NANDA, and it has been quoted at us in meetings as though it were a controlled finding. It is not. It rests on roughly 150 executive interviews, a survey of about 350 employees, and a reading of 300 public deployments, with “failure” defined as not having reached full production. By that definition a pilot that is working and still in pilot counts as a failure.

If a figure about AI productivity cannot tell you who was measured, doing what, against what baseline, it is a mood rather than evidence. The honest version of this article has fewer numbers in it than the dishonest version would.

The test we actually use

Before putting a model on a task, we ask four things.

• Can we write the spec? If we cannot describe what a correct output looks like, the model cannot produce one and we will not be able to judge it.
• How fast is verification? Quick and objective, use it. If checking needs the same expertise as producing, the saving is imaginary.
• What happens if it is wrong and ships? Internal note, fine. Customer-facing claim, price, or anything a regulator could read, a person writes and a named person approves.
• Does this have to sound like us specifically? Positioning and founder voice are the differentiator. Keep them human.

The arrangement we would suggest for a small business is narrow and unglamorous. Start where errors are cheap and feedback is immediate: summarising calls, tidying spreadsheets, drafting internal briefs, sorting inquiries. Feed it your own material, your real objection handling and transcripts of calls that went well, because generic input is the usual cause of generic output. Move toward customer-facing work one task at a time. This is roughly the sequence we work through with clients during a marketing optimisation review, and the unglamorous end of it is where most of the hours come back.

And time yourself, properly, before and after, including the review. Sixteen developers with five years in their own codebases could not estimate this by feel. There is no reason to think we can.