The scans that reported success and stored nothing: how to tell an agent is really working
Our competitor scan recorded 23 successful runs and 440 successful skips. On real brands it had stored no competitor ads at all. What the records showed, why every one of them read as success, how we fixed it, and six questions that tell you whether any agent is really working.
One of the jobs our agents run is a daily competitor scan. Olivia, our creative strategist, reads the ads a competitor is running from Meta's public Ad Library and takes a capture of the competitor's homepage, so the Competitor ads page can show what is running and what changed. On 10 October we read every record that job had ever written. Every one said it had worked.
What the records said
| Record | Count | Marked as |
|---|---|---|
| Scan runs, 10 August to 9 October | 23 | Succeeded |
| Of those, runs whose record says nothing about what they did | 14 | Succeeded |
| Runs skipped because the previous run was still going, 30 September to 1 October | 440 | Succeeded |
| Competitors watched for real brands | 2 | Both marked as ads synced, neither with an error |
| Competitor ads stored for real brands | 0 | |
| Homepage captures stored for real brands | 0 |
So the job had a perfect record and, for the brands that were not demos, nothing to show for it. Nobody was alerted, because nothing had failed. The Competitor ads page showed the sync as current, because the record said it was.
Why every record read as success
There were four separate ways a run could do nothing and still be written down as a success, and the records show all four.
- A run's success meant the code finished, not that anything was read. Fourteen runs from August and September have an empty message. The record cannot say how many competitors they looked at, what they found or what they stored. It can only say they ended.
- A skip was written as a success. On 30 September and 1 October the job was started 440 times while an earlier run was still going. Each start stood down, correctly, and each was recorded as succeeding. A job that did no work 440 times looked healthier than one that worked once.
- "Synced" was stamped whether or not anything came back. Until a rebuild on 1 October, the scan kept a summary of each competitor rather than the ads themselves, and it marked the ads as synced whenever the Ad Library returned a result, whatever that result held. One real competitor's last summary says zero ads, with no error. From the record alone you cannot tell whether that competitor ran no ads or whether the answer was empty. That is the finding: the record could not tell the difference.
- A count is not a copy. The other real competitor's last summary, from 5 October, counts 21 active ads. None of them is stored. That record is in the older summary shape, so the 21 was a number about the ads rather than the ads, and once the scan was rebuilt to keep each ad, there was nothing behind the number to show.
Meanwhile the demo brands we use for previews held 128 stored competitor ads, 18 homepage captures and 9 recorded changes. The page looked full every time we showed it. It was full of demo data, and none of the counts above would have looked wrong to anybody who only ever saw the demo.
The fix, and where the first fix was wrong
The rebuild on 1 October changed what was stored: each ad is kept one by one with when it was first and last seen, each homepage capture is kept, and a change is only reported when it can be built from two stored readings. That made it possible to see that nothing was stored. It did not stop a run that stored nothing from calling itself a success.
The first fix, on 9 October, named the cause in one line: a run was only marked failed when the Ad Library did not answer and the homepage could not be captured, so a run that got neither stored nothing, wrote no error and stamped the ads as synced. That fix made "stored nothing" a failure.
It was wrong the same day. A competitor that genuinely runs no new ads this week stores nothing new, and calling that a failure would have raised an alert every morning for a correct reading. An alert that fires on correct readings is one people stop reading. The rule that replaced it splits the cases the old record had merged:
| What happened | How the run is recorded now |
|---|---|
| Read everything, something new | Succeeded, with what was stored |
| Read everything, nothing new | Succeeded, and says it read nothing new |
| Could not read: the Ad Library did not answer, the answer had no list of ads, the homepage was empty or not captured, or the code failed | Failed, and raises one alert per competitor until a later scan reads it |
| Not run because a rule said so, for example the brand is paused | Skipped, and not counted as a success |
Each of these is pinned by a test that fails on the old behaviour, and the page shows the sync as current only when something has actually been stored. The counts in this post are from 10 October, the day after the fix landed in our code, so they describe the records the old rules wrote.
Six questions that tell you whether any agent is really working
None of this is specific to competitor scans. Any agent that runs on a schedule, ours or anybody's, can fail the same four ways. These are the questions we now ask of every job.
- Count what was stored, not how often it ran. Ask for the number of records the agent has written for your brand. A run count is a statement about a clock.
- Read the message, not the status. A run should say what it examined and what it found: so many competitors due, so many read, so many with nothing new, so many failed. A blank success is not information.
- Make sure a skip cannot pass as a success. "Did not run" and "ran and found nothing" need different words on the record, or a stalled job looks like a quiet one.
- Separate "nothing new" from "could not read". The first is a result. The second is a failure. A tool that shows the same empty state for both cannot tell you when it is blind.
- Check that "last synced" only moves when something was read. A timestamp that advances on an empty answer turns every failure into a fresh-looking page.
- Look at your own data, not the demo. If every example of the feature working is a demo account, the feature has not yet been shown working on yours.
The same questions apply to changes an agent makes as well as to things it reads. That is why every change in Cresva writes a record before it is attempted and, where the platform allows, reads the account back afterwards, as set out in what agents should be allowed to do in an ad account. A scan that reads nothing is an annoyance. A change that reports success and did nothing, or did something else, is the version that costs money.
See what the agents read and wrote for your brand Every change an agent makes is recorded before it runs, and every scan now says whether it read something, read nothing new, or could not read.
Frequently asked questions
How can I tell whether a scheduled AI agent is actually working?
What is the difference between "nothing new" and "could not read"?
Where does Cresva read competitor ads from?
Written by Shubham Raghav, Founder, Cresva
Have a question? Email us