PilotOpen protocol for stores.Explore ACP
Skip to paper
Research

CRV-2026-001

Proof of Write: An Evidentiary Standard for Agent Action Capability

Authors
Shubham Raghav
Published
August 5, 2026
Identifier
CRV-2026-001
Version
1.0

Abstract

Agentic products describe their ability to act on external systems in the language of breadth: platforms connected, verbs shipped, integrations closed. Breadth answers a question nobody asked. Running one hostile, evidence-gated audit against one system on one day produced four different coverage figures from the same 130-capability matrix, differing only in what the auditor accepted as evidence: a breadth narrative for which no value was ever computed, 56.6% for code that exists and looks contract-correct, 5.4% once a requested permission scope had to be enforced before the call rather than merely requested at consent, and 0.0% for writes proven to have reached a vendor. We propose an evidentiary standard that resolves the ambiguity: seven tiers, each with a criterion a third party can apply without access to the code, the artifact that satisfies it, and who is able to verify it. We then apply the standard to the system it was derived from. The top-tier count is zero.

1. Introduction

An agent that can act on a system is worth more than an agent that can only describe what should be done, and the industry has settled on breadth as the way to say so. Platforms connected. Verbs shipped. Integrations closed. None of those statements is false, and none of them answers the question a buyer, an auditor, or a regulator is actually asking, which is whether a particular action, on a particular account, has ever been observed to happen.

In July 2026 an evidence-gated audit was run against the system described in this paper. Its posture was hostile by construction: the default verdict for every capability was failure, and a capability moved up only on a file and line reference. The corpus was 130 canonical write capabilities across six vendor surfaces, weighted by tier so that money-moving operations counted for more than configuration ones.1

The audit produced four coverage figures. Not four estimates of one quantity, and not the output of four methods. Four figures computed from the same matrix by the same script, differing only in the definition of evidence supplied to it.

Counted as breadth, coverage was a narrative rather than a number. The internal record is explicit that the figure quoted in commercial material was a paraphrase of the closed-platform and verb-count story, and that no value of that kind was ever computed from the capability map.2

Counted as code that exists, is reachable from a real trigger, is authorized, and whose vendor contract appears correct, coverage at the audit commit was 56.6%.3

Counted with the additional requirement that the permission scope be enforced at the writer boundary before the call, rather than merely requested at consent and stored, the same corpus scored 5.4%. On one platform the figure moved from 73.7% down to 1.3%, and the distance between those two numbers is precisely one unshipped check.4

Counted as writes proven to have reached a vendor and returned a success status, coverage was 0.0%, on every platform and every tier.5

One capability map, four coverage figuresFour horizontal bars for the same corpus of 130 capabilities. Breadth is drawn as a hatched band because no value was ever computed for it. Plausible coverage reads 56.6 percent, enforcement-required coverage reads 5.4 percent, and proven coverage reads zero.SAME SYSTEM · SAME 130 CAPABILITIES · SAME DAYBreadthplatforms closed, verbs …no valueplatforms closed, verbs shippedPlausiblebuilt, reachable, author…56.6%built, reachable, authorized, contract looks correctEnforcedscope refused pre-flight…5.4%scope refused pre-flight, not merely requestedProvenrecorded live vendor suc…0.0%recorded live vendor successPLAUSIBLE OVER TIME56.6% → 95.9% → 86.1%the fall is the standard tightening, not capability lost
Figure 1. One capability map, four coverage figures. The breadth bar is drawn hatched and unlabelled because no value of that kind was ever computed; the other three are recomputed values from the deterministic score scripts named in the notes. The inset trajectory shows what happened to the plausible figure across two later engineering sprints.

The trajectory in that figure carries the paper's second argument. Four remediation sprints moved the plausible figure from 56.6% to 95.9%. The next sprint moved it down to 86.1%, because one platform's writers were reclassified as vendor-blocked after a scope-matching defect made it impossible for any of them to execute. Nothing was deleted and no capability was lost. The measurement got stricter, so the number got worse, and it got worse by getting truer.6

We take three positions in this paper. First, that action capability without a stated evidentiary basis is not a claim at all, because any figure between a breadth narrative and zero can be defended from the same code. Second, that the evidence classes the field currently accepts, including passing tests and a clean typecheck, fail in ways that are specific, reproducible, and worth naming individually. Third, that a standard is only worth proposing if its author is willing to be graded by it first, which is what Section 7 does.

2. Scope and terminology

The definitions below are numbered so later sections can cite them precisely. They are narrower than ordinary usage on purpose: most of the ambiguity this paper is about lives in words that felt unambiguous until somebody tried to grade them.

D1. Action. A single intended mutation of state held by a system outside the acting party's own datastore, addressed by a named verb. Changing a campaign budget is an action. Recording an intention to change it is not, and neither is flipping an internal flag that a scheduled job will later read.7

D2. Actor. The principal on whose authority an action is issued. Three are distinguished throughout: a human operating a surface directly, a human approving a proposal that software composed, and software firing on a schedule with no human in the loop at that moment. An evidentiary record that does not name the actor cannot support any claim about autonomy, because the strongest and weakest cases produce identical rows.

D3. Surface. The fully-qualified vendor entity an action addresses, including the API version. Not the endpoint family and not the resource name in isolation. A later section explains why this precision is load-bearing: in one vendor's schema, three distinct messages share a single unqualified name, and a field verified against the global vocabulary can be absent from the specific message a writer actually sends.

D4. Write mode. What the terminal call actually did. Four values are distinguished here, and the fourth is the one usually missing: live, meaning the vendor was called; shadow, meaning a real vendor call was deliberately suppressed; test, meaning a live call against a disposable target; and local-configuration, meaning the verb has no vendor call in its definition at all. Collapsing the fourth into either of the first two corrupts any later count of executions, in opposite directions.8

D5. Reversibility class. A declared property of a verb, with three values. Full means an inverse verb exists that restores the prior state. Semi means an inverse exists that stops the effect without restoring what it consumed, which is the correct grade for undoing a duplicate that already spent money. None means no programmatic inverse is shipped, which describes the acting party's capability rather than the vendor's. Two obligations attach to the class: a verb graded none requires an explicit second-stage confirmation before it may fire, and a verb that creates a new entity never fires automatically regardless of its class.9

D6. Idempotency. A property of the action's lifecycle, not of the request. A duplicate-suppression check that only recognises completed prior attempts leaves a window open between the moment a vendor call is issued and the moment its result is recorded, and a crash inside that window makes a retry look like a first attempt. The correct scope of the check is every state an attempt can be in, including in-flight and unresolved.10

D7. Requested scope against enforced scope. A requested scope is one the acting party asked for at consent. A stored scope is one a provider said it granted. An enforced scope is one that will cause the acting party's own code to refuse the call before it is made. These are three different things, only the third has any effect on what actually happens, and a system can hold the first two for years without holding the third.

D8. Evidence artifact. A durable record that a specified party can inspect without trusting the acting party's narration. The specification of who can inspect it is part of the definition. A log line the vendor cannot see, a screenshot the reader cannot re-derive, and a test the author wrote against their own implementation are all artifacts in the loose sense and none of them is an evidence artifact in this one.

D9. Grade. The highest tier in Section 4 whose criterion a given capability satisfies, together with the artifact that satisfied it. A grade is per capability, per surface, and per point in time. It is not a property of a product.

3. Why existing evidence fails

Each subsection below names an evidence class in current use, the mechanism by which it fails, and an instance from our own record. Every instance is a defect we shipped. That is the only reason they can be described this precisely.

3.1 The demonstration

A recorded demonstration shows a user interface reaching a state. It does not show a vendor receiving a request, because the interface is downstream of the acting party's own code and renders whatever that code returns. Our write layer is gated by a per-platform mode switch whose default suppresses the vendor call and returns a tagged envelope. That envelope is deliberately explicit, and it logs at critical severity rather than pretending, but a surface that renders the result as success would look identical on video to one that changed a real account. A demonstration is evidence about a rendering.

3.2 The sandbox run

A sandbox run is a real execution and is a genuine improvement on a demonstration. Its limit is scope: it establishes that the code path works against the sandbox, and vendor sandboxes routinely differ from production in permission model, rate limiting, and object lifecycle. A sandbox result is admissible evidence for the tier it supports and is not evidence of production capability. The standard in Section 4 therefore treats the target as part of the artifact rather than as a footnote to it.

3.3 The self-reported log

A log written by the acting party is evidence only to the extent that its fields are hard to write incorrectly. Two of our verbs stamped an execution mode of live on operations that made no vendor call whatsoever: they flipped an internal flag that a scheduled job would later act on. Any query asking how many live writes had occurred was counting them. The fix was a distinct mode value for the case, chosen deliberately over reusing the shadow value, because shadow asserts that a real write was suppressed and nothing had been suppressed.8

The second failure of self-reported logs is subtler and we have three instances of it. A stored vendor response that reads as a bare success object, with no vendor-assigned identifier and no request id, is indistinguishable from an envelope the application synthesised for itself. Under our own standard those rows grade as ungradeable, and the backfill script we wrote to classify historical rows leaves them null rather than guessing.11

3.4 The passing test suite

This is the failure with the highest ratio of confidence to information, and it is the one we most want other teams to reproduce on their own code.

Our integration code, written with LLM assistance, called a Klaviyo endpoint that has never existed. The path it used was a send sub-resource on the campaign object. The real operation is a separate top-level resource, scheduling is a two-step flow that first patches a send strategy onto the campaign and then creates the send job, and the vendor documents no idempotency header for it, so duplicate suppression has to be built at the application level for an irreversible operation. This was our defect and not the vendor's: the documentation was correct and public the whole time, and the code did not consult it.12

The suite was green throughout. It was green because the transport mock was shaped to the implementation rather than to the vendor. A mock that resolves successfully regardless of the path it is handed blesses every path, including paths that do not exist. We classified our own writer tests against this criterion and found the class was not rare: two Meta test files could not have caught an invented endpoint, one because it asserted only that the transport had been called and the other because it asserted request bodies while never asserting the URL.13

Mirror-class mock against vendor-surface mockTwo panels. On the left a transport mock shaped to the implementation returns success for an endpoint that does not exist, and the test passes. On the right a transport mock shaped to the published vendor surface refuses the same call and the test fails.THE SAME INVENTED ENDPOINT, PUT TO TWO TESTSMirror-class mockanswers every path the same waywriter calls /campaigns/1/jobs/sendtransport.mockResolvedValue({ ok: true })shaped to the implementationPASS (endpoint does not exist)Vendor-surface mockanswers only documented pathswriter calls /campaigns/1/jobs/sendcase /api/campaign-send-jobs: return okshaped to the vendor surfaceFAIL CONTRACT VIOLATION
Figure 2. The same invented endpoint under two mock shapes. The left panel is not a weaker test of the same thing; it is a test of a different thing, namely whether the implementation calls what the implementation calls.

The mutation test. The remedy is a mock shaped to the published vendor surface, and the only way to know it is shaped correctly is to break the implementation on purpose and watch the test notice. The procedure below is reproducible on any codebase with a single transport chokepoint per vendor.

  1. Identify the one function every writer for a vendor calls to reach the network. If there is more than one, that is the first finding.
  2. Fetch the vendor reference for each endpoint the writers use, and record the URL and the retrieval date alongside the writer.
  3. Replace the transport mock with one that answers only the paths and methods named in that reference, and that throws on anything else with a message containing a fixed, greppable marker.
  4. Run the suite. It must pass. If it fails here, the implementation is already calling something undocumented, and that is the whole finding.
  5. Change one writer's endpoint to a path that is plausible and wrong. Prefer a real endpoint of the same vendor over an invented one, because the interesting failure is a wrong real path rather than a typo.
  6. Run the suite again. It must fail, and the failure message must contain the marker and name the wrong path. If it passes, the test is mirror-class and proves nothing about the endpoint.
  7. Revert the change and confirm the working tree is clean before the result is recorded.

Run against our own writers, this produced the following. Changing the pixel-creation endpoint to a plausible neighbour caused two tests to fail with the marker naming the wrong path. Changing the campaign-creation path to a legacy name caused a failure. On a second vendor, changing an inventory write to a different and genuinely real endpoint of the same vendor produced three failures. On Klaviyo we re-introduced the program's actual historical invention, the send sub-resource, and it now fails loudly where the identical change had previously stayed green. Each change was reverted and the empty diff verified.14

The rule this yields is short. Shape the transport mock to the vendor surface, never to whatever the implementation happens to call.

3.5 The clean typecheck

A type system checks the code against its own declarations. When the declarations are generic, it checks almost nothing about the vendor. Upgrading our Google Ads client across three major versions renamed two fields on the campaign resource, replacing a start date and an end date with date-time equivalents carrying new field numbers. The compiler reported no errors, because the mutation helper is generic in its resource type and accepts any object literal. One writer would have created a campaign whose start date was silently dropped, so the campaign would have begun serving immediately rather than on the requested day. The other selected the removed field in a query and would have failed outright before reaching the mutation.15

The first verification pass missed this. It checked field names against the vendor's global vocabulary for that version, and both removed names still exist on other message types, so the check came back clean. A bare name lookup also lies in a second way: three distinct messages in that schema share the unqualified name Campaign. The method that survives is verification against the fully-qualified message or endpoint, never against the vocabulary, and never with a clean compile standing in for it. Re-run that way, the audit covered 22 of 22 mutate entity names, 7 of 7 service methods, 26 of 26 enum types, 0 renumbered values across all 348 enums, and 77 of 77 field names in use.15

3.6 The requested OAuth scope

A consent screen records what was asked for. It does not record what will happen when the code runs. Our writers requested and stored an ads-management scope on one platform for months without any check that refused a call when the connection lacked it, so a connection that had consented before the scope was added would attempt the mutation and receive a refusal from the vendor, while the operator saw a success. The shared pre-flight that closes this shipped later, wired into every write-preparation chokepoint and failing closed on a read error.16

Reading a granted scope back turns out to be its own family of defects, and the four we found do not share a root cause, which is the point.17

  • A sentinel matched as a substring. One platform's check tested whether the granted-scope list contained a short digit string. Any scope identifier containing that digit satisfied it, so the check passed for connections that had never been granted anything.
  • A comparison that can never hold. The same platform is documented to return numeric scope identifiers, which would be stored as strings and compared against scope names. That mismatch is documented behaviour we have not observed on a real token, and we say so rather than fixing it against identifiers we would have to invent.
  • A readback list shorter than the consent request. A third platform's readback omitted two of the scopes its own consent flow asks for, so the interface reported a store as write-capable while it lacked them.
  • No readback at all. A fourth vendor publishes no endpoint that returns an API key's own scopes, which leaves inventing one, probing with a real write, or asking a human. We chose the third, and store it as an attestation carrying who recorded it and when, deliberately kept out of the field that means the provider told us this.
Evidence classHow it failsInstance from our record
Demonstration or screenshotShows a rendering, which is downstream of the acting party's own codeShadow mode suppresses the vendor call and the surface still renders a result
Sandbox runEstablishes the sandbox, which differs from production in permissions and lifecycleTreated as tier-limited evidence rather than proof of production capability
Self-reported logFields that are easy to write incorrectly, and envelopes indistinguishable from synthesized onesTwo verbs stamped live with no vendor call; three stored responses ungradeable
Passing unit testsA transport mock shaped to the implementation blesses any path, including one that does not existAn invented Klaviyo send sub-resource stayed green until the mock was reshaped
Clean typecheckGeneric resource parameters carry no vendor schema, so removed fields are invisibleTwo renamed campaign date fields survived a three-major upgrade with zero compiler errors
Requested OAuth scopeConsent is not enforcement, and readback has its own failure familyA scope requested and stored for months with no pre-flight refusal anywhere
Table 1. Evidence classes in current use, the mechanism by which each fails, and the instance from our own record that demonstrates it.

4. The standard

The ladder below has seven tiers. Each is stated as a criterion a third party can apply without access to the code, the artifact that satisfies it, and the party able to verify that artifact. Tiers are cumulative: a capability graded at a tier satisfies every tier beneath it. A grade attaches to one capability on one surface at one time, never to a product.

The evidence ladderSeven tiers from T0 Asserted at the bottom to T6 Confirmed at the top. A boundary between T4 Governed and T5 Executed separates the tiers a team can reach by writing code from the tiers that require a recorded execution.STRONGESTT6Confirmedpost-state read back, reversal exercisedT5Executedvendor response persisted with status codeT4Governedscope enforced, duplicates suppressed, magnitude cappedT3Contract-verifiedendpoint checked per message, mutation test bindsT2Reachablea path from an external trigger to the writerT1Implementedcode names a vendor endpoint and versionT0Asserteda claim with no artifactABOVE THIS LINE, CODE IS NOT ENOUGHWEAKEST
Figure 3. The evidence ladder. The dashed boundary is the load-bearing feature: everything beneath it can be reached by writing code, and nothing above it can.

Tier 0, Asserted. A capability appears in a claim with no artifact behind it. Criterion: none. This tier exists so that the ungraded case has a name, because in practice most published capability claims sit here and are not labelled.

Tier 1, Implemented. Code exists that names a specific vendor endpoint, method, and API version for the capability. Artifact: a published manifest listing, per capability, the endpoint, the method, the version, and where the version is sourced from. A version written as a literal in the code rather than read from configuration is a defect at this tier, because it is a scheduled failure with no owner. Verifier: anyone, once the manifest is published; the manifest is checkable against the vendor's reference without seeing the code.

Tier 2, Reachable. A path exists from an external trigger to the writer, and it is the path a user would actually take. Artifact: the trigger chain, named end to end. Verifier: anyone with access to the product surface. A writer with no trigger is a library, and a great deal of published capability is exactly that.

Tier 3, Contract-verified. The endpoint, method, and request fields have been checked against the vendor's published reference for the fully-qualified message, and a mutation test demonstrates that the check binds. Artifact: the reference URL with its retrieval date, plus a recorded mutation result showing the test failing on a deliberately wrong path and passing after revert. Verifier: anyone who can re-run the procedure in Section 3.4. This is the lowest tier at which a test result means anything, because it is the first tier where the test is shaped by something other than the implementation.

Tier 4, Governed. Four conditions, all of which precede the vendor call. 4a, scope enforcement: the required permission scope is enforced at the writer boundary and refuses before the call, failing closed when the scope cannot be read. 4b, duplicate suppression: the check covers every state a prior attempt can be in, not only completed ones, and an unresolvable in-flight attempt blocks rather than proceeds. 4c, magnitude ceiling: a limit applies to the operation, expressed in the unit the operation moves rather than in a count of operations. 4d, explicit grant: the capability is granted to each actor per verb, because a pattern-matched grant admits a new verb silently the moment it is registered. Artifact: a refusal record for each condition, produced by exercising it. Verifier: anyone able to present a deliberately unscoped or duplicate request and observe the refusal.

A capability that satisfies some of the four but not all of them grades at Tier 3, with the satisfied sub-conditions listed alongside the grade. Partial governance is worth recording and is not worth a tier, because the conditions do not substitute for one another: a magnitude ceiling does not help a write that should have been refused for scope, and an enforced scope does not stop a duplicate.

Tier 5, Executed. A vendor response has been received and persisted with its transport status code, in a field that cannot be confused with an application-generated envelope, alongside the actor, the timestamp, the write mode, and the fully-qualified target. Artifact: the persisted row. Verifier: an auditor with read access to the record, or a counterparty who can be shown it. A row that omits any of those fields grades ungradeable, which is a distinct outcome from failing and must be reported as its own category.

Tier 6, Confirmed. After the write, an independent read call confirms the intended state, and the declared reversal has been exercised against a real entity and confirmed the same way. Artifact: the pre-state, the post-state, the reversal, and the post-reversal state, all captured by read calls rather than inferred from the write response. Verifier: an auditor, or the account holder. A reversibility class that has never been exercised is a design intention, and a rollback path that has never run is the least trustworthy code in any write system.

Two properties of this ladder matter more than the tier names. The first is the boundary in the figure: Tiers 1 through 4 can be reached by writing code, and Tiers 5 and 6 cannot be reached by any amount of it. A team can work for months and move nothing above the line. The second is that the tiers above the line are the only ones a counterparty can verify without trusting the acting party's description of its own internals, which is why they are the only ones worth putting in a claim.

The standard also settles what counts as a write at all, which sounds pedantic until a count depends on it. An operation that changes only the acting party's own state is not an action under D1, cannot reach Tier 5, and must be recorded in a mode value distinct from both live and suppressed. Any capability count that includes such operations is measuring something else.

5. Verification procedure

The following establishes Tier 6 for exactly one capability, on one surface, once. It is deliberately boring and deliberately narrow. Producing this record for a capability is cheap; the reason it is rare is that it cannot be done by the people who wrote the code without also being done by somebody who controls an account.

  1. Nominate a disposable target: an account, an entity, and where relevant a property, that nobody depends on. Record its identifier in configuration, never in the writer, so the writer is the same code that will run against a real account.
  2. Establish the token's scope. Read the granted scopes from the provider and record the response. Where the provider exposes no such endpoint, record an attestation naming the person, the source they read, and the time, and store it in a field distinct from provider-reported scopes so the two can never be confused later.
  3. Capture the pre-state with a read call against the target, and persist it. Do not derive it from local state, which is the copy most likely to be wrong.
  4. Issue the write. Persist the request path, method, and API version actually used, resolved at call time rather than transcribed from the source.
  5. Persist the vendor response verbatim, with its transport status code, in a field reserved for it. If the response carries a vendor-assigned identifier, persist that separately, because it is the field that later distinguishes a real response from a synthesized one.
  6. Read the post-state back with an independent read call and compare it against the intent. A write response that echoes the request is not a confirmation.
  7. Exercise the declared reversal against the same entity, then read back a third time and confirm the prior state was restored, or record precisely what the reversal did not restore.
  8. Grade the resulting record. Actor, timestamp, write mode, status code, pre-state, post-state, reversal, and target must all be present. If any is missing, the grade is ungradeable, and ungradeable is reported as its own outcome rather than folded into either success or failure.

Two notes on the procedure. The reversal in step seven is the step most often skipped, and it is the one that converts a reversibility class from a declaration into a fact. The grading in step eight has to admit a third outcome, because a system that reports only success and failure will silently classify its own missing evidence as one or the other, and which one it picks tells you about the reporter rather than the write.

6. What the standard cannot establish

The standard grades whether an action happened and whether it can be shown to have happened. It is silent on four questions that matter at least as much, and reading a grade as an answer to any of them is a misuse.

It says nothing about frequency. A capability graded at the top tier has been proven once. Once is the entire content of the claim, and a system with one confirmed write and a system with a million of them earn the same grade on that capability.

It says nothing about safety. A confirmed write is a write that happened and was recorded, which is orthogonal to whether it should have. The governance conditions at Tier 4 constrain the blast radius of a bad decision; they do not evaluate the decision.

It says nothing about economic value. Whether an action improved anything is a question of measurement design, counterfactuals, and time horizon, and none of that is in scope here.

It says nothing about unattended behaviour. Every tier is defined for an action, and an action has an actor. A system that has proven a capability under human approval has proven exactly that, and the case where no human is present is a different claim requiring different evidence, which we do not attempt to specify.

7. Applying the standard to Cresva

We wrote the standard, so this section is where it is worth the least and has to be done anyway. Everything below is graded by the ladder in Section 4 against the state of the repository at the time of writing.

The count of capabilities at Tier 5 or above is zero. No capability in our write surface has a persisted vendor response with a transport status code tied to an actor and a target. Not one, on any of six platforms. Every capability we ship sits at Tier 4 or below, which means every claim we could honestly make about it is a claim about code rather than about anything that has happened.

That is the finding, and the rest of this section is context for it rather than mitigation of it.

7.1 What the enforcement collapse measured

The 5.4% figure in Section 1 was a recompute of the same matrix under one changed rule: a capability earns credit for authorization only if the writer refuses a mis-scoped connection before calling the vendor. Under that rule, three of the six platforms scored 0.0%, one fell from 73.7% to 1.3% on the strength of a single writer that enforced its own scope, and the two that were unchanged were unchanged for opposite reasons, one because it already failed closed and one because its vendor has no scope layer to drift.4

That gap is now closed in code by a shared pre-flight at every write-preparation chokepoint.16 It is not closed in evidence, and under this paper's own standard those are different states. The refusal path has unit tests; it has never refused a real mis-scoped connection in production, because nothing has been proven in production.

7.2 The cost of failing closed

Failing closed is not free and we paid for it in the expected direction. The scope pre-flight read a top-level column that the OAuth callback had never populated, because the callback wrote the granted scopes into a metadata field instead. Every Meta connection therefore read as having no scopes at all, and on an armed platform the pre-flight would have refused every write, including all the correctly scoped ones. The fix was a resolution chain across the places the data actually lived, plus a canonical write going forward and a backfill for existing rows.18

We record this as the correct failure. A control that refuses when it should have proceeded costs an operator an approval and a support conversation. The same control failing open costs a customer's account. A standard that demands fail-closed behaviour should be honest that it is buying the first outcome with the second, rather than pretending the first does not happen.

7.3 Three writes we cannot grade

The production action-audit table is small enough to read row by row, and at the time of the migration verification it held five rows.11 Three of them carry a stored vendor response that reads as a bare success object with no vendor-assigned identifier. One of those three is a write against a real campaign whose captured pre-state carries a lifetime-budget value of a shape only a real vendor would return, which is suggestive and is not evidence. All three were issued against internal targets rather than customer accounts.

Under the standard in Section 4 those three rows grade ungradeable, not executed. We could argue for them. The pre-state looks real, the timing lines up with a period when one platform was configured live, and the operations were the kind that would have succeeded. Every one of those is an inference about our own system offered by us, which is the precise thing D8 excludes. The backfill script that classifies historical rows leaves them null and says so in its output rather than guessing, and this paper does the same.

There is a second-order finding here. Our own audit trail was, until recently, capable of recording a write in a way that could not later be graded, and nobody noticed because nothing queried it for grading. An evidence system that is never queried degrades silently.

7.4 The auditor is not exempt

Two of our own audit findings were wrong, in both directions, and both were caught by reading code rather than documents.

The first was pessimistic. An audit reported that the spending guardrail was a count cap rather than a currency cap and that it failed open on a database read error, and a remediation sprint was scoped to rebuild it. Inspection of the evaluator showed a cumulative per-day currency cap with a binding default, read strictly, enforced on every path including the scheduled one. The finding had been carried forward from an internal document that predated the fix. The sprint was rescoped to extend the existing control rather than rebuild it, and the stale documents were flagged for correction.19

The second was optimistic, and it is the more instructive one. The original audit concluded that production arming was unverifiable from the repository, and scored a platform accordingly. It was verifiable, from the deployment platform's environment listing, which the audit did not consult. The correction is recorded in an addendum rather than folded into the original text, because an audit that silently rewrites its own conclusions is not an audit.20

The rule both cases produce is the same one that governs Tier 3. Documentation is not evidence, in either direction. A document asserting a defect and a document asserting a fix are the same class of artifact, and the code and the deployment are the authorities.

7.5 Where each platform actually stands

Every remaining blocker is an operator action or a vendor observation. None of them is a code change, which is a comfortable thing for an engineering team to say and is also why the count at the top tier has stayed at zero: the work that remains is not the work we are good at.21

SurfaceHighest tier reachedWhat stands between it and Tier 5
MetaTier 4Nominate a disposable ad account, campaign and pixel, then approve one reversible proposal and read the resulting row
Google AdsTier 4Re-consent the connections whose scope provenance is unprovable, and nominate a test customer and base entities
ShopifyTier 4Every connected store must re-consent, because four scopes were added and this vendor never expands a grant silently
KlaviyoTier 4Record the key's scopes as an attestation, since the vendor publishes no endpoint that reports them
GA4Tier 4Nominate a writable test property and record the measurement credential against it
TikTokTier 1Capture one real token response and read its scope value; the writers are vendor-blocked until the identifier shape is known
Table 2. Self-assessment by surface at the time of writing. TikTok sits lower than the others because its scope comparison cannot succeed against the identifier shape the vendor documents, and the mapping cannot be written correctly until a real response has been observed.

One reading of this table is that the system is nearly there. That reading is available because the table stops at the boundary in Figure 1, and the boundary is where the difficulty is. A second reading is the one we hold: the table describes a write engine whose ignition has not been observed to turn over, and the standard exists so that the first reading cannot be published without the second.

If this section reads as favourable, the criterion is wrong. That is not modesty. A standard authored by a participant is only useful if it can produce an unflattering result about the author, and the test of whether it can is whether it did.

8. Related work and open questions

The agent evaluation literature measures task success: given a goal and an environment, did the agent reach the goal state. WebArena builds functional websites across four domains and scores functional correctness of the completion.22 SWE-bench draws its tasks from real repository issues and scores whether the resulting patch passes the project's tests.23 GAIA scores answers to questions built to be easy for a person and hard for an assistant.24 tau-bench is the closest of the four to the concern of this paper, because its agents call tools against a domain database and it reports a consistency measure over repeated trials rather than a single pass rate.25 All four are well designed for what they measure, and their reported numbers are sobering on their own terms: the best agent in the WebArena paper completes 14.41% of tasks against 78.24% for humans, and the strongest model in the SWE-bench paper resolves 1.96% of issues.

None of them answers the question here, and the reason is structural rather than a shortfall. In every case the environment is instrumented by the benchmark author, so the side effect and the record of it come from the same source, and the record is true by construction. In production the side effect lands on a system controlled by a third party while the record is written by the agent's operator, and the gap between those two authors is where every failure in Section 3 lives. The two lines of work are complementary: a benchmark grades the decision, and this standard grades the evidence that the decision was carried out. A system could score well on either while failing the other completely, and knowing which one a claim rests on is the whole point.

One published result bears directly on Section 3.4 and is worth stating precisely, because it establishes that the failure described there is a property of the method rather than an anecdote about one team. A study of package hallucinations in generated code sampled 576,000 programs across 16 models and found 205,474 distinct non-existent package names, at an average rate of at least 5.2% for commercial models and 21.7% for open-source ones.26 Its concern is supply-chain risk, which is not ours. The shared mechanism is the one that matters: a model will name a dependency that does not exist, confidently and in syntactically valid code, and nothing downstream of the model objects. An invented API endpoint is the same failure aimed at a different target, and the reason ours survived a green test suite is that the test was written against the same assumption as the code. We know of no comparable published measurement for invented endpoints specifically, which is a gap worth someone filling.

Three questions we cannot currently answer are worth naming.

Multi-actor writes. The ladder assumes one actor per action. A plan that fans out into dependent writes, some approved individually and some inherited from the parent approval, has no obvious grading. Grading the plan hides which leg was proven; grading each leg loses the fact that a human approved only the root. We currently record both and have no principled rule for which one a claim should cite.

Unattended operation. Section 6 declines to specify evidence for it, and the reason is that we do not know what would be sufficient. A confirmed write with no human present proves the mechanism and says nothing about the population of decisions the system would have made. This is the question that matters most commercially and is the one this standard is furthest from answering.

A shared registry. A grade is only useful to a counterparty if the grading is comparable across vendors, and comparability requires a shared vocabulary of capabilities, a shared tier definition, and somebody willing to hold the register. This paper supplies the second of those three. We would rather the first and third existed than that we owned them.

Artifacts. The first version of this paper cited its own numbers to paths inside a private repository, which is a failure of definition D8 by the paper's own terms: a reader outside the company could check nothing. Published alongside it, at cresva.ai/research/artifacts/crv-2026-001, are the capability matrix every figure traces to, the three deterministic score scripts that reproduce those figures from a clean directory with no other inputs, and the verbatim passage behind every footnote that cites an internal document. Identifiers were replaced with stable placeholders before publication: account and record ids, credentials, infrastructure names, and the display name of any customer or brand. The substitution map is withheld, because publishing it would reverse the redaction in one step, and the bundle README states the policy and its limits in full. The full audit report and the full sprint records are not published, so a reader is entitled to note that the party choosing which passages to excerpt is the party being audited.27

One closing observation about the trajectory in Figure 2. The number went down when the measurement improved, and there was a moment where the honest reporting of that felt like a regression to explain rather than a result to publish. Any standard that a participant applies to itself will produce that moment. Whether it survives it is the only test of the standard that matters.

Notes

  1. 1.Audit report at `audit/write-access/REPORT.md`, run against repository HEAD `2f1dbc36c`, audit date 2026-07-29. Corpus of 130 canonical capabilities across Meta, Google Ads, Shopify, Klaviyo, TikTok and GA4, tier-weighted three, two and one. Per-capability rows in `audit/write-access/matrix.json`; raw per-platform traces in the sibling EVIDENCE files. Public copy: artifact bundle, note 1. The matrix itself is published at matrix.json. ↩
  2. 2.`audit/write-access/PROGRESS.md` records the breadth figure as a paraphrase of the closed-platform and verb-count narrative rather than a computed value, and the report's claim-versus-reality table records that the literal figure appeared in commercial material rather than in the code documentation. Public copy: artifact bundle, note 2. ↩
  3. 3.Recomputed at authoring time by re-running `audit/write-access/compute.mjs`, which prints 56.6% overall and 68.8% for the money-moving tier. Per platform: 73.7%, 70.8%, 61.7%, 45.7%, 34.8%, 15.4%. Note a discrepancy in the source material: `audit/write-access/REPORT.md` prose says 129 capabilities and its inline histogram sums to 129, while `matrix.json` and the script both give 130. Public copy: artifact bundle, note 3. The script is published at compute.mjs and reproduces these figures from a clean directory. ↩
  4. 4.`audit/write-access/REPORT-ADDENDUM.md`, section 4(b), the enforcement-sensitivity recompute. A second discrepancy, recorded rather than smoothed over: the addendum's baseline row reads 57.0% where `compute.mjs` yields 56.6%. The enforcement figure is quoted as published; the difference does not affect the conclusion, and neither figure is the point of the comparison. Public copy: artifact bundle, note 4. ↩
  5. 5.`audit/write-access/REPORT.md`, section 3. Three independent facts each cap the strict score at zero: the live-verification harness had never been run, every ad platform ships with the vendor call suppressed by default, and no production telemetry surfaced a single row carrying a vendor success. Public copy: artifact bundle, note 5. ↩
  6. 6.`audit/write-program/SPRINT-4.md` and `SPRINT-5.md`, with the score movement reproduced at authoring time by re-running `compute-sprint4.mjs` and `compute-sprint5.mjs`. The reclassification that lowered the figure is commit `cc0cd5b84`. Public copy: artifact bundle, note 6. Both scripts are published: compute-sprint4.mjs and compute-sprint5.mjs. ↩
  7. 7.The distinction is enforced in the type system at `lib/ads/writers/types.ts`, where the audit-facing execution mode is deliberately wider than the mode the environment gate can resolve to. Public copy: artifact bundle, note 7. ↩
  8. 8.`audit/write-program/SPRINT-5.md`, Part A.1. The two verbs enable and disable a catalogue sync flag that a scheduled job later reads. Public copy: artifact bundle, note 8. ↩
  9. 9.`lib/actions/reversibility.ts`. The explicit-confirmation requirement defaults to the value of the reversibility class, so a verb graded none inherits the gate rather than having to opt into it. Public copy: artifact bundle, note 9. ↩
  10. 10.`audit/write-program/SPRINT-6.md`, Part A. Ten duplicate-suppression implementations across six platforms guarded 66 verbs, all filtering on completed attempts only. The residual window between marking an attempt in flight and issuing the call remains open and fails in the refusing direction. Public copy: artifact bundle, note 10. ↩
  11. 11.`audit/write-program/SPRINT-2E.md`, Task 3, for the grading rule and the three rows it leaves null. The five-row table size is recorded in `docs/audits/live-verification/UNVERIFIED_INVENTORY.md` as of the 2026-07-08 migration verification. The pre-state value that reads as real is recorded in the same sprint document, Task 2. Public copy: artifact bundle, note 11. ↩
  12. 12.Commits `28608fa16` and `327a77a6d`; `audit/write-program/SPRINT-2.md`. Corrected paths: send now creates a top-level campaign send job; scheduling first patches a send strategy onto the campaign and then creates the job; cancellation patches the job. Vendor reference: developers.klaviyo.com, operations create_campaign_send_job and cancel_campaign_send. Public copy: artifact bundle, note 12. ↩
  13. 13.`audit/write-program/SPRINT-2C.md`, Task 4, which classified each writer test file as contract-asserting or implementation-mirroring and recorded which ones were structurally incapable of catching an invented endpoint. Public copy: artifact bundle, note 13. ↩
  14. 14.Commits `b12a5e197` and `bbefcb90e`; `audit/write-program/SPRINT-3.md` Task 1 and `SPRINT-4.md` section D2. Each deliberate change was reverted and the empty diff confirmed before the result was recorded. Public copy: artifact bundle, note 14. ↩
  15. 15.`audit/write-program/SPRINT-5.md`, Part B. The renamed fields are on the campaign resource, replacing field 63 with field 104 and field 64 with field 105. The re-verification totals are drawn from the client library's own descriptor rather than from the vendor's prose reference. Public copy: artifact bundle, note 15. ↩
  16. 16.Commits `d5bfda266`, `45017a82e` and `acaaab825`; `audit/write-program/SPRINT-1.md`. The shared helper is `lib/guardrails/assert-write-scope.ts`, wired into four write-preparation chokepoints and failing closed on a read error. Public copy: artifact bundle, note 16. ↩
  17. 17.Sentinel and readback defects: `audit/write-program/SPRINT-1.md`, Task 2. Documented numeric scope identifiers: `audit/write-program/SPRINT-5.md`, restated in `SPRINT-6.md` as evidence-gathering rather than a code change, because the mapping cannot be written without observing a real response. Attestation storage: `SPRINT-6.md`, Part D. Public copy: artifact bundle, note 17. ↩
  18. 18.`audit/write-program/SPRINT-2E.md`, Task 1. The resolution chain reads the top-level column first, then the metadata field where the callback had actually been writing. Public copy: artifact bundle, note 18. ↩
  19. 19.`audit/write-program/SPRINT-1.md`, premise correction. The evaluator is `lib/executor/guardrailEvaluator.ts`, called on every path. Public copy: artifact bundle, note 19. ↩
  20. 20.`audit/write-access/REPORT-ADDENDUM.md`, section 1, which states the error and its class explicitly: asserting from what was in front of the auditor rather than checking the authoritative source. Public copy: artifact bundle, note 20. ↩
  21. 21.`audit/write-program/SPRINT-6.md`, final arm-readiness table. Every row is a human action; none requires a code change. Public copy: artifact bundle, note 21. ↩
  22. 22.Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. The abstract states that the best GPT-4-based agent reaches an end-to-end task success rate of 14.41%, against 78.24% for humans. Retrieved 2026-08-05. ↩
  23. 23.Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. The benchmark draws 2,294 problems from real issues and pull requests across 12 Python repositories, and the abstract reports that the best-performing model resolves 1.96% of them. Retrieved 2026-08-05. ↩
  24. 24.Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. (2023). GAIA: a benchmark for General AI Assistants. arXiv:2311.12983. The abstract reports 92% for human respondents against 15% for GPT-4 with plugins. Retrieved 2026-08-05. ↩
  25. 25.Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. The abstract reports that state-of-the-art function-calling agents succeed on under half of the tasks, and that the pass-to-the-eighth consistency measure falls below a quarter in the retail domain. Retrieved 2026-08-05. ↩
  26. 26.Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., and Jadliwala, M. (2024). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. arXiv:2406.10279. Retrieved 2026-08-05. The figures quoted here are the abstract's own: 576,000 code samples in two languages across 16 models, 205,474 unique hallucinated package names, and an average hallucination rate of at least 5.2% for commercial models and 21.7% for open-source models. We cite the preprint because that is the version we retrieved and verified; we make no claim here about its publication venue. ↩
  27. 27.The bundle is at cresva.ai/research/artifacts/crv-2026-001, with the policy in README.md and an archive at crv-2026-001-artifacts.tar.gz. Before publication the three scripts were extracted to a directory outside the repository and run with no other inputs; their output matched the figures printed here, which is what establishes that redaction changed no result. ↩

References

  1. [1]Meta, Marketing API reference: ads pixel. Retrieved 2026-07-30 during the contract-verification pass. https://developers.facebook.com/docs/marketing-api/reference/ads-pixel/
  2. [2]Meta, Conversions API: using the API. Retrieved 2026-07-30. https://developers.facebook.com/docs/marketing-api/conversions-api/using-the-api/
  3. [3]Klaviyo, API reference: create campaign send job. The operation that replaced the invented send sub-resource. https://developers.klaviyo.com/en/reference/create_campaign_send_job
  4. [4]Klaviyo, API reference: cancel campaign send. https://developers.klaviyo.com/en/reference/cancel_campaign_send
  5. [5]Google, Google Ads API reference: CampaignService, version 24. The message whose date fields were renamed. https://developers.google.com/google-ads/api/reference/rpc/v24/CampaignService
  6. [6]Google, Google Ads API guide: create campaigns. The source for the date-time format used at the writer boundary. https://developers.google.com/google-ads/api/docs/campaigns/create-campaigns
  7. [7]Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. Retrieved 2026-08-05. https://arxiv.org/abs/2307.13854
  8. [8]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. Retrieved 2026-08-05. https://arxiv.org/abs/2310.06770
  9. [9]Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. (2023). GAIA: a benchmark for General AI Assistants. arXiv:2311.12983. Retrieved 2026-08-05. https://arxiv.org/abs/2311.12983
  10. [10]Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. Retrieved 2026-08-05. https://arxiv.org/abs/2406.12045
  11. [11]Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., and Jadliwala, M. (2024). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. arXiv:2406.10279. Retrieved 2026-08-05. https://arxiv.org/abs/2406.10279
  12. [12]Cresva. CRV-2026-001 artifact bundle: the capability matrix, the three deterministic score scripts, the redacted excerpts behind every footnote citing an internal document, and a README stating the redaction policy. Published alongside this paper; the scripts run offline with no inputs beyond the bundle. https://cresva.ai/research/artifacts/crv-2026-001/
  13. [13]Cresva, write-access truth audit. Internal artifact, repository path audit/write-access/, including REPORT.md, REPORT-ADDENDUM.md, matrix.json and compute.mjs. The published subset is in the artifact bundle above.
  14. [14]Cresva, write program sprint records. Internal artifacts, repository path audit/write-program/, SPRINT-1 through SPRINT-6 with the two deterministic rescore scripts. The published subset is in the artifact bundle above.

How to cite

Shubham Raghav (2026). Proof of Write: An Evidentiary Standard for Agent Action Capability. Cresva, CRV-2026-001, version 1.0. https://cresva.ai/research/proof-of-write

BibTeX
@techreport{raghav2026proofofwrite,
  title       = {{Proof of Write: An Evidentiary Standard for Agent Action Capability}},
  author      = {Shubham Raghav},
  institution = {Cresva},
  year        = {2026},
  number      = {CRV-2026-001},
  version     = {1.0},
  url         = {https://cresva.ai/research/proof-of-write}
}