Checkout Agents: Stop One Step Before the Buy Button

Getting a checkout agent from a demo to something you would trust near a credit card is mostly not a prompting problem. The hard parts are what the model sees, what it is allowed to do, and how you decide that a run succeeded.

TL;DR

• Run in two phases: every purchase starts with a pre-flight run that stops one step before the final order button and pulls out the final price for the user to confirm. Only after the user agrees does the last step run.
• Design the action space before the prompt: a screenshot plus a numbered element list goes in, a small set of actions comes out, and none of them places an order.
• Define success strictly in evals: a run succeeds only if it stops in the right place with the right fields extracted. Clicking the order button is the worst failure.
• Classify each failure before fixing it: tooling failures and model failures need completely different fixes, and early on tooling usually dominates.
• Keep card numbers out of the model’s context, and give every purchase an idempotency key.

Two kinds of hard

Checking out on real websites on a user’s behalf is hard in two separate ways:

1. Finishing checkout reliably across merchants: every merchant has different steps, forms and pop-ups. The agent has to get through flows it has never seen without getting stuck or going in circles.
2. Handling the user’s money: no double charges, no leaked card numbers, and the user’s privacy protected.

The first decides whether you can ship, and it is where most of the engineering time goes. The second decides whether you dare to ship, and it has to be designed in from the start, not added at the end.

Pre-flight first, then the last step

When the purchase happens days or weeks after the user set it up (say, “buy it when the price drops”), running straight through to the end runs into two problems:

• the original payment authorization may have expired;
• once tax and shipping are added, the total may fall outside the price range the user set.

So split execution into two phases:

1. Pre-flight: the agent walks the whole flow, stops one step before the final order button, and extracts the final total, including tax and shipping.
2. Last step: compare that total with the user’s limit and ask the user to confirm. Only after they agree does the agent place the order.

That way “the last step needs the user’s approval” is not a reminder buried in a prompt. It is the shape of the flow itself.

Design the action space before the prompt

What the model sees at each step (the observation):

• a screenshot of the current page;
• a numbered list of interactive elements, with the same numbers drawn on the screenshot (set-of-marks style);
• the goal, and the steps taken so far.

What the model can call (the actions):

click(id)          # click the element numbered id
type(id, text)     # type text into the input numbered id
scroll(direction)  # scroll the page
navigate(url)      # open a URL
extract(fields)    # read fields from the page, such as total, shipping, tax
stop(reason)       # finish, and say why

There is no “place order” action here. If the final button is just another clickable element on the page, add a hard rule in the harness that blocks clicks on it. The answer to “how do you guarantee the agent never buys by mistake” should be a mechanism, not a line in the prompt that says “please do not click Buy.”

There is one more requirement on observations and actions that is easy to miss: they must describe the same moment. If the numbered list and the screenshot were not captured together, the element 17 the model sees in the screenshot may not be the element 17 it actually clicks. So capture again before every action, and before clicking, check that the target element’s features (its text and type) have not changed.

Evals: define success strictly

Evals run on real merchant flows, but never actually place the order. Each run is judged like this:

Outcome	Verdict
Stopped one step before the final order button, with the total and other fields extracted correctly	Success
Stopped in the right place, but extracted the wrong fields	Failure
Got stuck, or looped between pages	Failure
Clicked the final order button	Failure, and the worst kind

Besides the success rate itself, record with every eval run: the model and its version, how many merchants and flows the eval set covers, and how long one run takes. Each time you ship a category of fix, write down how many points it added, and build up a breakdown table. Then “which fixes helped most” has an answer in data, not in memory.

Classify failures first: tooling or model

When reading through failed runs, first decide which kind each one is:

• Tooling failure: the information the model received was itself wrong or incomplete. For example:
  • content inside an iframe (such as the payment form) never made it into the DOM snapshot, so the model could not see it;
  • the screenshot and the DOM did not match;
  • a promo pop-up appeared after the screenshot, every element number shifted, and the model clicked something else using the old number.
• Model failure: the information was right, but the model made the wrong call or took the wrong step.

The fixes are completely different. Tooling failures are fixed in the capture layer: include iframe content, capture again before each action, check the element before clicking. Only model failures are a job for the prompt, for example splitting the flow into stages with a more focused prompt for each, or writing a few site-specific notes for merchants with unusual flows.

Early on, tooling failures usually dominate. Make the capture layer solid before tuning prompts: if the observation itself is wrong, no amount of prompt tuning will make the agent stable.

Keeping the money safe

• Card numbers never enter the model’s context: payment goes through a token or a hosted payment step, and the card number appears nowhere the model can see.
• One idempotency key per purchase: network retries and agent reruns cannot cause a double charge.
• The last step requires the user’s approval: that is the two-phase execution above.
• The tools have no order action: that is the action space design above.

Each of these four covers a different layer, so when one of them fails, the others still hold.