qa-runner® ★ GitHub ↗

§ 00

Version
v0.9.0
License
MIT
Deps
0 npm
Harnesses
7

What you want tested.
Run by browser agents.
Signed off by you.

Agent skills for the coding agent you already use. Point it at one flow or the whole app, from a ticket, a spec or a sentence in chat: it drafts the test cases for your approval, runs each one in a real browser, and backs every verdict with marked screenshots and a recording.

npx skills add he1ox/AgenticTestRunner

Then ask your agent to “test the checkout flow” or “run the test cases for SHOP-42”.

§ 01 · get started

Start with one flow.
Grow into a release gate.

qa-runner turns the coding agent you already use into a QA team that works your way. It plans the cases, tests them in a real browser and brings you the proof. You make every decision that matters, and each one is reused by the next run.

  1. 01

    Install and set up you · once

    Run one command, then ask your agent to “set up qa-runner”. It checks Node, npx and ffmpeg, registers the browser and writes .qa/config.json with your app, accounts and variants.

    npx skills add he1ox/AgenticTestRunner
    Plugins for each agent ↓
  2. 02

    Say what to test you

    One flow, a feature or the whole app, in your own words or from a ticket, a spec or a pull request.

    “test the checkout flow with a coupon”
    “run the test cases for SHOP-42”
    “regression pass on the admin before the release”

  3. 03

    Approve the plan you

    The agent drafts the cases into .qa/plans/ and shows you the list. Nothing runs until you approve it, so a wrong assumption costs you a minute instead of a run.

  4. 04

    Agents test in a real browser agent

    For every expected result the Test Agent marks a screenshot, and it can record the tab. When it needs a value it doesn’t have, such as an approver account, it asks you in the viewer and waits.

    Watch an agent test ↓
  5. 05

    Review every verdict you

    Check the evidence in the viewer and pick Approve, Needs retest or Reject. Ask the agent what it saw. A retest reruns only the cases you rejected.

    See a review ↓
  6. 06

    Approve what it learns you

    Agents propose lessons about your app, and only the ones you approve are kept in .qa/memory/. Cases you approve become scripts that replay with no agent, at $0, and can gate your CI.

    See it compound ↓

Why your review matters

Agents do the clicking. You decide what counts. validate checks the evidence before you see a verdict, no verdict is final until you sign it off, and nothing is remembered until you approve it. Every decision lands in files in your repository that the agents read before the next run. The second run asks fewer questions than the first, and over time the agents test the way your team does.

§ 02 · watch it test

Every verdict
comes with proof.

For each expected result the Test Agent decides to take a screenshot and marks what it checked: boxes, arrows, numbered steps, blur for personal data. Up to three agents test different flows of the same app at once, each in its own isolated browser.

act 1 · one agent, one flow

Animation: an agent tests a cart, applies a coupon, sees the expected total, decides to take a screenshot and marks it with numbered steps, a box and an arrow. Then three agents test login, cart and checkout of the same shop in parallel; checkout fails and waits for your sign-off.

§ 03 · you sign off

Every answer you approve makes the next run need you less.

  1. 01 Answer once
  2. 02 Review every verdict
  3. 03 Approve what agents learn
01 / 03 · answer once

Animation: an agent asks which account can approve a refund; you answer and tick Remember, and the answer is saved as a memory with the password as an env:NAME reference. You approve three results and reject one, and only that case is retested. You approve one lesson an agent learned and discard another, and it is saved to memory.

§ 04 · insights and replays

Your feedback compounds.

The dashboard shows where to look. The chat explains what an agent saw. Approved cases become Test plans that replay with no agent. And every answer, verdict and lesson you approve is read by the agents before the next run.

  1. 01 Read the insights
  2. 02 Ask the agent
  3. 03 Replay a Test plan
  4. 04 Watch it compound
01 / 04 · insights example data

Animation: the dashboard surfaces insights — a case that flip-flops between runs, mobile failing four times more than desktop, the slowest flows and where agent context goes. You open the unstable case and ask the agent why it failed; it explains a toast covered the Pay button, marks it on the screenshot and proposes a lesson. You build a Test plan called Release 1.0.1 from library tests and run it: every script replays deterministically, with no agent, at $0 and 0 tokens. Over six runs, questions to you fall from five to zero and reviewer agreement rises from 72% to 97% as approved memories grow.

§ 05 · replays, CI and export

$0

per replay. Every case you approve is saved as a script that replays later in its own headless browser, with no agent and no tokens. Same steps, same marked screenshots, same recording.

# gate the merge: exits 1 when a case fails

node qa-runs.mjs suite run smoke --ci --junit qa-junit.xml

# own the tests as code

node qa-runs.mjs export --tag smoke --out e2e/qa

npx playwright test --grep @smoke

When the UI changes on purpose, Repair with agent re-records only that case.

§ 06 · spend where it counts

Tokens go only where a case needs them.

Each case gets the model it needs. On a migration, the legacy system is the spec. And when the UI changes on purpose, only the stale case goes back to an agent.

  1. 01 Route each case
  2. 02 Compare with legacy
  3. 03 Repair what changed
01 / 03 · route each case example data

Animation: the orchestrator weighs six cases by their steps, data changes, recent failures and legacy checks, groups them into three batches that each take their hardest case's tier, and launches one subagent per batch: simple, standard and complex, each in its own browser. When the simple subagent fails a case, the orchestrator retries only that case on a standard subagent, and it passes. Next, an agent runs the order history on the new app and on the legacy app: totals are the same, the date format deviates and the new status filter is an improvement. Last, a Test plan replays with no agent and one case fails because the Apply button is now called Redeem; you press Repair with agent, only that case goes back to an agent, it passes and its script becomes version 3 while the other cases keep replaying at $0.

§ 07 · how it compares

Starts earlier. Ends with your signature.

qa-runnerPlaywright Test AgentsAI testing platformsGeneral browser agents
Starts fromWhat you want tested, from a ticket, a spec or chatTest code to write and repairTests on their platformAn app to explore
You getEvidence per case, your verdict, $0 replays.spec.ts files to ownRuns under a subscriptionA report of what it found
Runs onYour machine, with the agent and model you already pay forYour machineTheir platformYour machine or theirs
TogetherPlans, evidence and memory live as files in your repoApproved cases export as .spec.ts—qa-runner adds the plan, evidence check, review and retest

Playwright Test Agents: planner, generator, healer. AI testing platforms: Momentic, QA Wolf, testRigor. General browser agents: browser-use, gstack’s /qa.

§ 08 · install

One command for any agent.

Or install the plugin for your agent, which also registers the browser. Pick one, not both, or every skill shows up twice.

npx skills add he1ox/AgenticTestRunner

Detects your agents and installs the four skills for each. Then ask your agent to “set up qa-runner” to register the browser once.

Needs an agent that runs shell commands and MCP tools, Node.js 20+, npx for the browser (Playwright MCP), and ffmpeg only for recordings.

For agents → llms.txt · run-test-plan/SKILL.md