QA automation — the ladder

Status: Active · Drafted: 2026-09-04 (owner ask: automate as much QA as possible; device remains the last rung) · Last reviewed: 2026-11-13 (reset to the readiness build; the owner's hand-test script is List 53)

The principle is the efficiency ladder from CLAUDE.md: every QA question is answered on the cheapest rung that can answer it, and every device finding is turned into a lower-rung check so it never needs the device again.

Rungs (what exists as of 2026-11-13)

Rung	Answers	Command	State
0 · static	types, architecture, lint, docs, terminology, tests that read the real clock	yarn check (typecheck, jest, ArchUnit, docs:check, cycles, check:test-clock), yarn lint:strict, Stop hook	green gate; the clock gate is new (see Rung 0c)
0b · screen tests	the five query states and the defining action of each page	each feature's jest project	76 pages, mocked kit (below)
0c · real-UI tests	does the REAL kit draw the page's buttons, headings, names and rows	packages/*/src/__tests__/real-ui/** (ten files, 68 tests)	new 2026-11-13 (below)
1 · data truth	RLS, derived states, money rules, venue invariants	yarn qa:data (pgTAP plus fact checks)	unchanged
1b · server health	the swallowed-error sink and each cron job's last verdict	yarn qa:health (local), --hosted prints the call, --hosted --run reads	new 2026-11-13; developer-gated; needs allow_dev_rpcs on hosted
2 · scenario	every screen state the seeds can produce	yarn qa:seed --local, node scripts/run-scenario.mjs {chain} --local	chains below; the readiness chains are new
2b · web-render	does a page render through apps/web	yarn screenshot-web	unchanged (below)
2c · review tools	measurements from every rendered page; whole journeys clicked through as the seeded people	node scripts/screenshot-web.mjs --manifest=<file> --audit --audit-out=<dir>, --journeys=<defs.mjs> --out=<dir>	built in batch 63; run by the lead
2d · compiled web	the same pages and journeys with the React Compiler on, as the phone build compiles them	the 2b and 2c commands on a web build made with TWOMORE_WEB_REACT_COMPILER=1 (off by default)	batch 67; run by the lead; a browser, so it cannot show native-only faults
3 · flows (emulator)	navigation, RSVP, scoring, share, club funnels, venue finder, fee attest, membership offer, create session, message	yarn qa:emulator	14 flows rewritten 2026-11-13; none has run (below)
4 · visual judge	spacing, density, truncation, wrong copy	scripts/qa/visual-judge.md	needs the rendered check's screenshots
5 · telemetry	crashes and perf regressions after a run	yarn qa:telemetry, yarn debug:logs	unchanged; developer-gated
6 · device (owner)	gestures, keyboard, haptics, real network, native share, KakaoTalk	List 53	the first device run since the update pause

Every device finding becomes a lower-rung check in the same commit (the 2026-09-02 sweep converted 105 items; that is the template).

Rung 0b — screen characterization tests (2026-09-12)

Every one of the 76 screens has a characterization test (<screen dir>/__tests__/<screen>.test.tsx, 70 files, ~440 tests) that locks its query states — loading skeleton, QueryErrorState with retry, offline cold start (OfflineEmptyState, never the empty copy), empty, populated — plus its defining action. They run in yarn check through each feature's jest project. All of them stand on @twomore/test-kit (packages/test-kit): one @twomore/ui mock (any primitive renders its text props; the lists, shells, tabs and ModalPanel are modelled faithfully), one @twomore/app mock (the REAL t() and routes, a QueryBoundary that runs the five-phase precedence, an unmocked data hook throws by name), and queryState/composerState fixtures in TanStack's exact flag shape. Every feature package's jest.config.js is one line over @twomore/test-kit/jest-feature-config — the eight near-identical configs (and three packages with none) are gone. Recipe for a new screen: copy packages/features/clubs/src/__tests__/club-album-screen.test.tsx. Copy is never hand-typed in a test. A test that builds its own @twomore/app mock (a helper module, a hook) still takes t from the kit — t: require('@twomore/test-kit').t inside the factory, import { t } from '@twomore/test-kit' for assertions, setLocale('en') at module scope when it asserts English — and asserts t().ns.key / t().common.x(n), never a literal (the 2026-09-13 consolidation moved 1,300 keys and every hand-typed t: () => ({ … }) skeleton went stale at once; t is typed off the real table, so a moved key now fails typecheck). The kit also passes the entity bounds through (DISPLAY_NAME_MAX, CLUB_NAME_MAX, DUES_DAY_MAX, the club-group and venue court-count limits — a screen compares length >= MIN against them, and the generic ALL_CAPS fallback would hand it {}), and the jest.mock factory must stay an expression (() => require('@twomore/test-kit').createUiMock(...)) — a block body is what check:test-kit-mocks (AGENTS.md CODE-3) flags as a hand-rolled mock; hoist any require-heavy setup into a mock-prefixed module-scope helper. The kit also hands the real pure vocabulary through the mock — matchHistoryScope*, formatSessionFormat, deriveMatchResult, TIER_ORDER/tierLabel — because the shared list/filter modules read those at import time. The node component harnesses under scripts/__tests__/helpers take the same real table through loadLocaleSync() from scripts/lib/i18n-bundle.mjs.

What this rung cannot see: pixels, gestures, keyboard, and the four apps/mobile/app/{lb,m,p,s} share receivers (no jest project for the app shell yet).

Rung 0c — real-UI tests (2026-11-13)

The mocked screen tests (Rung 0b) replace @twomore/ui with a stand-in, so a page can pass while a button is not a button or a heading is missing. The real-UI tests render the REAL @twomore/ui under a Tamagui provider with the app's real tokens and themes; only @twomore/app is mocked, through createAppMock. The harness is packages/test-kit/src/real-ui-harness.tsx (imported by path, never from the test-kit index). Ten files cover login, 홈, 알림, the club list and club page, session detail, the scorecard, the create-session wizard, 기록 and 프로필. They prove what the mocked kit cannot: the control is a real pressable with a role and a name, a press sends exactly one write, a long name is not clipped to one line. They cannot prove layout, swipe-to-delete, scrolling, the keyboard or insets: those are the device's (List 53).

The clock gate. yarn check:test-clock (part of check:static) fails a test file under packages/**, apps/** or scripts/__tests__/**, and a test helper that is not named .test. (packages/test-kit, __test-utils__, scripts/__tests__/helpers), that reads the real clock (Date.now(), Date['now'](), a bare Date() call, a new Date() or new globalThis.Date() with no argument, performance.now(), setTimeout() without pinning it (useFakeTimers, setSystemTime, mock.timers). Today's baseline is 74 files and 130 reads in scripts/test-clock-baseline.json; a file's count may only fall, and the gate refuses a baseline that holds more than its copy at HEAD (git show HEAD:scripts/test-clock-baseline.json; the first introduction, when HEAD has none, is the one exception). node scripts/check-test-clock.mjs --refresh-baseline lowers it after a clean-up. Limit: the scanner is a text scan, so one pin word anywhere in a file exempts the whole file: a file that pins one test can still hide an unpinned read in another. An aliased read (const n = Date.now; n()) and Temporal.Now are not seen.

The phone flows (Maestro, apps/mobile/.maestro)

Fourteen flows, all rewritten on 2026-11-13 against the rebuilt pages, written and linted, not run: the new build does not exist yet, and the old build cannot run them. node scripts/check-maestro-flows.mjs checks every id exists in source, every text is a Korean string in the i18n source and every command is known. The maestro program itself is not installed on the build machine, so its own syntax check has not run either.

Flow	Proves	Seed chain
01 login	typing the e-mail and password lands on 홈	none
02 main-tabs-smoke	five tabs open to their roots	none
03 rsvp-toggle	참석하기 then 참석 취소 on 홈's next game	home_upcoming
04 score-submit	홈 task, scorecard, score sheet, 6-4 saved	home_score_verification
05 fee-attest	seat holder: 입금 대기 중, 입금 안내, 확인 대기 중이에요	readiness_qa
06 share-session	공유 opens the share sheet	home_upcoming
07 membership-offer-answer	the offered person sees 가입 제안이 와 있어요 and accepts	readiness_qa
08 club-growth-media	클럽 찾기 rows, a row opens the club page	club_growth
09 club-join-funnels	the FAB menu, 초대 코드로 가입	club_growth
10 club-create-first-session-cta	클럽 만들기, name step to region step, leave	r1_create_club_zero
11 venue-finder	경기 tab 테니스장 rows, a venue page	none
12 venue-detail-edit	page menu to the 장소 정보 제안 form	none
13 create-session	club FAB, 일정 만들기, 정기 일정, review, made	club_showcase
14 message	the 메시지 inbox, send a typed message	dense_social

Retired: 05-disputed-alert (the alert is now a 홈 task row) and 07-onboarding-under-14-gate (its texts are gone). Run: QA_EMAIL=... QA_PASSWORD=... yarn qa:emulator [--flows 01,02]; flows 05 and 07 sign in as other people and read QA_EMAIL_05/QA_PASSWORD_05 and QA_EMAIL_07/QA_PASSWORD_07. The runner refuses to start without them and names only the variable names; it masks the values in the command it prints. No flow file holds a credential (the flow lint refuses one). Whether --debug-output under the git-ignored .qa/ records typed text is unverified: delete .qa/ after a run. Known gaps, from the E2E lane: whether 홈's next-game block is the seeded session (03, 05, 06), whether the hidden score pad accepts typed input (04), the join code and club name in 09, a 정기 일정 on the first club (13), the inbox row position (14), and every step of 10 after the name step. The e2e:* package scripts name chains that exist (batch 59 integration: e2e:rsvp seeds home_upcoming; e2e:disputed-alert went with its flow). The flow lint is yarn check:maestro-flows, part of check:static; confirmed in source in the integration: the code JOINR12 (09) and the seeded clubs' regular meets (13); still unconfirmed on a device: 03, 04, 05, 06, 14.

Seed chains for the readiness build

Chain	Builds	Use
readiness_qa	club_showcase, then qa_readiness	the owner's List 53 world: the developer account owns "QA 운영 클럽"; qaready{n} accounts hold the other roles
readiness_full	full_qa, then qa_readiness	the same on the largest world; its world is left unreconciled (the reconcile takes about 28 s and is cancelled at the API's 8 s; the summary says so), so score corrections on it cannot complete: not for the hosted bed, seed readiness_qa there (the small world reconciles its own ratings, so List 53's score-correction steps work)
session_fee_manual	a fee'd pickup hosted by the caller and a cancelled one (refunds open)	fees and refunds only
club_showcase, qa_showcase, full_qa, dense, home_comprehensive	unchanged	cards, live surfaces, density

qa_readiness is a layer: it never truncates, and a second run over the same world is refused (ReadinessAlreadySeededError) because its states are single-use. Seeded accounts sign in with a function secret (SEED_ACCOUNT_PASSWORD), never a literal in source: the seed refuses to create accounts when it is missing. Which secrets a hosted seed needs is in credentials.md. The states each account holds are listed in List 53, "Accounts"; the local run on 2026-11-15 (batch 63) builds 38 accounts and 9 clubs and node scripts/qa/data-states.mjs passes all of its checks (58: the number is printed by the run; the check that the achievements were awarded is local-only, since the scenario calls the awarding function only against the local stack), twice in a row with the same counts. The seed changes only the world it made: its UPDATEs and DELETEs are scoped through seeded-scope.ts (seeded accounts and clubs, or rows written since the oldest seeded club was made; scripts/__tests__/seed-sql-scope.test.mjs fails for an unscoped one), the developer account's rate counters are cleared only for friend_request and dm_create_group, and the platform-staff account that answers the venue claims and suggestions holds the admin role only while it answers (removed in a finally; a data-state check asserts no seeded account is a platform admin afterwards). Two things the seed does not hold still: the club_showcase part is random (its matches and RSVPs differ a little run to run; every state a page needs is pinned and deterministic), and the growth events and the backdated joins are direct inserts (no app path writes a past visit or a past join), accepted for the QA world. Storage copies run one at a time (concurrent copies into one folder failed intermittently). The 참석 확인 page draws no excusal in any situation: officials read excusal requests on the member's 신뢰 기록, and 참석 확인 holds the 불참 confirmation only. The chain tail's dev_reconcile_seeded_world is cancelled by the 8 s statement limit on a large world (dense, home_comprehensive, full_qa): the log now says it timed out and that the world was NOT reconciled (it used to say "migration 00288 not applied"); that is why qa_readiness, which runs its own reconcile, must be the LAST step of its chain (the reconcile turns no_show into attended). yarn qa:health runs locally: cron jobs that cannot run there (no project URL for the HTTP jobs) read "n/a loc" and do not fail the exit status.

The unit-test runs are bounded in memory (batch 61): every jest config spreads packages/test-kit/jest-workers.js (four workers, a worker is recycled above 800 MB, JEST_MAX_WORKERS overrides), held by scripts/__tests__/jest-workers.test.mjs; an unbounded run of nine workers per workspace took the shared machine past its memory cap.

Status (2026-11-13)

• Rungs 1 and 2 run on the local stack. yarn qa:seed --local did not run in the batch-57 fix round (the local functions server was not up); its result for the readiness chains is in the seed lane's report.
• Rung 3: flows written and linted, none run (above). The first emulator run is the post-build checklist's step (post-build).
• Rung 5: yarn qa:telemetry --since is developer-gated (the account AND app_config.allow_dev_rpcs).
• The rendered check (the web app built against the local stack, every page in light, dark and 200% text, beside its board frame) is the last rung before the owner's hands; it is the screenshots rung 2b produces.

Note: there is ONE hosted Supabase project and it is the QA bed. run-scenario.mjs without --local seeds the same database the venue corpus lives in (its truncate is fail-closed since 2026-08-26), and the deployed seed-scenario function refuses with 403 whenever the project's ENV secret is production (fail-closed; it is production as of 2026-09-05). Pre-launch owner unlock (2026-09-08): because the hosted project is the only QA bed, the function also admits seeds when BOTH secrets SEED_PROD_UNLOCK=true and SEED_ALLOWED_UIDS=<dev account id> are set — only those account ids pass, anonymous and foreign JWTs still get 401. This is what makes the dev panel's 전체 QA work on the phone. Launch removes SEED_PROD_UNLOCK (pre-launch checklist #11); the fail-closed default returns the moment either secret is missing. Rung 2 therefore runs as yarn qa:seed --local against the local Supabase stack: npx supabase start, npx supabase functions serve --no-verify-jwt --env-file supabase/functions/.env, then the script resolves the local URL/keys from supabase status, creates the dev account locally if missing, and builds and uploads the seed media fixtures (node scripts/seed-media.mjs --local: the real club photos and logos every seeded club copies, see canon/data-and-hooks.md); the hosted project takes the same fixtures with --linked. Verified 2026-09-05: all 39 scenario steps across the three chains plus every individual home_*/card_showcase/club_growth/r1_create_club_zero key pass locally through the split seeder.

Rung 2b — web-render screenshots

scripts/screenshot-web.mjs (slice F0-c, flat-layout-implementation-plan.md) answers one question no lower rung can: does a screen render at all through apps/web — the Next.js target that imports the same @twomore/* feature screens as mobile — without touching a device or emulator. Substitutes for the slice-completion loop's device-check stage until F8.

• Run: yarn qa:seed --local first (rung 2, so real rows exist), then yarn screenshot-web (or node scripts/screenshot-web.mjs [routes...] [--out <dir>] [--session=<id>] [--club=<id>] [--user=<id>] [--venue=<id>]).
• What it does: reuses the BUILT web app that serves port 3100 against http://127.0.0.1:54321 (it never starts a web server: a next dev server grew past 10 GB and was killed four times; build and serve with yarn workspace @twomore/web build / start -p 3100, and a run ends with exit 3 when nothing answers /login); drives a headless Chromium (~/.cache/ms-playwright) over raw CDP — no Playwright package dependency; logs in through the real /login form (first probing where the form sends credentials with a throwaway pair and the request held back, and refusing to type the real password into an app that does not sign in at the local API; pre-filled from NEXT_PUBLIC_DEV_EMAIL/PASSWORD, mapped from .env's EXPO_PUBLIC_DEV_* via next.config.ts); resolves route <id> placeholders from the local data API (signed in as the dev account — RLS blocks anon-key reads); screenshots the default route list (/, /records, /profile, /courts, /sessions/<id>, /clubs/<id>, /p/<id>, /directory-venues/<id>) full-page at 390×844@2x.
• Capture timing (2026-09-18): after hydration the rung waits for every Skeleton to leave the DOM (each carries data-testid="skeleton" on web; 30s ceiling) before it shoots, so a cold route's loading state is never captured as the page. A route that settles on the app's query error state (t().common.genericError + retry) classifies as error, never ok — an unresolved [sessionId] placeholder produced exactly that and had been passing.
• Measuring, not guessing: --probe='<js expression>' is evaluated in the settled page after each capture and written to <route>.probe.json — the way to get real layout numbers (chip widths, computed flex, text overflow) for a web-parity fix instead of reasoning from a PNG. Routes with real ids are passed literally (/sessions/<uuid>/payments); the <id> placeholder form is what the resolver substitutes, not [sessionId].
• Manifest run (the rendered check): node scripts/screenshot-web.mjs --manifest=<file> --out=<dir> --results=<file.jsonl> reads a manifest of captures (page code, state, concrete route, the account to sign in as, steps for tap/before/viewport/offline, widths, the board frame, and an expect), signs in per account through the real form with the seeded password read by NAME (--password-env=<NAME> or the functions env file; --password is refused) and asserts the stored session belongs to that account, then writes <code>/<state>.<light|dark>.<width>.png with a .console.txt and appends one result row per capture (ok, wrong-state, error, blank, unsettled, native-only; console count, final URL, title, document height). It is resumable (a capture whose picture exists is skipped unless --force; --only=<code|family|key prefix> narrows), reuses one page (renewed every 25 captures), --sheets=<dir> writes the contact sheets, and steps.offline (after-load or cold) fails every request to the API origin and reports navigator.onLine === false.
• A browser error is an error: a capture whose final URL is not the app (Chromium's chrome-error: page) or whose main document failed is error with the reason server unreachable, and the run ends (exit 3); a GET /login runs before every capture. Each entry's expect (visible text, testIds, absent text) is checked after the page settled and any steps ran: a miss is wrong-state with the heading and error text found instead (the picture is kept), and every entry also fails on the app's load-error texts (불러오지 못했어요, 문제가 발생했어요) unless it sets allowError.
• Known web-target gap (found running this, not fixed — out of scope for the screenshot script): apps/web/lib/stubs/expo-location.ts exports getForegroundPermissionsAsync/getCurrentPositionAsync but not getLastKnownPositionAsync. The stub's getForegroundPermissionsAsync reports granted whenever navigator.geolocation exists (true in every browser, regardless of actual permission), so packages/app/src/presentation/hooks/queries/use-user-location.ts sails past its permission check and calls the missing function, throwing and tripping the route's error boundary. This hits every screen that calls useUserLocation (directly or via venue-row.tsx / use-venue-finder-data.ts / use-directory-venue-detail.ts) — /courts, /sessions/[id], /directory-venues/[id] in the default route list. Not a bug in this script; recorded here as a rung-2b finding for whoever picks it up.

Rung 2c — the review tools (batch 63)

Two modes of scripts/screenshot-web.mjs answer what a picture cannot; both drive the BUILT app on port 3100 against the LOCAL stack, never start the app, and need yarn qa:seed --local (the readiness_qa chain) first.

• Measuring mode: node scripts/screenshot-web.mjs --manifest=<manifest.json> --audit --audit-out=<dir> [--out=<dir>] visits each page of a manifest (route, account, state) and records measurements instead of judging pictures: overflow and clipped text, content off the rail, tap targets under 44, contrast, broken images, stray Latin text outside the allow-list read from the Korean word files, missing accessible names, and a dump of every visible sentence (<audit-out>/…, one row per page in audit.jsonl; a page already recorded is skipped on a re-run).
• Journey runner: node scripts/screenshot-web.mjs --journeys=<defs.mjs> --out=<dir> [--only=J01,J02] [--group=g1] [--force] clicks through the journeys a definitions file lists as the seeded people (real input, through the form's own controls), each followed by a read-only SQL assertion on the local database; one row per journey in <dir>/journeys.jsonl (ok, failed-at-step, db-mismatch or native-only), pictures under <dir>/journeys/<id>/. A journey that is ok or native-only is skipped on a re-run, a failed one (failed-at-step, db-mismatch) is tried again, --force redoes all, --group runs one group (a group starts from a fresh seed). A step's open takes only a path that starts with a single / (the app's own origin; @host/x, //host/x and a full URL are refused when the definitions load). A SQL error in a db or save step fails THAT journey (failed-at-step, the database's own words in the row) and the run goes on. The SQL guard is the database session: psql runs with default_transaction_read_only=on, so a function that writes (SELECT some_writer()) is refused by the session, not by the check in front of it (one SELECT, no writing word), which only catches a typo early.

The lead runs both through wrappers kept outside the repository (run-audit.sh, run-journeys.sh, which add the memory guard and the fresh seed); the tool's own flags above are what the repository holds. A heavy run (a browser, a seed, the full gate) starts only when ~/twomore-tools/quiet.sh reports no busy reason, and only one at a time.

Order of work

1. yarn qa:data and yarn qa:seed (rungs 1–2): pure scripts, no device — ship first, run in CI nightly.
2. Local emulator runner + the two venue-finder flows (rung 3): needs the WSL↔Windows emulator path from the guide; smoke it once on the owner's machine, then it is the default pre-OTA check.
3. Visual judge (rung 4): once screenshots flow, the agent rubric is the same one the wireframes are graded by (canon/components.md, screen-blueprint.md).
4. Enable 03–06; telemetry gate last.

Script-test timing rule

yarn test:scripts (Node test runner) never spells a short wall-clock wait itself: no setTimeout(..., <50), timeoutMs: <50, AbortSignal.timeout(<50) or fixed keep-alive timer in scripts/__tests__/**. Use tinyDeadline() / settleAfter() / whileAlive() from scripts/__tests__/helpers/deadline-fixture.mjs (scaled by TWOMORE_TEST_TIME_SCALE, default 1; set 4 on loaded machines). scripts/__tests__/check-test-timing.test.mjs fails with file:line on any raw literal; the helper is the only exemption. Jest suites use fake timers for the same reason.

What stays manual

Keyboard-controller behaviour, draft-resume across process death, gesture conflicts, haptics, and anything Naver/Kakao/PortOne WebView-bound (external credentials). These are the owner's rung-6 checklist; every other item is on a lower rung or becomes one.