QA automation — the ladder

Status: Active · Drafted: 2026-09-04 (owner ask: automate as much QA as possible; device remains the last rung)

The principle is the efficiency ladder from CLAUDE.md: every QA question is answered on the cheapest rung that can answer it, and every device finding is turned into a lower-rung check so it never needs the device again.

Rungs (what exists, what is added)

Rung	Answers	Gate today	Added by this plan
0 · static	types, architecture, lint, docs, terminology	yarn check (typecheck · jest 3,744 · ArchUnit · docs:check · cycles), lint:strict, Stop hook	—
1 · data truth	RLS, derived states, money rules, venue invariants, classifier	pgTAP 887 (CI-blocking), scripts/audit/*	yarn qa:data = supabase test db + check-facts on a fresh corpus snapshot (fails on any new C1–C9 flag class)
2 · scenario	every screen state the seeds can produce	scripts/run-scenario.mjs <chain> (9 chains, ~50 scenario keys)	yarn qa:seed --local = local stack (supabase start + functions serve) → qa_showcase + home_comprehensive + club_showcase; asserts every step passes and its headline counts are positive
2b · web-render	does a screen render at all through the web target, without a device or emulator	manual node scripts/screenshot-web.mjs	yarn screenshot-web: boots apps/web (Next dev, port 3100) against the LOCAL Supabase stack, logs in as the dev account through the real /login form, and screenshots a route list at 390×844@2x into the scratchpad — see below
3 · flows (emulator)	navigation, RSVP, scoring, share, club funnels, venue finder	Maestro 01/02/08/09/10 + generated share flow, hosted EAS workflows (build_id reuse)	local runner scripts/qa-emulator.mjs: boots the AVD (docs/guides/android-emulator.md), installs the last preview APK, seeds, runs the CI set + new 11-venue-finder.yaml (chips with counts, 테니스장/연습장/샵 views, region filter, detail sections per kind, correction modal per kind) and 12-venue-detail-edit.yaml; takeScreenshot per screen into .qa/screens/<flow>/
4 · visual judge	"would Toss ship this?" — spacing, density, truncation, wrong copy, empty states	owner on device	scripts/qa-visual-judge.md: a Sonnet agent reads .qa/screens/* with the Read tool against the screen-blueprint + wireframe of that screen and files defects (.qa/visual-findings.json); the owner reviews findings, not screens
5 · telemetry	crashes/perf regressions after a run	yarn debug:logs	yarn qa:telemetry --since <run start> fails on any client_errors row from the QA account
6 · device (owner)	gestures, keyboard, haptics, draft-resume, real network	manual	unchanged — but each finding must land as a rung-0/1/3 check in the same commit (the 2026-09-02 sweep converted 105 items; that is the template)

Rung 0b — screen characterization tests (2026-09-12)

Every one of the 76 screens has a characterization test (<screen dir>/__tests__/<screen>.test.tsx, 70 files, ~440 tests) that locks its query states — loading skeleton, QueryErrorState with retry, offline cold start (OfflineEmptyState, never the empty copy), empty, populated — plus its defining action. They run in yarn check through each feature's jest project. All of them stand on @twomore/test-kit (packages/test-kit): one @twomore/ui mock (any primitive renders its text props; the lists, shells, tabs and ModalPanel are modelled faithfully), one @twomore/app mock (the REAL t() and routes, a QueryBoundary that runs the five-phase precedence, an unmocked data hook throws by name), and queryState/composerState fixtures in TanStack's exact flag shape. Every feature package's jest.config.js is one line over @twomore/test-kit/jest-feature-config — the eight near-identical configs (and three packages with none) are gone. Recipe for a new screen: copy packages/features/clubs/src/__tests__/club-album-screen.test.tsx. Copy is never hand-typed in a test. A test that builds its own @twomore/app mock (a helper module, a hook) still takes t from the kit — t: require('@twomore/test-kit').t inside the factory, import { t } from '@twomore/test-kit' for assertions, setLocale('en') at module scope when it asserts English — and asserts t().ns.key / t().common.x(n), never a literal (the 2026-09-13 consolidation moved 1,300 keys and every hand-typed t: () => ({ … }) skeleton went stale at once; t is typed off the real table, so a moved key now fails typecheck). The kit also passes the entity bounds through (DISPLAY_NAME_MAX, CLUB_NAME_MAX, DUES_DAY_MAX, the club-group and venue court-count limits — a screen compares length >= MIN against them, and the generic ALL_CAPS fallback would hand it {}), and the jest.mock factory must stay an expression (() => require('@twomore/test-kit').createUiMock(...)) — a block body is what check:test-kit-mocks (AGENTS.md CODE-3) flags as a hand-rolled mock; hoist any require-heavy setup into a mock-prefixed module-scope helper. The kit also hands the real pure vocabulary through the mock — matchHistoryScope*, formatSessionFormat, deriveMatchResult, TIER_ORDER/tierLabel — because the shared list/filter modules read those at import time. The node component harnesses under scripts/__tests__/helpers take the same real table through loadLocaleSync() from scripts/lib/i18n-bundle.mjs.

What this rung cannot see: pixels, gestures, keyboard, and the four apps/mobile/app/{lb,m,p,s} share receivers (no jest project for the app shell yet).

Deferred flows get their seeds

Maestro 03 (RSVP), 04 (score submit), 05 (disputed alert), 06 (share) are skipped in CI "until a deterministic seed exists". The seeds exist now (home_rsvp_pending, home_score_verification, live_showcase, home_upcoming); wire each flow to its seed in the e2e:* scripts and enable them in config.yaml.

Status (2026-09-04)

• Rungs 1–2 shipped: yarn qa:data (pgTAP + fact checks vs scripts/qa/facts-baseline.json), yarn qa:seed (run-scenario.mjs prints a QA_JSON summary line).
• Rung 3: yarn qa:emulator runner + flows 11/12 + seeds wired for 03–06 (built this session; first device smoke is the owner's, the WSL host has no emulator attached).
• Rung 4: scripts/qa/visual-judge.md prompt ready; runs on the screenshots rung 3 produces.
• Rung 5: yarn qa:telemetry --since (folded into the emulator runner's exit code).

Note: there is ONE Supabase project. run-scenario.mjs without --local seeds the same database the venue corpus lives in (its truncate is fail-closed since 2026-08-26), and the deployed seed-scenario function refuses with 403 whenever the project's ENV secret is production (fail-closed; it is production as of 2026-09-05). Pre-launch owner unlock (2026-09-08): because the hosted project is the only QA bed, the function also admits seeds when BOTH secrets SEED_PROD_UNLOCK=true and SEED_ALLOWED_UIDS=<dev account id> are set — only those account ids pass, anonymous and foreign JWTs still get 401. This is what makes the dev panel's 전체 QA work on the phone. Launch removes SEED_PROD_UNLOCK (pre-launch checklist #11); the fail-closed default returns the moment either secret is missing. Rung 2 therefore runs as yarn qa:seed --local against the local Supabase stack: npx supabase start, npx supabase functions serve --no-verify-jwt --env-file supabase/functions/.env, then the script resolves the local URL/keys from supabase status and creates the dev account locally if missing. Verified 2026-09-05: all 39 scenario steps across the three chains plus every individual home_*/card_showcase/club_growth/r1_create_club_zero key pass locally through the split seeder.

Rung 2b — web-render screenshots

scripts/screenshot-web.mjs (slice F0-c, flat-layout-implementation-plan.md) answers one question no lower rung can: does a screen render at all through apps/web — the Next.js target that imports the same @twomore/* feature screens as mobile — without touching a device or emulator. Substitutes for the slice-completion loop's device-check stage until F8.

• Run: yarn qa:seed --local first (rung 2, so real rows exist), then yarn screenshot-web (or node scripts/screenshot-web.mjs [routes...] [--out <dir>] [--session=<id>] [--club=<id>] [--user=<id>] [--venue=<id>]).
• What it does: starts yarn web (Next dev, port 3100 — apps/web/package.json's dev script pins this, not the Next.js default 3000) against http://127.0.0.1:54321 if nothing is already serving it; drives a headless Chromium (~/.cache/ms-playwright) over raw CDP — no Playwright package dependency; logs in through the real /login form (pre-filled from NEXT_PUBLIC_DEV_EMAIL/PASSWORD, mapped from .env's EXPO_PUBLIC_DEV_* via next.config.ts); resolves route <id> placeholders from the local data API (signed in as the dev account — RLS blocks anon-key reads); screenshots the default route list (/, /records, /profile, /courts, /sessions/<id>, /clubs/<id>, /p/<id>, /directory-venues/<id>) full-page at 390×844@2x.
• Cold-compile reality: Next dev compiles each route on first request — 60–85s observed for /profile and /courts on this workspace's package count, ~1–3s once warm. The script's nav timeout (120s) accounts for this; a route that was already visited in the same dev-server lifetime is fast.
• Known web-target gap (found running this, not fixed — out of scope for the screenshot script): apps/web/lib/stubs/expo-location.ts exports getForegroundPermissionsAsync/getCurrentPositionAsync but not getLastKnownPositionAsync. The stub's getForegroundPermissionsAsync reports granted whenever navigator.geolocation exists (true in every browser, regardless of actual permission), so packages/app/src/presentation/hooks/queries/use-user-location.ts sails past its permission check and calls the missing function, throwing and tripping the route's error boundary. This hits every screen that calls useUserLocation (directly or via venue-row.tsx / use-venue-finder-data.ts / use-directory-venue-detail.ts) — /courts, /sessions/[id], /directory-venues/[id] in the default route list. Not a bug in this script; recorded here as a rung-2b finding for whoever picks it up.

Order of work

1. yarn qa:data and yarn qa:seed (rungs 1–2): pure scripts, no device — ship first, run in CI nightly.
2. Local emulator runner + the two venue-finder flows (rung 3): needs the WSL↔Windows emulator path from the guide; smoke it once on the owner's machine, then it is the default pre-OTA check.
3. Visual judge (rung 4): once screenshots flow, the agent rubric is the same one the wireframes are graded by (canon/components.md, screen-blueprint.md).
4. Enable 03–06; telemetry gate last.

What stays manual

Keyboard-controller behaviour, draft-resume across process death, gesture conflicts, haptics, and anything Naver/Kakao/PortOne WebView-bound (external credentials). These are the owner's rung-6 checklist; every other item is on a lower rung or becomes one.