Venue Metadata Ingest — runbook & status

Status: Reference

Status: Active (foundation shipped 2026-07-07) Related: external-apis.md · api-naver-search.md · migrations 00283 (spine) / 00332–00334

Populates the club-independent venues + venue_courts spine (migration 00283) from public + map-provider sources so the app can show real venue metadata (hours, fees, amenities, coords, surface, per-court detail) with graceful degradation.

⚠️ Seed vs enrich — the non-negotiable rule

The spine is SEEDED from regulation/government data ONLY (data.go.kr std_data, KSPO, 공유누리/eshare, Overpass named census). These carry surveyed coordinates + official identity.

Kakao / Naver / TMAP are ENRICHMENT + CROSS-CHECK, not a primary seed. They add phone, category, place id, and validate coordinates. Two disciplined uses (migrations 00341–00343):

• Coordinate triangulation — gather gov + Kakao + Naver + TMAP coordinates and adopt the centroid of the tightest-agreeing cluster (≤150m); the outlier is discarded whoever it is (map-provider consensus can beat a km-off gov coord). coord_confidence records agreement. Guarded by region-consistency (a coord must land in the venue's address 시/도) so a generic name can't teleport across cities. Scripts: triangulate-coords.mjs, repair-coords.mjs.
• Cross-verified supplementary seed — a private court gov data misses is seeded ONLY when it appears in BOTH Kakao AND Naver with agreeing coords (data_source='cross_verified'; can't produce phantoms). crawl-private-courts.mjs (nationwide) added ~1,033. facility_kind from name markers + franchise pattern; ambiguous → 미확인 (host-confirmable).
• TMAP verification — enrich-tmap.mjs sets tmap_poi_id (a 4th-source confirmation) and pulls precise coords + official region names + fee/hours from TMAP POI detail. RESUMABLE + free-tier-friendly (skips already-verified rows; --limit/--max-calls).
• Naver Place-detail scrape — scrape-naver-place.mjs is the ONLY source with court count and a definitive full-vs-practice signal: the Place-detail __APOLLO_STATE__ Menu items (볼머신/타석/스크린 ⇒ practice booth; 코트 대관/N면/1시간예약 ⇒ real court). Sets facility_kind_source='naver_scrape', court_count, has_indoor, phone. Search-HTML → Place ID → m.place detail; name-verified. FRAGILE + heavy (anti-bot, ~1MB/venue) → RESUMABLE (naver_scraped_at), low concurrency + delay, stops on repeated blocks; run in --limit N batches.

A venue is never created from a single map source; that was the legacy bug (below).

Why (learned the hard way, migration 00340): a legacy pilot used Kakao keyword search as a seed. Generic district queries ("강남구 테니스장") got stored as venue names with whatever scattered coordinates the search returned, and no later pass corrected coordinates (dedup grouped by the bad coords; enrichment only layered metadata on top). Result: hundreds of venues km-off from reality and phantom rows for venues that don't exist. 00340 purged all 993 data_source='kakao' rows; the spine is now gov-seeded (~1,258). Venues gov data misses (private/indoor courts) are added via a proper registration feed or host submission (venue_corrections), NEVER a Kakao bulk seed.

---

Architecture

• Schema — venues (venue-level) + venue_courts (per-court) from 00283, extended by:
  • 00332 — venue amenity/hours/fee columns (operating_hours, fee_schedule, has_parking/shower/shop, amenities, place_refs) + a provenance/confidence layer (data_source, verification_status, field_provenance, import_hash) on both tables, mirroring 00282's rating_source/rating_state.
  • 00333 — full unique index on venues.import_hash (so ON CONFLICT can target it).
  • 00334 — public.ingest_venues_batch(jsonb): a merge-safe upsert (COALESCE per field → additive passes fill/refresh, never clobber; merges field_provenance; never downgrades host_verified). SECDEF, service_role-only.
• Edge function — supabase/functions/ingest-venues/:
  • sources/*.ts — one extractor per source → normalized SourceRecord[].
  • match.ts — name-normalize + coord-proximity (≤150m) clustering, per-field source precedence (std_data > kspo > eshare > overpass).
  • write.ts — calls the 00334 RPC (merge-safe).
  • index.ts — region-scoped orchestrator.

Provenance / confidence → app rendering

verification_status: unverified < imported (auto) < host_verified (a host confirmed) < official. The app should render by tier: verified court detail as truth; imported as provisional ("추정", host-confirm CTA); venue-level aggregate when there are no court rows; free-text session.location when there is no venue. field_provenance records each field's source for audit.

---

How to run

Deployed (--no-verify-jwt); gated by an x-ingest-token header (secret INGEST_TOKEN, value saved in the session scratchpad ingest-token.txt). Secrets: DATA_GO_KR_KEY, INGEST_TOKEN, TWOMORE_SECRET_KEY (service role), NCP_* (already set).

GET {SUPABASE_URL}/functions/v1/ingest-venues?region=<code>[&dry_run=false][&sources=std_data,overpass]
    header  x-ingest-token: <INGEST_TOKEN>

• region: 시/도 code — 11서울 26부산 27대구 28인천 29광주 30대전 31울산 36세종 41경기 42강원 43충북 44충남 45전북 46전남 47경북 48경남 50제주.
• dry_run defaults true (safe report, no write). Pass dry_run=false to write.
• sources (optional) filters the extractors run — default: all four (std_data, overpass, kspo, eshare).
• ?action=reset (token-gated) deletes every verification_status='imported' row (host / bridge rows untouched) — for a clean re-key re-run after an import_hash scheme change.
• Run regions SEQUENTIALLY — 17 concurrent invocations hit the 150s edge idle-timeout and rate-limit data.go.kr. std_data + kspo re-pull their nationwide data.go.kr feeds per region (best-effort, 50s cap); Overpass + eshare are region-scoped (fast).

---

Current state (full 4-source import complete)

• ~1,353 distinct venues imported nationwide (all 17 regions), verification_status='imported'.
• Four sources wired + cross-merged: std_data (15013117 — hours/fees/amenities), kspo (15113986 — fresh identity/coords/phone, 폐업 filtered), eshare (공유누리 — booking URLs), overpass (named census). The coord-cell import_hash (~110m, source-order-independent) converges them onto one row per venue — e.g. 삼청테니스장 = std_data + kspo + overpass, carrying hours + fees + phone together.
• Best-effort gaps (re-runnable, merge-safe so a re-run only adds): a few regions returned std=0 or eshare=0 on their pass (data.go.kr per-region variance / transient; e.g. 광주 kspo/esh=0) — just re-run that region.

---

Directory activation + cleanup (migrations 00335–00339, 2026-07-07)

The spine is now LIVE in the app (경기 tab → 코트 찾기 + /directory-venues/[id] detail) and the imported data was cleaned/enriched to a browsable state:

• 00335 — coord-cell dedup + facility_kind classification (provenance + court_count + screen name-blocklist) + adm1_id backfill.
• 00336 + scripts/enrich-venues.mjs + enrich-venues edge fn — per-venue Kakao Local + Naver 지역 (coord-matched, local script → privileged write-back): fills phone / place-id, captures provider category (full court = 테니스 > 테니스장 leaf; academy = 테니스; Naver 공유누리 개방자원 = booking fragment; 배드민턴/스포츠시설 = non-tennis), refines facility_kind (unknown 347→18).
• 00337 — non-tennis removal + orphan-fragment deletion + name-annotation strip + dedup by kakao_place_id within 400m (4,912→2,251 venues, residual dups 544→5) + adm2_id backfill.
• 00338 — seed venue_courts from court_count (242 venues, 675 rows: surface + indoor/ outdoor); extend enrich RPC to backfill adm ids.
• 00339 + scripts/backfill-venue-adm.mjs — reverse-geocode adm-null venues via Kakao coord2regioncode (region 100% / district 94%); venue_corrections moderation-queue table behind the detail "코트 정보 확인해주기" affordance.

Follow-ups (prioritized)

Done: 4-source backfill; dedup + classification; Kakao/Naver enrichment; app read-path + directory + spine detail; venue_courts seed (count-based); adm region/district backfill; correction submission queue.

1. Richer per-court data — the seed is count-based (N면 per surface). Per-court numbering (A1–A8), per-court surface variety, and photos need the 공유누리 booking units + Naver Place scrape (m.place.naver.com __APOLLO_STATE__) — see api-naver-search.md §enrichment.
2. Apply accepted corrections — venue_corrections collects member submissions (status pending); build the admin/service apply-path (→ verification_status='host_verified', facility_kind_source='host').
3. Complete adm2 — the ~6% still null are rows without coords; needs an address source.
4. session→spine-venue links + scenario seeding at spine venues.

Known limitations

• data.go.kr rate-limits under concurrency → run sequentially; std is best-effort per region.
• Court composition is known for only the 242 venues with a real court_count; the rest render the "코트 정보 준비 중 · 호스트 확인 대기" state until seeded (follow-up 1) or host-confirmed.