Venue Metadata Ingest — runbook & status

Status: Reference

Status: Active (foundation shipped 2026-07-07) Related: external-apis.md · api-naver-search.md · migrations 00283 (spine) / 00332–00334

Current authority: This page preserves the original ingest architecture and history. For fact precedence and present-day validation, docs/canon/venue-data.md controls: Naver Place leads every fact it publishes; other sources corroborate or fill a gap.

Populates the club-independent venues + venue_courts spine (migration 00283) from public + map-provider sources so the app can show real venue metadata (hours, fees, amenities, coords, surface, per-court detail) with graceful degradation.

⚠️ Record creation vs fact precedence

The original bulk spine was seeded from regulation/government data (data.go.kr std_data, KSPO, 공유누리/eshare, Overpass named census). Cross-verified Kakao + Naver results can create a record when government coverage has a gap. The later private-court discovery pass also allows a Naver-only tennis result to enter as a provisional data_source='naver', medium- confidence row because some real private operators have no Kakao or government listing; it remains visibly uncorroborated until a second source or host confirms it. Kakao-only bulk results are still forbidden.

Record creation is distinct from fact ownership. Once identity is attached, Naver Place is the leading authority for every fact it publishes. Kakao, TMAP, government, and operator sources corroborate it or fill facts that Naver does not publish. Uses include:

• Legacy coordinate triangulation (diagnostic only; do not run in write mode) — triangulate-coords.mjs and repair-coords.mjs compute provider centroids but do not enforce the current Naver-first pin hierarchy. A person-supplied pin leads, then attached Naver Place, then guarded Naver Local; other providers may fill a gap or raise confidence but must not move a Naver-led pin. Guard or replace the utilities before re-enabling their write paths. Region consistency remains a validation constraint.
• Supplementary private-court seed — crawl-private-courts.mjs uses Naver as the discovery source. Agreement with Kakao produces data_source='cross_verified'; a Naver-only result is retained provisionally as data_source='naver' with medium coordinate confidence so operators missing from Kakao/government coverage are not erased. A Kakao-only result is skipped. Initial facility_kind comes from bounded name/category evidence; ambiguous rows remain 미확인 until Place-detail, another provider, or a host resolves them. When both map sources agree, the seed keeps their raw names, pins, phones, addresses, categories, and homepages separately, and Naver supplies the canonical name/pin/address/phone/category/homepage where it has a value; Kakao confirmation raises confidence but does not replace Naver's canonical pin. The edge validator and database ingest boundary both reject a Naver seed without raw Naver identity or a cross_verified seed without two independently attached raw provider records. Canonical Naver-led fields must match the retained Naver values; neither provider observation may inherit a merged or averaged canonical pin (00594).
• TMAP verification — enrich-tmap.mjs sets tmap_poi_id (a 4th-source confirmation) and pulls precise coords + official region names + fee/hours from TMAP POI detail. RESUMABLE + free-tier-friendly (skips already-verified rows; --limit/--max-calls).
• Naver Place-detail scrape — scrape-naver-place.mjs supplies the leading full-vs-practice evidence from Place identity, address, menus, photos, and amenities. court_count is the exception: an explicit government or venue_courts roster leads; unambiguous inventory wording such as 총 N면 or N면 운영 from Naver can only corroborate, while booking units, conflicting components, and generic 코트 N text are ignored. Records a provider-tagged Naver observation plus scrape markers, then invokes the resolver and reviewed-kind profile; it does not write a naver_scrape kind or metadata fields directly. Search-HTML → Place ID → m.place detail; identity is accepted only after normalized-name and ≤150 m pin checks. The observation stores the page's own name and pin; the privileged write endpoint reloads the canonical venue and repeats the attachment check before invoking the observation RPC, whose database implementation independently enforces the same map-observation boundary. Existing Place rows from the pre-pin payload shape remain a named refetch cohort: rows with no meaningful page name are quarantined, while name-agreeing rows are refreshed in bounded exact-ID batches instead of having coordinates invented from the venue row. Generic rows/seed enrichment routes reject scalar court counts; the dedicated government writer and database accept a bounded 1–40 value only when its field provenance is KSPO or standard-data. The Naver-seeded legacy public_courts bridge keeps its number only as raw source evidence. The club bridge may seed a manual scalar only when a venue_id-null row actually creates a new canonical venue; already-linked rows and all later court_venues count edits stay club-local, so canonical changes use an approved venue correction or a structured court roster. That narrow creation path does not increase the identity source count. Structured rosters are bounded independently and stamp roster provenance; the rows generated from legacy scalars by 00338 are not retroactively treated as host evidence. If an existing active roster and canonical non-court kind conflict, deployment stops for explicit adjudication instead of silently choosing either legacy assertion. FRAGILE + heavy (anti-bot, ~1MB/venue) → RESUMABLE (naver_scraped_at), low concurrency + delay, stops on repeated blocks; run in --limit N batches.

A venue is never bulk-created from a Kakao-only keyword result. A Naver-only private venue is the documented narrow exception and must remain provisional rather than being presented as cross-verified. Address-only region maintenance omits data_source, so re-ingesting an existing row cannot refresh or manufacture provider evidence. The canonical import_hash merge ranks each Naver-led field independently: human input, then active Naver Place evidence, then guarded Naver Local evidence, then other or evidence-free inputs. Lower-ranked rows still contribute their own observation and may fill genuine gaps, but cannot replace or relabel a stronger name, pin, phone, address, Naver category, homepage, or venue kind. Recognized government kinds remain rank-zero gap fills so they can classify an otherwise unowned row without outranking Naver or a person. A source marker alone is not authority: Naver-owned fields must equal a live observation, and same-hash imports serialize with observation mutation. naver_category and kakao_category are discarded unless the same seed carries the corresponding guarded raw provider record (00595). The legacy enrich_venues_batch marker rail is narrower still: exact active observations must back provider IDs/categories; each rating must also match the same active Naver Place observation and its numeric domain. Direct kind/capability/phone/hours/fee/indoor writes are removed before the existing court-count and homepage guards run. Matching human observations preserve the coupled classification/closure state and independently authoritative count through that legacy delegate (00596). Observation record, quarantine, and restore paths lock canonical venues before validation or observation mutation, in deterministic UUID order for batches.

Why (learned the hard way, migration 00340): a legacy pilot used Kakao keyword search as a seed. Generic district queries ("강남구 테니스장") got stored as venue names with whatever scattered coordinates the search returned, and no later pass corrected coordinates (dedup grouped by the bad coords; enrichment only layered metadata on top). Result: hundreds of venues km-off from reality and phantom rows for venues that don't exist. 00340 purged all 993 data_source='kakao' rows; the spine is now gov-seeded (~1,258). Venues gov data misses (private/indoor courts) are added via a proper registration feed or host submission (venue_corrections), NEVER a Kakao bulk seed.

---

Architecture

• Schema — venues (venue-level) + venue_courts (per-court) from 00283, extended by:
  • 00332 — venue amenity/hours/fee columns (operating_hours, fee_schedule, has_parking/shower/shop, amenities, place_refs) + a provenance/confidence layer (data_source, verification_status, field_provenance, import_hash) on both tables, mirroring 00282's rating_source/rating_state.
  • 00333 — full unique index on venues.import_hash (so ON CONFLICT can target it).
  • 00334 — public.ingest_venues_batch(jsonb): a merge-safe upsert (COALESCE per field → additive passes fill/refresh, never clobber; merges field_provenance; never downgrades host_verified). SECDEF, service_role-only.
• Edge function — supabase/functions/ingest-venues/:
  • sources/*.ts — one extractor per source → normalized SourceRecord[].
  • match.ts — name-normalize + coord-proximity (≤150m) clustering, per-field source precedence (std_data > kspo > eshare > overpass).
  • write.ts — calls the 00334 RPC (merge-safe).
  • index.ts — region-scoped orchestrator.

Provenance / confidence → app rendering

verification_status: unverified < imported (auto) < host_verified (a host confirmed) < official. The app should render by tier: verified court detail as truth; imported as provisional ("추정", host-confirm CTA); venue-level aggregate when there are no court rows; free-text session.location when there is no venue. field_provenance records each field's source for audit.

---

How to run

Deployed (--no-verify-jwt); gated by an x-ingest-token header (secret INGEST_TOKEN, value saved in the session scratchpad ingest-token.txt). Secrets: DATA_GO_KR_KEY, INGEST_TOKEN, TWOMORE_SECRET_KEY (service role), NCP_* (already set).

GET {SUPABASE_URL}/functions/v1/ingest-venues?region=<code>[&dry_run=false][&sources=std_data,overpass]
    header  x-ingest-token: <INGEST_TOKEN>

• region: 시/도 code — 11서울 26부산 27대구 28인천 29광주 30대전 31울산 36세종 41경기 42강원 43충북 44충남 45전북 46전남 47경북 48경남 50제주.
• dry_run defaults true (safe report, no write). Pass dry_run=false to write.
• sources (optional) filters the extractors run — default: all four (std_data, overpass, kspo, eshare).
• ?action=reset (token-gated) deletes every verification_status='imported' row (host / bridge rows untouched) — for a clean re-key re-run after an import_hash scheme change.
• Run regions SEQUENTIALLY — 17 concurrent invocations hit the 150s edge idle-timeout and rate-limit data.go.kr. std_data + kspo re-pull their nationwide data.go.kr feeds per region (best-effort, 50s cap); Overpass + eshare are region-scoped (fast).

---

Historical state at foundation completion (2026-07-07)

• ~1,353 distinct venues imported nationwide (all 17 regions), verification_status='imported'.
• Four sources wired + cross-merged: std_data (15013117 — hours/fees/amenities), kspo (15113986 — fresh identity/coords/phone, 폐업 filtered), eshare (공유누리 — booking URLs), overpass (named census). The coord-cell import_hash (~110m, source-order-independent) converges them onto one row per venue — e.g. 삼청테니스장 = std_data + kspo + overpass, carrying hours + fees + phone together.
• Best-effort gaps (re-runnable, merge-safe so a re-run only adds): a few regions returned std=0 or eshare=0 on their pass (data.go.kr per-region variance / transient; e.g. 광주 kspo/esh=0) — just re-run that region.

---

Directory activation + cleanup (migrations 00335–00339, 2026-07-07)

The spine is now LIVE in the app (경기 tab → 코트 찾기 + /directory-venues/[id] detail) and the imported data was cleaned/enriched to a browsable state:

• 00335 — coord-cell dedup + facility_kind classification (provenance + court_count + screen name-blocklist) + adm1_id backfill.
• 00336 + scripts/enrich-venues.mjs + enrich-venues edge fn — per-venue Kakao Local + Naver 지역 (coord-matched, local script → privileged write-back): fills phone / place-id, captures provider category (full court = 테니스 > 테니스장 leaf; academy = 테니스; Naver 공유누리 개방자원 = booking fragment; 배드민턴/스포츠시설 = non-tennis), refines facility_kind (unknown 347→18).
• 00337 — non-tennis removal + orphan-fragment deletion + name-annotation strip + dedup by kakao_place_id within 400m (4,912→2,251 venues, residual dups 544→5) + adm2_id backfill.
• 00338 — seed venue_courts from court_count (242 venues, 675 rows: surface + indoor/ outdoor); extend enrich RPC to backfill adm ids.
• 00339 + scripts/backfill-venue-adm.mjs — reverse-geocode adm-null venues via Kakao coord2regioncode (region 100% / district 94%); venue_corrections moderation-queue table behind the detail "코트 정보 확인해주기" affordance.

Follow-ups (prioritized)

Done: 4-source backfill; dedup + classification; Kakao/Naver enrichment; app read-path + directory + spine detail; venue_courts seed (count-based); adm region/district backfill; correction submission queue.

1. Richer per-court data — the seed is count-based (N면 per surface). Per-court numbering (A1–A8), per-court surface variety, and photos need the 공유누리 booking units + Naver Place scrape (m.place.naver.com __APOLLO_STATE__) — see api-naver-search.md §enrichment.
2. Apply accepted corrections — venue_corrections collects member submissions (status pending); build the admin/service apply-path (→ verification_status='host_verified', facility_kind_source='host').
3. Complete adm2 — the ~6% still null are rows without coords; needs an address source.
4. session→spine-venue links + scenario seeding at spine venues.

Known limitations

• data.go.kr rate-limits under concurrency → run sequentially; std is best-effort per region.
• Migration 00338 expanded then-current scalars into 242 venue rosters (675 court rows). Those legacy rows are useful composition data but are not, by themselves, independent host confirmation; they remain reviewable under their original evidence. Venues without a roster render the "코트 정보 준비 중 · 호스트 확인 대기" state until independently sourced or host-confirmed.