# Methodology — RuntimeWire same-story & content-quality population study

Run date: 2026-08-17. Window: articles/items with `published_at >= 2026-08-01T05:00:00Z` and
`< 2026-08-16T05:00:00Z` (Aug 1–15, America/Chicago). All data extracted from the production database in a
single read-only pass; extraction start/finish and DB `now()` timestamps are recorded in
`audit_manifest.json`. This is a population study: every eligible RuntimeWire (RW) article and every
eligible competitor item in the window was processed — no sampling at corpus, cluster, or pair stage.
The only sampling anywhere is the seeded random win-sample drawn for the human audit (seed 424242) and
bootstrap resampling for CIs (seed 20260817).

## 1. Corpus construction

**RW side:** all `articles` rows with `status='published'` in-window. Digest/roundup formats (daily
recaps, link roundups; 11 items) excluded — they aggregate many stories and cannot be graded as a single
same-story comparison; their ids are listed in `audit_manifest.json` exclusions.

**Competitor side (Analysis A):** all `ingested_stories` rows in-window from the feeds of the 11 named
competitors (TechCrunch, VentureBeat, The Verge, WIRED, Bloomberg Technology, Axios, Ars Technica,
404 Media, The Information, SiliconANGLE, Crunchbase News). Effective completeness varies by feed and is
reported per-outlet in `source_health.csv`; see §8 Limitations — this is effectively a 9-feed study, and
Bloomberg's feed delivers truncated/paywalled text.

**Axios classification:** Axios is a general-news outlet; a gpt-5.5 classifier split its in-window items
into tech (35) vs general (138). Tech-classified items enter volume denominators; pair formation allowed
any Axios item that clustered with an RW story (classification recorded per-item in `corpus.csv`).

**Analysis B (cited sources):** every outbound link in RW article bodies/source lists was extracted
(3,252 links), resolved against the DB URL index, and typed by a gpt-5.5 classifier into
{company_announcement, social, primary_document, research, wire_pr, named_competitor, other_news} —
distribution including unmatched links is in `direct_source_results.csv` companion counts. Gradeable B
pairs require the cited source to be matched in the DB with ≥800 chars of text: 655 pairs formed,
512 valid. Analyses A and B are never conflated: A = independent competitor coverage of the same story;
B = RW vs the sources it cites, stratified by source type.

## 2. Embedding & candidate clustering

Texts embedded with Gemini `gemini-embedding-001` (768-dim), mean-centered and re-normalized (centering
computed over the full corpus; removes the shared "tech news" direction). Candidate same-story edges:
cross-outlet cosine ≥ 0.40, OR shared extracted entity AND cosine ≥ 0.30; same-outlet edges require
≥ 0.55 (dedupes republications without merging an outlet's distinct stories). Thresholds calibrated on a
labeled subsample (`04b_calibrate`); near-miss band logged to `pair_candidates.csv` with reasons so the
cut is auditable.

## 3. LLM edge verification & cluster QC

Every candidate edge was verified by gpt-5-mini ("do these two items primarily cover the same specific
news development?"); only SAME_EVENT verdicts union into clusters. Chain-merged clusters (A~B, B~C, A≁C)
were detected by diameter checks; 5 were split by gpt-5.5 into coherent sub-clusters (`cluster_splits`
records the decisions). Final: 1,531 clusters, 237 multi-outlet, 109 containing RW + a named competitor.
As a final gate, the pair judges re-ask same-story blind; pairs where both passes said not-same-story were
invalidated (3 in A) — cluster QC and grading agree at the 98.6% level.

## 4. Pair formation

**A:** within each cluster, RW article × each named-competitor member → 276 pairs; validity floor:
competitor text ≥ 2,500 chars raw (208 valid). **B:** RW article × each cited source with ≥ 800 chars
(512 valid of 655). Caps: judge-time texts capped at 8,000 chars (sentence-boundary) by de-identification;
corpus bodies capped at ingest at 12,000 for some feeds (`comp_capped` flagged per pair).

**Known limitation (quantified post-hoc):** the validity floor measures RAW chars, but boilerplate-heavy
pages (video-player chrome, consent banners, related-link rails) pass the floor with under 1,000 chars of
prose. `diag_prose_density.csv` reports post-de-ID prose density per A pair (line-level heuristic:
drops nav/markup/consent lines, <40-char lines, low-letter-ratio lines). 85 of 205 graded A pairs have
stub-class competitor text (<1,500 prose chars) — 64 of 66 Bloomberg pairs, 20 of 72 TechCrunch. Findings
report headline results both ways (all graded vs clean-text-only). Future runs should gate validity on
prose chars.

## 5. De-identification (blind grading)

`deid.mjs` strips outlet self-references (alias list per outlet → "[PUB]"), bylines, URLs (→ "[link]"),
and known boilerplate lines before judging; 8,000-char sentence-boundary cap. Axios's house style
("Why it matters:" structure) is a residual tell that cannot be scrubbed without destroying the text —
disclosed, affects 7 valid A pairs. RW texts contain no outlet name after de-ID; judged blind in
randomized A/B position.

## 6. Blind grading & adjudication

Two independent gpt-5-mini passes per pair; A/B position per pass via
`hashSeed(pair_id + "|pass" + n) & 1` (deterministic, recorded). Rubric (verbatim in
`audit_manifest.json`): same-story gate; 0–10 scores on originality, depth, interestingness,
factual_specificity; unique-fact counts with examples; role tags; winner/confidence/scoop + rationale.
Composite = equal-weight mean of the four dimensions; pair verdict = sign of mean composite gap across
passes. gpt-5.5 adjudication (pass 3) triggered by: winner disagreement, same-story disagreement, low
confidence, |composite gap| ≤ 1, or a missing pass. ~7 transient JSON-parse failures were retried to
completion (none dropped). Both-passes-not-same-story → pair invalid.

## 7. Denominators & statistics

Three denominators reported throughout: (1) pair-level (every RW×competitor pairing); (2) unique-RW-article
level (article counted once; wins if it beats all paired competitors, loses if it loses any — see
`pair_results.csv` for the exact roll-up rule); (3) cluster-level (RW vs the best competitor result in the
cluster). Wilson 95% intervals on proportions; cluster bootstrap (resample clusters with replacement,
B=10,000, seed 20260817) for the pair-level win rate, respecting within-cluster correlation. Sensitivity
analyses: competitor text ≥ 5,000 chars; capped-competitor-bodies-only; no-judge-truncation;
clean-prose-only (from the density diagnostic).

## 8. Limitations

1. **Feed completeness (9-feed reality).** The Information and SiliconANGLE feeds deliver citation stubs
   only — materially incomplete, reported but NOT ranked. Ars Technica delivers excerpt-only bodies →
   0 valid pairs. Crunchbase News: 0 pairs in-window. WIRED's feed is its AI vertical.
2. **Bloomberg is paywall/feed-truncated.** Median usable text 3,894 raw chars, 499 prose chars; several
   items are video-segment pages whose prose is 1–3 sentences. Bloomberg pair results measure RW vs what
   Bloomberg's feed exposes, not vs Bloomberg's journalism. Reported as an artifact throughout findings.
3. **Text pollution.** TechCrunch/WIRED bodies frequently open with anti-bot interstitial text and
   nav/share cruft, taxing the competitor's judge budget (pro-RW bias in wins). Competitor wins over RW
   survived this handicap (strengthens loss validity). Quantified in `diag_prose_density.csv`.
4. **Judge model family.** Grader gpt-5-mini and adjudicator gpt-5.5 share a vendor; style affinity with
   RW's house style cannot be fully excluded. Mitigations: blind de-ID, position randomization, two
   passes, human audit of all losses + seeded win sample (zero verdicts overturned).
5. **B gradeable pool.** 512 of 3,252 cited links (16%) were gradeable (DB-matched + ≥800 chars); B
   results describe that pool, not all citations. Full citation-type mix reported alongside.
6. **Audit scope deviation from spec.** Spec asked for human review of "all large-gap" pairs; 89 pairs
   had |gap| ≥ 4, so the audit reviewed the top 10 per direction + ALL 22 losses + all ties/not-same-story
   + 10 seeded random wins + all VentureBeat pairs + a B-loss skim (notes in `audit_notes.md`).
7. **Window.** 15 days (Aug 1–15, 2026). Competitive standings in a news cycle this short are dominated
   by which stories broke; generalize with care. Prior-run comparison included in findings.
