# search.sitv.app — inclusion standard

## The bar

An entry is indexed only when it meets ALL of:

- **Total >= 6 of 8 points** (signals below)
- **Live probe passes** (its public URL answers 2xx)
- **A public machine-readable interface exists** (OpenAPI spec,
  instructions endpoint, or discovery API answering 2xx without a key)

Entries below the bar are listed in `index/pending.json` with the
missing signals, so the gap is visible and fixable. Nothing is hidden.

## Signals (max 8)

| Signal | Points | Rule |
|---|---|---|
| machine | 2 (public) / 1 (gated) | OpenAPI spec, instructions endpoint, or discovery API answers 2xx without a key (2). Exists but answers 401/403 without a key (1) — gated does NOT satisfy the public-interface requirement. An MCP endpoint answering a JSON-RPC `initialize` POST with 200 + `serverInfo` also scores machine = 2 (and live = 2 when the endpoint is the record URL); 401/403 on initialize scores gated = 1. |
| auth declared | 1 | The entry declares an auth method and its docs URL is live. |
| live | 2 | The entry's public URL answers 2xx. Hard gate. The live signal uses the DIRECT response code: a redirect to a login page does not count. |
| llms.txt | 1 | The entry's helpers answer 2xx: `llms.txt`, `llms-full.txt`, or `agents.txt` — 1 point total if any answers, never stacked per file. For repository links (github.com, gitlab.com) the helpers are checked inside that repository (raw file paths), not at the host origin, so a host-level file never inflates a repo link's score. |
| ownership proof | 1 | A declared proof URL (DNS TXT or /.well-known file) answers 2xx. |
| example call | 1 | The discovery/API URL answers 2xx with JSON without a key. |

## Ranking (agents first)

`final = bm25_norm*1.0 + readiness*0.6 + freshness*0.3 + authority*0.2`

- **BM25** (k1=1.2, b=0.75) over title + description + capabilities +
  tool names + enriched text (SEO title/meta, helper bodies, spec tags
  and operation summaries, MCP tool descriptions). Normalized by the
  best match in the result set.
- **Readiness** = signals total / 8. This is the agent-first edge:
  Google ranks for human popularity; this engine ranks for whether an
  agent can actually use the result right now.
- **Freshness** from `last_seen` at query time: <=7d = 1.0, <=30d = 0.7,
  <=90d = 0.4, older = 0.1.
- **Authority** is 0 in v1. No link graph is computed yet; the field is
  reserved for a future in-corpus indegree signal. It is not faked.

Empty query = browse mode: readiness desc, then freshness desc.
Every result carries its `score_breakdown` so an agent can re-rank
instead of blindly trusting the order.

## Probe cadence

Daily automated run (11:00 UTC): probe-batch 3000 queued records,
recheck 500 oldest-checked records, and a Phase 2 hunting budget of
500 records; Sunday full re-walk. `last_seen` updates only on a
passing probe — a stale entry decays in freshness instead of
silently pretending to be fresh.

## Redirects and URL corrections

- Live uses the DIRECT response code, never the final code: a 303
  login redirect to a 200 login page still scores live = 0.
- Docs, machine, and example URLs use the FINAL code when their
  direct code is 3xx: the document is reachable through its redirect.
- A permanent redirect (301/308) on the record URL corrects the URL
  to the final URL, keeps the old URL as `previous_url`, re-probes
  at the new URL in the same run, and records the correction source
  (`redirect`, or `openapi servers` when a spec's servers list
  supplies the corrected URL). A temporary redirect (302/307) keeps
  the original URL and notes the target in `probe_status`.
- Record ids never change when a URL changes, and dedupe matches
  `previous_url` so a corrected entry never mints a second id.
- Seed URLs are never rewritten: seed.json is hand-curated.

## Hunting (Phase 2)

After cheap verification, a daily hunting pass digs for the exact
missing points on live records, in this order per record: recorded
URLs, helpers (parsed for spec/MCP/docs links), conventional spec
paths (`/openapi.json`, `/swagger.json`, `/api-docs`,
`/api/openapi.json`, `/v1/openapi.json`), docs HTML links, then
repository files (README.md). Priority: records one or two points
short of the bar first (machine + live already proven), then live
records still missing machine. Hunting captures metadata only —
spec URL, version, content hash, servers, path/operation counts,
security schemes, tags; MCP endpoint, transport, serverInfo, tools;
auth types and where declared; helpers present and links extracted;
example URL and probe result; enriched matchable text. An example
URL is used only when docs or a spec names one — it is never
invented. MCP evidence is never sticky: machine-from-MCP is
re-verified by a fresh initialize on every recheck and hunt, and a
failed re-initialize falls back to the GET-based machine value.