Reproduce before trusting
Method & neutrality
CanaryIndex publishes the fixtures, thresholds, raw observations, ownership disclosures and deterministic ranking rules behind every answer. A missing observation never becomes a pass.
What is measured
| Category | Fixture | Pass threshold | Rights |
|---|---|---|---|
| Document to Markdown | w3c-dummy-pdf-v1 | 80% | Public W3C accessibility test fixture; only the three expected words are republished. |
| Audio and video to text | nasa-armstrong-v1 | 70% | NASA recording in the public domain; attribution and source are retained. |
Objective scores
- Document quality is exact expected-token recall on a one-page W3C fixture.
- Transcription quality is
1 − word error rateagainst the best match among two published interpretations of the short, historically ambiguous NASA phrase. - Reliability is accepted canaries divided by completed observations. Monitoring failures are visible but do not become product failures.
- Effective cost per accepted unit includes comparative spend from failed completed observations.
- Publisher-owned Actors use the same scorer. Their simulated public price is labeled because Apify exempts owner self-runs from custom events.
Published ranking rules
- Require a public listing and a current passing canary.
- Best-quality order: quality, historical reliability, effective cost, then latency.
- Budget routes: filter at the requested ceiling, then sort cost, quality and reliability.
- Exclude payment, sponsorship, affiliate status and publisher ownership from every ordering.
Budget safety
Before every call, its configured worst-case cap ($0.02–$0.15 in v0.1) is reserved in a public ledger. Pending or unknown reservations retain their full value. One complete weekly suite can reserve at most $0.36; the UTC monthly cap is $5. Only PAY_PER_EVENT Actors are called, and each receives maxTotalChargeUsd.
What a pass does not mean
A pass is not a security audit, privacy certification, endorsement, or promise about other inputs. A failed-to-observe result is not a tool failure. Results can drift.