Co-founded a vendor-neutral benchmark layer for AI infrastructure primitives—web search, extraction, reranking, retrieval, OCR, vector DBs, chunking, crawling, and memory—with 100+ GitHub stars, paid evaluation customers, and CalCompute partnership traction.
Co-founder & Lead Engineer
Modern AI products are assembled from infrastructure primitives, but teams still choose search APIs, extraction tools, rerankers, vector databases, and retrieval stacks through vendor demos, anecdotes, and one-off tests. PrimitiveBench exists because “one winner” is usually a lie: the best primitive depends on the slice, workflow, budget, latency, evidence requirement, and failure mode.
A modular benchmark harness shaped as dataset → task → adapter → scorer → result schema, with frozen run contracts, deterministic seeding, public dev splits, private held-out answers, per-primitive eval packages, confidence intervals, statistical separability checks, and an MCP surface agents can query for slice-specific primitive recommendations.
Reached 100+ GitHub stars, built a public trust anchor for vendor-neutral AI primitive evaluation, converted early paid evaluation-report customers, and positioned PrimitiveBench as a benchmark partner for CalCompute and public-sector AI infrastructure decisions.
AI evaluation is not only a metrics problem; it is a trust problem. The benchmark has to be reproducible, statistically honest, resistant to contamination, neutral to vendors, and useful enough for buyers to make real infrastructure decisions.
The constraints, reliability decisions, and system boundaries behind the visible product.
Structured the harness around dataset → task → adapter → scorer → result schema so data, vendor integrations, metrics, and reporting stay isolated and reproducible.
Designed for statistical honesty: per-slice scoring, Wilson/bootstrap confidence intervals, McNemar-style separability checks, and tie bands instead of fake global winners.
Separated public dev splits from private held-out golden answers with contamination controls, canary markers, and versioned methodology so the benchmark could act as a public trust anchor.
Extended the product thesis beyond a website by exposing slice-level benchmark results through an MCP server, letting AI agents query which primitive fits a constrained workflow.
Treated benchmark releases like software releases: versioned datasets, repeatable run manifests, provider-specific adapters, captured latency and cost telemetry, and CI-compatible artifacts make regressions auditable instead of anecdotal.