A measurement stack, open to everyone.
Open-source tools and data for rigorous AI measurement: an evaluation framework, an enterprise benchmark bank, and a browser-based benchmark check-up. All in early construction, with a public roadmap.
Built on firm commitments.
From estimation to inspection, every piece stands on its own and reinforces the others — shared infrastructure for the next generation of enterprise AI evaluation.
Composable by default
Metrics, data, and uncertainty estimates work together without custom pipelines.
Measurement-aware outputs
Every output makes its assumptions visible: scores always carry uncertainty and comparability information.
Built for community
Enterprises and researchers alike can adopt, extend, and contribute back to shared infrastructure.
aims-eval
An open-source framework for enterprise evaluation: construct definition, task sampling, uncertainty estimation, and report generation in one pipeline.
- Construct-first evaluation configs
- Uncertainty & significance built in
- Dual reports: for engineers and for the board
- Compatible with mainstream inference stacks
bench-db
A standardized bank of evaluation results for enterprise scenarios, designed for validity and reliability work.
- Item-level model responses
- Enterprise-first domain coverage
- Versioning with leakage-proof rotation
- Anonymization and compliance by default
bench-caliper
A benchmark check-up tool in the browser: inspect item behavior, reliability, and what a score actually measures.
- Item behavior inspection
- Reliability diagnostics
- What a score actually measures
- Runs entirely in the browser — data never leaves
Adopt it, extend it, contribute back.
The stack is early — which is exactly when joining matters most. Write for early access, or subscribe for release notes.