RDR-004 Accepted Data Interchange

Adopt Apache Arrow & Parquet for Bulk Evaluation Dataset Distribution

Mandate Apache Parquet for bulk archival snapshots and Apache Arrow IPC for streaming evaluation telemetry.

Recorded: 2026-09-20
Review Cycle: Annual (Next: 2027-09)

Context & Problem Statement

Publishing large evaluation registries and benchmark histories solely as paginated JSON endpoints creates high egress overhead and slows analytical replication by third-party auditors.

Decision

All bulk ResourceRank dataset exports shall be published as Apache Parquet files accompanied by SHA-256 manifests, using Apache Arrow schemas as the canonical type system.

Consequences

  • Positive: Up to 80% storage reduction over raw JSONL; direct querying via DuckDB, Polars, Spark, and R without custom parsers.
  • Trade-off: Human inspection requires CLI tooling (parquet-tools or DuckDB) rather than a plain text editor.