Dataset RR-DS-0029 CC-BY-4.0 / Crawl Terms

Common Crawl Web Corpus

Petabyte-scale open repository of web crawl data distributed in standardized WARC, WAT, and WET formats.

86 /100 Verified
Assessed 2026-10-02
Version: CC-MAIN-2026-38

Dimension Breakdown

Provenance & Lineage (25%) 84/100

Complete crawl timestamps, IP metadata, and WARC headers preserved; downstream domain copyright heterogeneity requires filtering.

Empirical Evidence & Reproducibility (30%) 88/100

Immutable S3/HTTPS crawl snapshots enable exact dataset reconstruction across academic and industrial pipelines.

Operational Reliability & Maintenance (25%) 85/100

Consistent monthly crawl cadence since 2011 with publicly accessible Parquet columnar index tables.

Open Standards & Interoperability (20%) 88/100

Built on ISO 28500 WARC standard and Apache Parquet URL indices.

Assessment Findings & Analysis

Executive Assessment

Common Crawl remains the primary open benchmark corpus for web-scale linguistics, information retrieval, and foundation model pretraining. Its immutable monthly partitions provide strong reproducibility guarantees.

Key Strengths

  • Immutable Snapshots: Partition identifiers (CC-MAIN-YYYY-WW) allow researchers to cite exact byte ranges and reproduce filtering pipelines deterministically.
  • Open Format Compliance: Native WARC and Parquet manifests integrate cleanly with distributed compute frameworks.

Risk & Limitations

  • Upstream Licensing Heterogeneity: While crawl archives are openly distributed, individual crawled pages retain their original copyright status, requiring downstream compliance filtering.

Verified Evidence Artifacts

Artifact ID Type Description Verification Hash / Reference
EV-029-A Schema ISO 28500 WARC / Parquet Columnar Index Validation s3://commoncrawl/cc-index/table/cc-main/
EV-029-B Audit Crawl Manifest Integrity & Robots.txt Compliance Report commoncrawl.org/blog/2026-crawl-metrics