|
|
--- |
|
|
viewer: false |
|
|
license: |
|
|
- apache-2.0 |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
|
|
|
**Model Summary** |
|
|
|
|
|
In order to be able to reproduce GneissWeb, we provide here a [Bloom filter](https://dl.acm.org/doi/10.1145/362686.362692) representing all the document ids of FineWeb 1.1.0 whose documents are part of GneissWeb. it is of size 28GB and is of the [rbloom](https://github.com/KenanHanke/rbloom) family of Bloom filters. It is to be probed with the id column of FineWeb 1.1.0 or of Common Crawl. |
|
|
|
|
|
Please refer to the [GneissWeb](https://huggingface.co/datasets/ibm-granite/GneissWeb) page for more details. |
|
|
|
|
|
**Developers**: IBM Research |
|
|
|
|
|
**Release Date**: Feb 21st, 2025 |
|
|
|
|
|
**License**: Apache 2.0. |
|
|
|
|
|
**Testing** |
|
|
|
|
|
The Bloom Filter was tested with |
|
|
|
|
|
Positive Examples : ~10M uuids from 192 parquet files in GneissWeb. These span all 96 snapshots |
|
|
|
|
|
Negative Examples : 10,000 uuids in CC-MAIN-2024-51 (not present in FineWeb 1.1.0 and also not in GneissWeb) |
|
|
|
|
|
The Bloom Filter was able to return correct answers for all of them |
|
|
|
