qvac/GenesisIII

QVAC Genesis III

This repository contains the QVAC Genesis III dataset associated with the COLM 2026 paper QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training.

Resource	Link
📄 Paper (COLM 2026)	arXiv:2609.19513
📝 Blog	huggingface.co/blog/qvac/genesis-iii
💻 Code	github.com/tether-ai-research/qvac-genesis-III
🤗 Models	huggingface.co/collections/qvac/genesis-iii

QVAC Genesis III is a large-scale synthetic STEM corpus for efficient language-model pre-training. It contains 159,646,553 educational documents and approximately 191 billion training tokens across 19 curriculum-aligned domains, two complementary generation pipelines, and four educational writing styles.

📋 Dataset Details

Field	Value
Curated by	Davide Vitabile, Nikhil Ranjan, Akshay Nambiar, Kamal Kumar Gupta, and Amril Nazir
Organization	Tether Data, S.A. de C.V. (Tether AI Research)
Shared by	QVAC
Language	English
License	CC BY-NC 4.0
Scale	159,646,553 documents and approximately 191 billion training tokens
Domains	19 STEM domains
Generation methods	Failure Analysis and Option-Level Reasoning
Styles	educational textbook, web article, question-answer tutoring, and conversational dialogue

✨ What Genesis III Adds

QVAC Genesis III uses both successful and unsuccessful answers from an edge-scale student model as generation signals:

• Failure Analysis (failure_analysis) converts incorrect or non-extractable student responses into corrective lessons. The generated document diagnoses the likely mistake, explains the correct approach, and restates the problem so the result is self-contained.
• Option-Level Reasoning (option_level) converts correctly answered questions into contrastive explanations that justify the correct option and explain why each distractor is wrong.

Both pipelines render their output in four styles to increase structural and pedagogical diversity.

Under controlled training settings, models trained from scratch on Failure Analysis, Option-Level Reasoning, and their combination substantially improve ARC, GPQA Diamond, and MMLU STEM performance over token-matched Cosmopedia-v2 baselines.

🧪 Research Checkpoints

Models trained in the paper's controlled from-scratch experiments:

Checkpoint	Role	Hugging Face
🔍 Failure Analysis	FA data ablation	qvac-genesis-iii-qwen3-1.7b-fa
🧠 Option-Level	OL data ablation	qvac-genesis-iii-qwen3-1.7b-ol
🔗 Combined FA+OL	Full Genesis III mix	qvac-genesis-iii-qwen3-1.7b-combined

🗂️ Dataset Configurations

The dataset has one train split and 20 configurations:

• full (default): all 19 domains
• one configuration for each individual domain

The configurations reference the same underlying Parquet files; selecting a domain does not create a duplicate copy of the data.

Domains

astronomy	college_biology	college_chemistry
college_computer_science	college_maths	college_medicine
college_physics	conceptual_physics	econometrics
electronic_science	geography	high_school_biology
high_school_chemistry	high_school_computer_science	high_school_maths
high_school_physics	high_school_statistics	machine_learning
professional_medicine

Release identifiers preserve two source-directory names: electronic_science corresponds to electrical engineering, and geography corresponds to high school geography.

🧱 Dataset Structure

Each row contains the final self-contained training document and explicit metadata:

Field	Type	Description
text	string	Final synthetic educational document used for pre-training.
domain	string	One of the 19 curriculum-aligned STEM domains.
pipeline	string	failure_analysis or option_level.
style	string	educational_textbook, web_article, qa, or conversational_dialogue.

Generation prompts, generator scratch reasoning, duplicated output fields, and the redundant correctness flag are not included in the public schema. We applied the same record filter used for training, keeping documents whose text is present, non-empty, and longer than 100 characters and whose reasoning_output is present and valid. This filtering removed approximately 1.02 million incomplete or placeholder source records.

Illustrative, abridged example:

{
  "text": "**User:** I have a question about geostationary orbit...",
  "domain": "astronomy",
  "pipeline": "option_level",
  "style": "conversational_dialogue"
}

🚀 How to Load

The full corpus is very large. Streaming is strongly recommended unless you intentionally want a local copy.

from datasets import load_dataset

# Stream the complete corpus (the default configuration is "full").
dataset = load_dataset(
    "qvac/GenesisIII",
    split="train",
    streaming=True,
)
example = next(iter(dataset))
print(example)

Load one domain:

from datasets import load_dataset

astronomy = load_dataset(
    "qvac/GenesisIII",
    "astronomy",
    split="train",
    streaming=True,
)

Filter by pipeline or style:

failure_analysis = astronomy.filter(
    lambda row: row["pipeline"] == "failure_analysis"
)

textbook = astronomy.filter(
    lambda row: row["style"] == "educational_textbook"
)

🏗️ Dataset Creation

1. Seed acquisition and quality filtering

Seed passages were sampled from FineFineWeb STEM categories and mapped to curriculum-aligned target domains. The Ultra-FineWeb classifier retained high-quality passages.

2. Multiple-choice question generation

QwQ-32B generated self-contained multiple-choice questions with four mutually exclusive options and one gold answer. A format validator rejected malformed generations.

3. Student answering and answer extraction

Qwen3-1.7B-Base produced a free-form answer to each question. CompassJudger-2-32B-Instruct acted as a parser: it extracted one final option or returned a non-answer/ambiguity outcome. It was used to recover the student's committed choice, not to solve the question.

4. Dual-route educational generation

Incorrect and non-extractable responses were routed to Failure Analysis. Correct responses were routed to Option-Level Reasoning. QwQ-32B generated self-contained documents in four styles:

• educational textbook;
• web article;
• question-answer tutoring;
• conversational dialogue.

The complete methodology and prompt templates are documented in the Genesis III publication.

🛡️ Content Safety and Personal Information

The complete corpus was scanned before release. The scan covered all source text fields rather than a sample:

• The toxicity pipeline used Detoxify for broad screening and Qwen3Guard-Gen-8B for contextual review of flagged passages.
• The PII pipeline detected common personal-data categories, including email addresses, phone numbers, network addresses, physical addresses, and financial identifiers. CompassJudger-2-32B-Instruct reviewed ambiguous findings in context.
• Confirmed personal identifiers were removed by replacing them with safe, non-resolving synthetic placeholders. The cleaned corpus was then scanned again to verify that no confirmed identifiers remained and that its structure was intact.

The personal-information clean-up modified 379,336 of 160,668,773 source records, approximately 0.24%, removing confirmed identifiers through placeholder substitution. Contextual review found zero toxic passages, so no toxicity-based removals were necessary.

These automated and model-assisted scans can produce false positives or false negatives. Redaction may also affect a small amount of non-personal text that resembles an identifier. The scan addresses toxicity and personal information; it does not establish factual correctness, copyright clearance, or suitability for a particular downstream use. Although automated cleaning, filtering, and deduplication were applied, the dataset was not exhaustively audited for personal, sensitive, offensive, or otherwise harmful content. Users should perform additional screening appropriate to their use of the dataset.

🧹 Deduplication and Benchmark Decontamination

• MinHash near-deduplication with 60-gram signatures flagged 47,927 candidate pairs and identified 1,729 unique near-duplicate documents, less than 0.002% of the corpus.
• AllenAI Decon overlap checks identified 17 verified benchmark-contaminated documents across the corpus.
• All verified matches occurred in the Failure Analysis data; Option-Level Reasoning had no verified matches.
• GSM8K test leakage was 0%; MMLU test overlap was 0.005%.

These automatic and manually reviewed checks quantify known overlap but cannot prove that the corpus is free of all benchmark material. The retention or removal status of the identified near-duplicates and benchmark matches must be confirmed before publication.

🎯 Uses

✅ Direct Use

The dataset is intended for research and education purposes and in connection with those uses, for:

• pre-training and continual pre-training of language models;
• research on synthetic educational and STEM data;
• controlled studies of failure-focused and option-level supervision;
• curriculum construction by domain, pipeline, or educational style;
• research on token-efficient training of small language models.

🚫 Out-of-Scope Use

The dataset should not be treated as:

• a factual reference or source of ground truth;
• a replacement for textbooks, instructors, clinicians, engineers, or other domain experts;
• human-authored educational material;
• validated training data for medical, legal, safety-critical, or other high-stakes deployment;
• evidence that a downstream model is safe or reliable.

Models trained on this corpus require independent domain-specific and safety evaluation before deployment.

⚠️ Bias, Risks, and Limitations

• The documents are synthetic and may contain hallucinations, reasoning errors, contradictory explanations, or incorrect answer keys.
• Generator, student, parser, and filtering models can transmit their own biases and systematic errors.
• Coverage is restricted to English-language STEM and is not representative of broader languages, cultures, disciplines, or educational settings.
• Domain and difficulty distributions are uneven.
• Educational styles are generated templates, not naturally occurring classroom interactions.
• Automated PII and toxicity screening can produce both false positives and false negatives.
• Redaction placeholders can reduce fidelity or collapse distinct values into the same token.
• Benchmark decontamination is threshold-based and may miss semantic or paraphrased overlap.
• Strong benchmark results do not imply suitability for high-stakes applications.

Users should inspect and filter the corpus for their specific use case.

⚖️ Licensing

Licensing Information: This QVAC Genesis III dataset is licensed by Tether Data, S.A. de C.V. under the CC-BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/legalcode.en). As described in the blog post (https://huggingface.co/blog/qvac/genesis-iii/), the data was generated using FineFineWeb (https://huggingface.co/datasets/m-a-p/FineFineWeb) licensed under the Apache 2.0 license, the Ultra-FineWeb-classifier (https://huggingface.co/openbmb/Ultra-FineWeb-classifier) licensed under the Apache 2.0 license, QwQ-32B (https://huggingface.co/Qwen/QwQ-32B) licensed under the Apache 2.0 license and Qwen/Qwen3-1.7B-Base (https://huggingface.co/Qwen/Qwen3-1.7B-Base) licensed under the Apache 2.0 license.

FineFineWeb data was generated from FineWeb data (ODC-BY 1.0) using data that originated in Common Crawl (Creative Commons terms). The FineFineWeb authors used Qwen2-7B licensed under the Apache-2.0 license, Qwen 2-72B-Instruct licensed under the Tongyi Qianwen license, and a BERT model licensed under the Apache 2.0 license.

Users are responsible for reviewing the applicable licenses and terms of these upstream resources. The dataset is provided as-is, without warranties of factual accuracy, fitness for a particular purpose, or non-infringement.

📮 Copyright Complaints

We will take appropriate action in response to notices of copyright infringement. If you believe your work has been used or copied in a way that infringes your intellectual-property rights, email data-apps@tether.io and identify the copyrighted work and the allegedly infringing material.

📖 Citation

@misc{vitabile2026qvacgenesisiii,
  title         = {QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training},
  author        = {Davide Vitabile and Nikhil Ranjan and Akshay Nambiar and Kamal Kumar Gupta and Amril Nazir},
  year          = {2026},
  eprint        = {2609.19513},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  institution   = {Tether Data, S.A. de C.V. d.b.a. Tether AI Research},
  note          = {Accepted at the Conference on Language Modeling (COLM) 2026},
  url           = {https://arxiv.org/abs/2609.19513}
}

✉️ Dataset Card Contact

Questions and feedback can be submitted through the QVAC organization on Hugging Face. Copyright complaints should be sent to data-apps@tether.io.