Initial release

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+pareto_doc_ner.png filter=lfs diff=lfs merge=lfs -text
+camembert-bio-logo.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,112 @@
+---
+language:
+- fr
+license: mit
+library_name: transformers
+tags:
+- biomedical
+- clinical
+- encoder
+- modernbert
+- fill-mask
+datasets:
+- rntc/biomed-fr
+- almanach/OntoBook
+base_model:
+- almanach/ModernCamemBERT-bio-large
+pipeline_tag: fill-mask
+widget:
+- text: "Le patient présente une [MASK] aiguë du myocarde."
+---
+
+<a href=https://camembert-bio-model.fr/>
+  <img width="210px" src="./camembert-bio-logo.png">
+</a>
+
+# ModernCamemBERT-bio-v2-large
+
+ModernCamemBERT-bio-v2-large is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of [ModernCamemBERT-bio-large](https://huggingface.co/almanach/ModernCamemBERT-bio-large) with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 72.8 micro-F1, the best score we measured for coding and long-document classification at this size.
+
+### Authors
+
+Rian Touchent and Eric de la Clergerie  
+Sorbonne Université, Inria Paris
+
+## Overview
+
+ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Rather than plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The goal is to inject structured coding knowledge without adding any task-specific head. This model is our document specialist.
+
+| Architecture | ModernBERT |
+|---|---|
+| Parameters | 350M |
+| Context length | 8,192 tokens |
+| Language | French |
+| Base model | [almanach/ModernCamemBERT-bio-large](https://huggingface.co/almanach/ModernCamemBERT-bio-large) |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+model_id = "rntc/ModernCamemBERT-bio-v2-large"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForMaskedLM.from_pretrained(model_id)
+
+text = "Le patient présente une [MASK] aiguë du myocarde."
+inputs = tokenizer(text, return_tensors="pt")
+logits = model(**inputs).logits
+i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
+print(tokenizer.decode(logits[0, i].argmax(-1)))
+```
+
+## Evaluation
+
+Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.
+
+| Model | FrACCO-30 | FrACCO-100 | CANTEMIST | MORFITT | MedDialog-FR | Avg |
+|---|---|---|---|---|---|---|
+| ModernCamemBERT-bio-v2-large | 82.0 | 67.8 | 74.6 | 74.3 | 65.2 | 72.8 |
+| ModernCamemBERT-bio-v2-base | 78.0 | 63.7 | 70.9 | 73.5 | 63.7 | 70.0 |
+| ModernCamemBERT-bio-base | 74.2 | 61.1 | 71.2 | 72.9 | 63.7 | 68.6 |
+| DoctoModernBERT | 74.0 | 58.3 | 69.3 | 73.9 | 64.7 | 68.0 |
+| DrBERT | 54.2 | 39.6 | 41.2 | 71.0 | 64.1 | 54.0 |
+| DoctoBERT | 51.6 | 34.7 | 36.2 | 73.5 | 62.1 | 51.6 |
+| CamemBERT-bio | 44.0 | 22.3 | 17.7 | 70.9 | 45.8 | 40.1 |
+
+ModernCamemBERT-bio-v2-large has the best average and the best score on each coding task. It scores 4.2 points above ModernCamemBERT-bio-base, the strongest publicly available French biomedical encoder we evaluated.
+
+
+## Environmental impact
+
+The OntoBook phase reported here ran for about 3.4 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.10 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.
+
+## License
+
+MIT
+
+## Citation
+
+```bibtex
+@inproceedings{touchent:hal-05697506,
+  TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
+  AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
+  URL = {https://hal.science/hal-05697506},
+  BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
+  ADDRESS = {Palma de Mallorca, Spain},
+  YEAR = {2026},
+  MONTH = May,
+  PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
+  HAL_ID = {hal-05697506},
+  HAL_VERSION = {v1},
+}
+
+@misc{touchent2026causallanguagemodelingdetour,
+  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
+  author={Rian Touchent and Eric de la Clergerie},
+  year={2026},
+  eprint={2605.12438},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2605.12438},
+}
+```

config.json

@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "embedding_dropout": 0.0,
+  "eos_token_id": 2,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 0,
+  "position_embedding_type": "rotary",
+  "sep_token_id": 2,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.0",
+  "vocab_size": 32768
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b183c79f95db3f0e551d20c29e4f0681209b32dfd9f3a76b106490bd515e4b8
+size 1645602688

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,61 @@
+{
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "errors": "replace",
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "trim_offsets": true,
+  "unk_token": "[UNK]"
+}