Instructions to use sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")
model = AutoModelForCausalLM.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B

SGLang

How to use sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B with Docker Model Runner:
```
docker model run hf.co/sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B
```

⚠️ EXPERIMENTAL / ALPHA — NOT FULLY TESTED

Early-stage derivative, not a fully benchmarked release. Intended as a base for task-specific LoRA fine-tuning (16GB-VRAM GPU deployments, 24GB+ laptops/desktops). Expect rough edges — validate on your own workload before relying on it.

Nemotron 3.5 Lightning 30B-A3B — REAP-20B (general calibration)

A 20B-total / 3B-active derivative of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, pruned with REAP expert pruning (128 → 77 routed experts per MoE layer, 40% sparsity), calibrated on a general-purpose corpus with a slight coding focus (general instruction 44% / coding 33% / math 22%, 538 samples generated by the full model). MTP head stripped. No LoRA included — this is the pre-LoRA base; a domain LoRA (subagent orchestration + coding/tool-calling mix) is the intended next step.


Base model	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Params before / after	31.58B → 19.87B (active stays ~3B)
Pruning	REAP, 40% expert sparsity, 128→77 experts/layer × 23 MoE layers
Calibration data	538 samples: instruction 238, coding 180, math 120 (from the owner's general/coding/math prompt banks)
Quantization	None in this repo (BF16); IQ4_NL GGUF ≈ 11.5GB (see below)
License	OpenMDW-1.1 (same as base)

This repository ships:

the REAPed model (weights + config, model-*.safetensors)
the methodology + recalibration kit (reap/): how the pruning was calibrated, the prompt-bank mix, and how to redo it on your own data.

Why prune

Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + Attention model with hidden=2688, moe_intermediate=1856. Neither is divisible by 256, so llama.cpp K/IQ quant formats (256-element superblocks) cannot represent the expert weights — every GGUF quant falls back to block-32 formats (~4.5 bpw floor) regardless of requested bit width. At 30B that floor is ~18GB; at 20B it is ~11.5GB. REAP removes the redundant experts so the model fits smaller quantization budgets without touching the active computation:

~37% smaller total memory at the same speed (3B active unchanged)
KV cache stays ~6KB/token (only 6 attention layers, 2 KV heads), so long contexts stay cheap on consumer hardware
A Q4_0/IQ4_NL GGUF ≈ 11.5GB fits a 24GB MacBook with room for 32K+ context (the block-32 floor is why IQ3-class GGUF quants are not achievable for this architecture at any size)

Calibration data (the important part)

REAP prunes the experts your calibration data doesn't use. This model was calibrated on a general-purpose mix so the surviving experts serve general use:

Source	Share	Content
instruction (openhermes-style)	44%	general QA, writing, reasoning, how-to
coding (deepseek trajectories + opencode)	33%	coding problems, agentic code tasks
math (metamath-style)	22%	word problems, arithmetic, reasoning

Completions were generated by the full model (NVFP4, natural thinking, per-domain token budgets). If your workload is heavily domain-specific, re-calibrate on your own data — see reap/.

Using this repo

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
    torch_dtype=torch.bfloat16, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")

For llama.cpp / LM Studio on Apple Silicon, quantize to IQ4_NL (~11.5GB):

llama-quantize model.gguf out-IQ4_NL.gguf IQ4_NL
llama-server -m out-IQ4_NL.gguf -c 131072 --reasoning-parser nemotron_v3

Context length: -c 131072 (128K) is a safe default for 16GB-VRAM GPUs and 24GB Macs — KV is only ~6KB/token (6 attention layers, 2 KV heads), so 128K context costs roughly 0.8GB. The model itself supports up to 1M tokens (256K is NVIDIA's single-GPU deployment length); on laptops the practical limit at longer contexts is prefill time, not memory — raise -c (e.g. 262144) if your workload needs it.

Validation (limited — alpha)

Sanity-checked with greedy generation on general QA, coding, and math prompts: coherent analysis, correct answers (e.g. math arithmetic verified), no repetition/looping. Full benchmark suites were not run. The closest published pruning analogue (Qwen3-30B-A3B, 128 experts, 8 active) retained ~99.8% of baseline at 50% sparsity.

Sources & how to make one yourself

Base model: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Algorithm: REAP — Router-weighted Expert Activation Pruning (Cerebras; implemented in vllm-project/llm-compressor)
Pruning tooling: llm-compressor's REAPPruningModifier, plus 3 small patches for the NemotronH architecture (documented in reap/README.md)
Calibration data: 538 samples generated by the full model from general instruction, coding, and math prompt banks (openhermes-style instruction data, DeepSeek coding-trajectory prompts, metamath-style word problems) — see reap/generate_general_data.py
Synthetic-data guidance: NVIDIA's Build a Bash Agent with Synthetic Data and sft-data-blending patterns
Why 20B / the quant floor: this architecture's hidden=2688 / moe_intermediate=1856 are not 256-divisible, so llama.cpp GGUF quants floor at block-32 formats (~4.5 bpw); at 20B that is ~11.5GB instead of ~18GB

To reproduce on your own data: serve the full model, generate your domain corpus with reap/generate_general_data.py (or your own loader), then run reap/reap_nemotron.py --data yours.jsonl --sparsity 0.40. Full step-by-step, patches, and hardware notes: reap/README.md.

LoRA adapter + IQ4_NL GGUF (recovery pass + quantized for laptops): Nemotron-3.5-Lightning-30B-A3B-REAP-20B-LoRA-IQ4NL
Base: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

License

OpenMDW License Agreement, version 1.1 (same as the base model). See LICENSE. If you upload derived model files, carry the same license and attribute NVIDIA for the base weights.

Downloads last month: 128

Safetensors

Model size

20B params

Tensor type

F32

BF16

Model tree for sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B

Base model

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Finetuned

(13)

this model

Adapters

1 model

Paper for sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Paper • 2510.13999 • Published Oct 15, 2025 • 23

sleepyeldrazi
/

Nemotron-3.5-Lightning-30B-A3B-REAP-20B