Deletion Tuning adapts an LLM by zeroing fewer than 0.05 percent of its weights
A NeurIPS 2026 paper with code claims that deleting a tiny, learned set of weights matches full fine-tuning, and forgets less. The code release is one commit old.
Fine-tuning usually means nudging a model’s weights, all of them or a small added set. A new paper from researchers at New York University and the University of Calgary tries something blunter: pick a very small number of existing weights and set them to zero. Change nothing else.
The code release for “The Surprising Effectiveness of Deleting Weights in LLM Post-Training” went up this week. The authors, Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou, and Mengye Ren, call the method Deletion Tuning. The paper is listed as NeurIPS 2026.
How it works
According to the README, the method learns a binary mask over one part of each transformer block, the gate projection. Where the mask selects a weight, that weight becomes zero. Every other weight keeps its pretrained value.
The mask is what gets trained. It starts from Wanda scores computed in a calibration step, and training decides which entries to switch off. The README names a regularizer but does not spell out the objective; for that you need the paper, which we could not open during this run.
The authors report that zeroing fewer than 0.05 percent of the weights matches full fine-tuning across three kinds of post-training: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation.
Two side effects follow from the design. First, the result of training is a mask, not a second copy of the model. The README says trained masks are hundreds to thousands of times smaller than a full fine-tuned checkpoint. Second, the authors say the model forgets less. Under reinforcement learning, accuracy on earlier tasks stays near the base model’s, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 points.
What was tested
The README lists the setups but has no results tables, so we have no scores to print.
| Regime | Models | Tasks |
|---|---|---|
| RL (GRPO) | Llama 3.2 1B and 3B, Llama 3 8B | GSM8K |
| RL (GRPO) | Qwen2.5 0.5B to 14B, Qwen3-30B-A3B | SimpleRL-Zoo |
| Supervised fine-tuning | Llama 3.2 1B and 3B | MMLU-Pro, BBEH |
| Self-distillation | Qwen2.5-7B-Instruct | ToolAlpaca, Science |
These are small and mid-sized models, up to a 30B mixture-of-experts. Every claim above is the authors’ own, and we have not seen an independent replication.
The release is also incomplete in a way that matters. Its roadmap lists the LoRA baseline and the geometric analysis as not yet published, so the comparison against LoRA cannot be reproduced from this code today. The repo is a single commit.
Try it yourself
The repo is MIT licensed. It provides shell scripts for each regime, a separate uv environment for each, and a script that applies a mask to a base model and saves a normal checkpoint. Pretrained masks are linked on Hugging Face. The README does not state GPU requirements; the stack includes vLLM and CUDA builds, so plan on a real GPU.
We have not run the code or read the full paper; this is based on the repository’s README.