Checkpoint

last edited

A checkpoint is a saved snapshot of a model's learned weights at one point in training. Training code writes a checkpoint at intervals so a run can resume after a crash. The final checkpoint, or the best one on a validation set, becomes the released model. The term covers language, image and audio models alike.

Training checkpoints

A resumable checkpoint also stores the optimizer state, the step count and the schedule position. Adam-style optimizers keep two extra values per weight, so this state can outweigh the weights. PyTorch Distributed Checkpoint saves shards from every GPU in parallel and can reshard them on load. Its asynchronous save stages state in CPU memory, then writes while training continues.

Released checkpoints

A release usually keeps only the weights, often at reduced precision. Labs publish base checkpoints, and fine-tunes on hubs such as Civitai specialize them. An image checkpoint may bundle a diffusion model, text encoder and VAE. The same prompt and seed give different results on each checkpoint.

Weight means from nearby checkpoints of one run can beat any single one, and merges of separate fine-tunes form whole lineages. A LoRA or embedding leaves the checkpoint frozen.

History

The word comes from training jobs: TensorFlow course notes from 2017 call the step where variables are saved a checkpoint. The original Stable Diffusion weights shipped as .ckpt files in August 2022. Those were pickle files, which can run arbitrary code when opened.

Safetensors

Nicolas Patry at Hugging Face started safetensors in 2022: a JSON header plus raw tensor bytes, with no code to run. A 2023 audit by Trail of Bits for EleutherAI and Stability AI made it the default. It joined the PyTorch Foundation in April 2026. Its 0.9 release candidate adds a faster prefetch loader.

Sources

These primary pages cover how the safetensors checkpoint format was started, audited and adopted as a default. The repository history dates the first commit, and the audit post names the partners.