DatologyAI’s Zephon Keeps Training Batches Identical When a Run Moves Between 8 and 64 GPUs

DatologyAI has open-sourced Zephon, the data loader it uses for its own text and multimodal training experiments. The repository went public on October 7, 2026 under the Apache-2.0 license, and version 0.1.0 reached PyPI the same day. Zephon’s central promise is what the company calls elastic determinism: a training run sees the same sequence of global batches whether it runs on 8, 16, 32 or 64 GPUs. That holds even when the pipeline tokenizes, packs and mixes data on the fly rather than reading a fully preprocessed copy from disk. A paper describing the system appeared on arXiv on October 2.

DatologyAI, which works on training-data curation, says it runs thousands of data experiments and that a growing share are now designed and run by agents. Zephon is meant to remove GPU count as a hidden variable.

When the GPU count quietly changes the data

Most loaders split data among GPUs directly. If a run starts on 16 GPUs and resumes on 8 after a fault, each GPU sees a different slice than before, and the model trains on a different sequence. DatologyAI’s blog post shows how much that matters. It trained the same 1-billion-parameter model from scratch for 20 billion tokens on 8, 16, 32 and 64 GPUs with a standard loader. Scores on the FineWeb evaluation suite varied by up to 0.64 points across GPU counts, and DCLM Core v1 scores by up to 0.82 points.

The company says that is roughly four times the thresholds the FineWeb and DCLM authors used when judging data curation methods. An ablation comparing two data mixtures on two different allocations could end up measuring the hardware instead of the data. With Zephon, the spread was 0.011 and 0.05 points. In a separate resume test, a checkpoint taken on 16 GPUs was restarted on 8, 32 and 64. With Zephon, every global batch’s SHA-256 hash matched the original run.

The post makes a narrower claim than “better models.” In the DCLM test, the single fixed data stream Zephon used happened to produce a slightly weaker model than the varied streams from the standard loader. DatologyAI says this was chance: “There is no trade-off between determinism and model performance, just less variance due to a fixed data stream.”

Why existing loaders fall short

Elastic determinism is not a new idea. Mosaic Streaming coined the term, and both Streaming and Grain provide it for indexable pipelines. DatologyAI says it relied on Mosaic Streaming before building Zephon. The gap is in stateful pipelines. When documents are split into subsequences, packed into bins and reordered to hit a target mixture, there is no sample index that says which record trains at step 1,234. Megatron Energon supports online packing with reproducible scaling, but only with a fixed total number of workers.

Lanes, accumulators and flush sentinels

Zephon combines three mechanisms. First, every operator is split into an accumulator, which sees records in order and makes every stateful decision, and a stateless transform that does the heavy work across any number of workers. A reorder buffer restores the original sequence, so adding workers changes speed but not output. Second, data is divided into a fixed number of logical streams called lanes rather than among GPUs. Changing the GPU count just redistributes lanes, as long as they divide evenly; the post suggests 32 lanes to cover 8, 16 and 32 GPUs. Third, Zephon does not checkpoint operator state. It periodically sends a flush sentinel through the pipeline, by default every eight chunks per lane, and on resume replays input from the last flush. The company says this costs about 0.02 percentage points of tokens in padding.

Pipelines are written with a builder-style Python API. They read Parquet, LitData, MDS, Vortex and JSONL, stream from S3, Google Cloud Storage or Azure, and run on process or thread runners. The output is a plain Python iterable, and the release includes example integrations for Torchtitan and Megatron-LM.

Throughput without giving up the guarantee

Online processing only works if the loader keeps GPUs fed. On a fetch-and-batch pipeline using vision-language data, DatologyAI measured Zephon at 16,000 samples per second, against 5,800 for Grain. LitData and Streaming never passed roughly 2,000. On a full vision-language pipeline with decoding, transforms, tokenization and packing, Zephon and Grain were effectively tied at 32-way parallelism. Both cleared the 16,900 usable tokens per second per GPU that the test model needed. None of the baselines offers elastic determinism for that stateful pipeline, the company notes.

The project is early, with 48 GitHub stars at the time of writing. Dynamic curricula that change mixture weights mid-run while keeping determinism are in progress, and a Ray-based execution engine is being prototyped for heavy audio and video preprocessing.