Genesis 1B, Run 2: 2× Throughput, Same Hardware

Architecture Comparison

Why Deeper, Not Wider

Run 1 was wide: 20 layers at dim 2048. Run 2 trades width for depth: 32 layers at dim 1536. Same parameter budget, fundamentally different compute graph.

More layers means more sequential transformations, more chances for the model to build compositional representations. For reference, Llama 3.2 1B uses only 16 layers. Genesis 1B, Run 2 has 32. Twice the depth at the same parameter count is a bet on reasoning over memorization.

The narrower hidden dimension (1536 vs 2048) also plays better with torch.compile: smaller per-layer tensors mean less memory pressure and better kernel fusion.

The Free 2× Speedup

Two changes account for nearly all of the throughput gain:

• torch.compile: Fuses operations, eliminates Python overhead, generates optimized CUDA kernels. This alone was a ~40% speedup with zero code changes to the model.
• Batch size 1 → 4: Activation checkpointing freed enough VRAM to quadruple the batch. Combined with torch.compile and real-valued RoPE (avoiding complex64 graph breaks), throughput jumped to ~19k tok/s.

Same hardware. Same parameter count. ~3× throughput. No tricks, just using PyTorch properly.

LR Schedule Fix

Run 1 had a bug: pure cosine decay from step 0. No linear warmup. The learning rate started high and the first few hundred steps were essentially random noise.

Run 2 uses proper linear warmup over 1,000 steps followed by cosine decay to 10% of peak LR. Standard practice, but it was missing before.

Checkpoint Infrastructure

Run 2 introduces DCP checkpoint versioning with full architecture metadata embedded in every checkpoint. Each save includes the complete model config (layers, dimensions, head counts, LR schedule parameters) so any checkpoint is self-describing.

Auto rotation keeps the last 5 checkpoints and prunes older ones. Try the latest checkpoint in the live playground on HuggingFace.

What's Next

Training is running now. At ~19k tok/s, we'll hit 20k steps in roughly 6 days. The loss curve will tell us whether the deeper architecture was the right call. Follow progress on the training progress blog or try the live playground.