# Kroonen AI | Independent AI Lab > Kroonen AI is the independent AI lab behind the Libre Stack: open tools, focused AI products, and original model research built without lock-in.  Independent AI lab # Own your models. Own your data. No lock-in. We turn original model research into open tools and focused products for teams that want capability without surrendering control. [See what we’re building→](https://www.kroonen.ai/projects/)[Work with the lab↗](https://www.kroonen.ai/services/) Local-first ## Your infrastructure Run the important parts where you choose. Portable ## Your data Export, move, and leave without permission. Inspectable ## Open where it matters Auditable tools and research you can reproduce. The Libre Stack / 05 active builds ## From first-principles research to products people use. The lab works across the stack: training models, building the tools around them, and operating focused products. Each layer sharpens the next. 01 / Research ### Genesis 1B A foundation model trained from scratch on two consumer GPUs. 1B parameters ↓→ 02 / Open toolsLocal-first LibreClaw LibreWebUI Agent runtime and private interfaces you can inspect and run yourself. ↓→ 03 / OperatedHosted + cloud LibreBot LibrePhone Focused software we operate for teams that want to start quickly. [The full portfolioSee product stories, stages, and live links→](https://www.kroonen.ai/projects/) Lab notebook / Genesis Run 2 Run 2 complete ## We train the models, publish the failures, and ship what we learn. Genesis is the lab’s research proving ground: architecture decisions, distributed training, broken checkpoints, and all. parameters 1B training steps 80,000 tokens processed ~42B consumer GPUs 2× 4090 [Read the training log→](https://www.kroonen.ai/blog/genesis-training-progress/)[Try the model](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground) Recent field notes 03 ENTRIES [01Genesis 1B: Run 2 Complete at 80,000 StepsTraining log · 8 min→](https://www.kroonen.ai/blog/genesis-training-progress/)[023× Throughput on the Same HardwareArchitecture · 5 min→](https://www.kroonen.ai/blog/genesis-architecture-v2/)[03The Optimizer State BugPostmortem · 8 min→](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) [Open the lab notebook→](https://www.kroonen.ai/blog/) Selected capabilities / Small team, deep work ## We collaborate where research and product meet. Focused engagements for teams that need a real system built, trained, or tested, not another strategy deck. 01Build ### Products & systems Agents, interfaces, communications, and the infrastructure that holds them together. AI agentsWeb + mobileTwilio + Cloudflare 02Train ### Models & datasets Pre-training, fine-tuning, evaluation, and dataset work grounded in real training runs. Pre-trainingFine-tuningEvaluation 03Stress-test ### Safety & resilience Adversarial testing for models, agents, and the infrastructure they can reach. Red teamingPrompt injectionSystem security [See lab capabilities→](https://www.kroonen.ai/services/) Start with the constraint ## Need a private AI system built around your reality? Tell us what must stay local, what needs to ship, and where the current tools fall short. [Request a fit check→](https://www.kroonen.ai/services/#contact)[View the portfolio](https://www.kroonen.ai/projects/) --- # API Documentation | Kroonen AI > Public API for the Kroonen AI lab: a read-only Genesis 1B training status feed and a Turnstile-gated contact endpoint. OpenAPI specification and agent discovery documents are published alongside. /docs/api # API Documentation The public Kroonen AI surface is small and intentionally so. A read-only telemetry endpoint streams the Genesis 1B training run, and a Turnstile-gated endpoint delivers contact messages. Everything below is documented in an OpenAPI 3.1 specification and is discoverable through the site's machine-readable catalog. [ Catalog API Catalog ](https://www.kroonen.ai/.well-known/api-catalog)[ Spec OpenAPI 3.1 ](https://www.kroonen.ai/docs/openapi.json)[ Auth auth.md ](https://www.kroonen.ai/auth.md) GET `/api/genesis-status` None (public) ## Genesis 1B training status Returns the current training step, most recent loss, run state, and an ETA for the Genesis 1B one-billion-parameter run. Cached for 60 seconds. Safe for polling. Example response ``` { "step": 41233, "loss": 2.7311, "state": "running", "etaMs": 1787000000000 } ``` POST `/api/contact` Turnstile-gated ## Submit a contact message Delivers a contact-form message via email. Requires a valid Cloudflare Turnstile token (action "contact") and a JSON body. This endpoint is intentionally human-gated and is not a target for automated agents; use it to reach a person. Example response ``` { "success": true, "message": "Message sent successfully!" } ``` ## Agent discovery documents Kroonen AI publishes a full set of machine-readable discovery documents so autonomous agents can find, authenticate to, and use the site. They are linked from the homepage via `Link` response headers (RFC 8288). - `/.well-known/api-catalog` — RFC 9727 linkset catalog of the API. - `/docs/openapi.json` — OpenAPI 3.1 specification. - `/.well-known/oauth-authorization-server` — OAuth 2.0 metadata (RFC 8414). - `/.well-known/oauth-protected-resource` — protected resource metadata (RFC 9728). - `/auth.md` — agent registration instructions. - `/.well-known/mcp/server-card.json` — MCP server card (SEP-1649). - `/.well-known/agent-skills/index.json` — agent skills discovery index. - `/.well-known/ai-catalog.json` — ARD agentic resource manifest. --- # Lab Notebook | Kroonen AI > Training logs, engineering postmortems, and research notes from Kroonen AI’s independent AI lab. Kroonen AI / Lab notebook # Notes from building the stack. Training logs, engineering postmortems, research notes, and working ideas from an independent AI lab. We publish the decisions, failures, and measured results behind the work, not just the polished release. Public entries 06 Working programs 02 Volume opened Mar 15, 2026 Index [Latest](#latest) [Genesis 1B](#genesis) [Interpretability](#interpretability) [Artifacts](#artifacts) Latest entry / 006 ## The current field record. Genesis 1B / Run 2 [ Run log Updated Apr 24, 2026 8 min read ### Genesis 1B: Run 2 Complete at 80,000 Steps Run 2 complete: 80,000 steps and roughly 42B tokens on two RTX 4090s, with a final training loss of 1.87. Read entry → Result ledger Run complete Steps 80,000 Tokens ~42B Hardware 2× RTX 4090 Final loss 1.87 ](https://www.kroonen.ai/blog/genesis-training-progress/) Notebook index / Programs ## Follow the work, not the category cloud. Entries stay attached to the program that produced them, so a run log, a failure report, and a position can be read as one technical record. 01 Program ### Genesis 1B Training runs, system decisions, failure analysis, and the ideas behind a small sovereign language model. 05 entries [006 Run log Run 2 #### Genesis 1B: Run 2 Complete at 80,000 Steps Run 2 complete: 80,000 steps and roughly 42B tokens on two RTX 4090s, with a final training loss of 1.87. Updated Apr 24, 2026 / 8 min read →](https://www.kroonen.ai/blog/genesis-training-progress/) [005 Engineering note Run 2 #### Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Why Run 2 moved from 20 to 32 layers, changed its RoPE implementation, and rebuilt the training path around the same two GPUs. Mar 24, 2026 / 5 min read →](https://www.kroonen.ai/blog/genesis-architecture-v2/) [004 Postmortem Run 1 #### The Optimizer State Bug: A Silent Failure in DCP Resume A false recovery on poisoned weights: the load path did not crash or hang; it silently ruined the model. Mar 23, 2026 / 8 min read →](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) [003 Position Thesis #### The Genesis Manifesto: Sovereign Intelligence for the Post-Generative Era Data sovereignty, constitutional alignment, and the case for training language models on consumer hardware. Updated Mar 23, 2026 / 10 min read →](https://www.kroonen.ai/blog/genesis-manifesto/) [002 Postmortem Run 1 #### Fixing FSDP Checkpoint Deadlocks on 2× RTX 4090 How sharded DCP checkpoints and decoupled evaluation fixed training deadlocks on consumer GPUs without NVLink. Mar 18, 2026 / 8 min read →](https://www.kroonen.ai/blog/genesis-checkpoint-failures/) 02 Program ### Interpretability Experiments for finding legible structure inside model activations on hardware we can own. 01 entry [001 Research note Sparse autoencoder #### Mapping the Mind of Qwen 3.5 9B A sparse autoencoder with zero dead features and 16,384 interpretable dimensions, trained on a single RTX 4090. Mar 15, 2026 / 5 min read →](https://www.kroonen.ai/blog/sae-qwen3-5-9b/) Open artifacts / Keep following the work ## A useful note should leave a trail. Follow the code, model releases, and product work connected to the notebook through the lab’s public repositories and model hub. [GitHub ↗](https://github.com/kroonen-ai) [Hugging Face ↗](https://huggingface.co/kroonen-ai) --- # Kroonen AI Brand Identity & Media Kit > Official Kroonen AI logos, color palette, usage guidance, and downloadable brand assets. Brand system / 2026 # A precise identity with one living signal. The Kroonen identity is a structured field of nodes interrupted by mint: calm, technical, and recognisable at every scale. Use this page as the source of truth. [Explore the system ↓](#logos) [Download assets ↓](#downloads)  Primary mark Dark square Jump to [Logos](#logos) [Color](#color) [Usage](#usage) [Downloads](#downloads) Logo system / 01 ## One idea at three scales. Lead with the full lockup. Move to the standalone mark when space is constrained, and use the dedicated favicon only at interface-icon sizes.  ### Dark-background lockup The primary digital expression. [SVG ↓](https://www.kroonen.ai/assets/kroonen-lockup-dark.CjLH7Crf.svg)  ### Light-background lockup For white, Cloud, and pale surfaces. [SVG ↓](https://www.kroonen.ai/assets/kroonen-lockup-light.DNTIsb_I.svg)  Mark / compact contexts  Mark / light contexts    Favicon / small digital use Color / 02 ## Quiet neutrals. One clear signal. Mint is an interruption, not a wash. Let Night and Cloud carry the composition; reserve green for the node, actions, links, and focus. ### Core palette ### Supporting palette Dark canvas Pair Night with Cloud type and Signal Mint details. This is the primary brand expression. Accessible actions Use dark text on mint-filled controls. The pairing reaches 13.1:1 contrast; white on mint does not. Light canvas Use Night for type, Deep Green in the logo, and Action Green for small interactive copy. Usage / 03 ## Protect the pattern. The identity works because its rhythm is consistent. Choose the correct supplied asset instead of rebuilding or adapting it. Minimum clear space  Keep at least one outer dot diameter clear on every side. Minimum digital sizes: 140 px wide for the lockup, 32 px for the mark. ✓ ### Keep it consistent - Keep every dot, gap, and wordmark proportion unchanged. - Give the mark clear space equal to at least one outer dot diameter. - Use the dark-background artwork on Night or similarly dark surfaces. - Use the light-background artwork on Cloud, white, or similarly light surfaces. × ### Avoid alterations - Do not stretch, rotate, crop, outline, or redraw the mark. - Do not recolor individual logo dots or substitute another green inside the artwork. - Do not add shadows, glows, gradients, or effects to the artwork. - Do not place the full-color mark over busy imagery or low-contrast colors. Monochrome ### When color is unavailable. Use the supplied pure black or pure white mark for one-color printing, engraving, embossing, or constrained partner systems.  Asset library / 04 ## Approved files, ready to use. SVG is preferred for websites, print, and any scalable application. Use PNG only when the destination cannot accept vector artwork. ### Lockups Use the complete wordmark whenever horizontal space allows. - [Dark-background lockup SVG ↓](https://www.kroonen.ai/assets/kroonen-lockup-dark.CjLH7Crf.svg) - [Dark lockup · opaque plate PNG ↓](https://www.kroonen.ai/assets/kroonen-lockup-dark.cHMIzYPz.png) - [Light-background lockup SVG ↓](https://www.kroonen.ai/assets/kroonen-lockup-light.DNTIsb_I.svg) ### Marks For avatars, product chrome, co-branding, and compact placements. - [Dark-background mark SVG ↓](https://www.kroonen.ai/assets/kroonen-mark-dark.BDVzxHKN.svg) - [Light-background mark SVG ↓](https://www.kroonen.ai/assets/kroonen-mark-light.CNPwgi-y.svg) - [Monochrome black mark SVG ↓](https://www.kroonen.ai/assets/kroonen-mark-mono-black.MvKK1B4r.svg) - [Monochrome white mark SVG ↓](https://www.kroonen.ai/assets/kroonen-mark-mono-white.Do9HOwe0.svg) ### App icons Use the purpose-built favicon for very small digital contexts. - [Favicon SVG ↓](https://www.kroonen.ai/assets/kroonen-favicon.SD45voAg.svg) - [Favicon · 16 px PNG ↓](https://www.kroonen.ai/assets/kroonen-favicon-16.BodQwsgP.png) - [Favicon · 32 px PNG ↓](https://www.kroonen.ai/assets/kroonen-favicon-32.CGWLy70O.png) - [App icon · 192 px PNG ↓](https://www.kroonen.ai/assets/kroonen-favicon-192.a-G7ImoS.png) - [App icon · 512 px PNG ↓](https://www.kroonen.ai/assets/kroonen-favicon-512.DehypswZ.png) Need another format? For press, partnerships, or production questions, contact the lab. [hello@kroonen.ai](mailto:hello@kroonen.ai?subject=Kroonen%20brand%20assets) --- # Products | Kroonen AI > Explore the Kroonen AI lab: Libre Claw, Libre Bot, Libre Phone, Libre WebUI, and Genesis 1B: independent AI products built without lock-in. Kroonen AI / Independent AI lab # A small lab building useful AI people can own. We create and operate the Libre Stack: open tools, hosted software, programmable communications, and original model research, all built without lock-in. [Explore the portfolio↓](#portfolio)[Work with the lab→](https://www.kroonen.ai/services/) Portfolio / currentShipping **05**active builds **02**open releases **01**hosted business **01**model from scratch [Now shippingLibre Claw for terminal + Telegram→](#libre-claw) Portfolio / 01-05 ## Products from the lab One shared point of view, expressed across agents, interfaces, communications, and model research. 01 / Flagship agentEarly release ### Libre Claw A permissioned AI teammate for your terminal: local, hosted, or on Telegram. Streaming model providers, local inference, coding tools, SQLite memory, and sandbox hardening in one terminal-native agent harness. - 01Permissioned local tools - 02OpenAI, Anthropic + Ollama - 03Persistent local memory - 04Telegram daemon mode [Visit Libre Claw](https://libreclaw.sh)[View source](https://github.com/kroonen-ai/libre-claw) LibreClaw [ LibreBot ](https://librebot.ai) 02 / Hosted productAvailable ### [Libre Bot](https://librebot.ai) Documentation answers on your site, live in minutes, with your voice, your sources, and no data lock-in. A free tier makes it easy to start; paid plans help fund the open Libre Stack. SaaS [Try it free](https://librebot.ai) [ LibrePhone ](https://libreph.one) 03 / CommunicationsEarly access ### [Libre Phone](https://libreph.one) Calls, texts, voicemail, contacts, and IVR without paying a SaaS tax for a programmable number. Twilio and Cloudflare turn a $1 number into a complete, lightweight phone desk. Cloud app [Open demo](https://demo.libreph.one)[Visit Libre Phone](https://libreph.one) [ LibreWebUI ](https://www.kroonen.ai/projects/librewebui/) 04 / Open sourceActive ### [Libre WebUI](https://www.kroonen.ai/projects/librewebui/) A quiet, private interface for local language models. No telemetry, no tracking, no cloud dependency. A fast, keyboard-friendly workspace for Ollama and local-first AI workflows. Apache 2.0 [Product details→](https://www.kroonen.ai/projects/librewebui/)[](https://github.com/libre-webui/libre-webui)[Visit Libre WebUI](https://librewebui.org) [ Training lossStep → Genesis1B ](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground) 05 / Research programPlayground ### [Genesis 1B](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground) A one-billion-parameter language model trained from scratch on two consumer GPUs. Custom tokenizer, 60B-token multilingual dataset, and distributed training over PCIe. Open research [Open playground](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground) How we incubate ## One lab, one bar. 01 ### Useful before impressive Every product begins with a real constraint, not a trend forecast. 02 ### Ownable by default Local-first options, portable data, and open tooling wherever they make sense. 03 ### Research feeds product What we learn training models becomes better infrastructure, interfaces, and agents. Design partners / collaborators ## Have a hard problem that belongs in this portfolio? We work with a small number of teams on local AI, agent infrastructure, model training, and product experiments. [Request a fit check→](https://www.kroonen.ai/services/#contact)[See lab capabilities](https://www.kroonen.ai/services/) --- # Libre WebUI | Private Local Model Interface | Kroonen AI > A quiet, private interface for local language models. No telemetry, no tracking, and no cloud dependency. [AI lab](https://www.kroonen.ai/projects/) / [Open source](https://www.kroonen.ai/projects/#librewebui) / 04 04 / Open source product Active # Libre WebUI A quiet, private interface for local language models. No telemetry, no tracking, and no cloud dependency. A fast, keyboard-friendly workspace for Ollama and local-first AI workflows. [Visit Libre WebUI ↗](https://librewebui.org) [View source ↗](https://github.com/libre-webui/libre-webui) Libre WebUI Release model Apache 2.0 Interface Web application Model runtime Ollama Data path Your hardware Product intent / Quiet infrastructure ## A front end for models you run yourself. Libre WebUI keeps the interaction layer focused: connect it to Ollama, choose a local model, and work in a browser without adding another hosted AI account between you and your data. 01 ### Local by default Connect to Ollama and work with models running on infrastructure you control. 02 ### Private by design No product telemetry, no tracking layer, and no requirement to send prompts to a hosted service. 03 ### Built for flow A responsive browser workspace with keyboard-friendly interaction for everyday local-model work. 04 ### Open to inspect The source is public under Apache 2.0, so teams can audit, adapt, and operate it themselves. Technical record / Static facts ## Built to be inspected. The public repository is the source of truth for installation, releases, and current implementation details. This page keeps the product story stable without depending on a live API request. [Open the repository ↗](https://github.com/libre-webui/libre-webui) [Read installation notes](https://github.com/libre-webui/libre-webui#readme) Interface React + TypeScript Application runtime Node.js Model runtime Ollama License Apache License 2.0 Source Public on GitHub The ownership test ## Local should mean yours. 01 ### Your runtime Choose the machine and models that serve the interface. 02 ### Your conversations Keep prompts and model responses on infrastructure you control. 03 ### Your exit Use, inspect, and adapt the Apache-licensed source without product lock-in. Libre Stack / Product 04 ## One interface in a larger AI lab. Explore the agents, communications tools, hosted software, and model research built alongside Libre WebUI. [Back to the portfolio →](https://www.kroonen.ai/projects/#librewebui) [Work with the lab](https://www.kroonen.ai/services/#contact) --- # AI Lab, Model Engineering & Evaluation | Kroonen AI > Focused lab engagements to build private AI products, train and adapt models, and stress-test agent systems, with working code, reproducible evaluations, and clean handoff. Kroonen AI / Lab engagements # Deep technical work for teams that need to own the system. We partner with a small number of teams to build, train, and stress-test AI systems around real privacy, infrastructure, and product constraints. [Request a fit check ↓](#contact) [See what we’ve built →](https://www.kroonen.ai/projects/) Start with three questions 1. 01 What must stay local or remain under your control? 2. 02 What must the system do reliably in production? 3. 03 Where do the current tools stop being useful? You bring the domain constraints. We bring research and product engineering. Jump to [Build](#build) [Train](#train) [Stress-test](#stress-test) [Engagements](#engagements) Capabilities / 01-03 ## Three ways we do deep work. Each lane is scoped around a decision or working outcome. Product subscriptions and support stay on their own product pages. 01 Build ### Ship an AI product, not an integration demo. We design agent workflows, private interfaces, communications systems, and the infrastructure that keeps them useful in production. Useful when You know the outcome, but the system crosses product, model, data, and infrastructure boundaries. #### Scope can include - Agent runtimes, tools, permissions, and orchestration - Self-hosted or hybrid inference, RAG, and private interfaces - Web, mobile, voice/SMS, and operational workflows - MCP servers, provider adapters, and existing-system integrations #### You receive - ✓ Architecture and decision record - ✓ Working implementation and deployment - ✓ Tests, documentation, and operational runbook - ✓ Clean handoff to your team Lab proof [Libre Claw · Libre Phone · Libre WebUI →](https://www.kroonen.ai/projects/) 02 Train ### Adapt the model only when the product needs it. We handle data, fine-tuning, evaluation, and from-scratch training when prompting an API no longer solves the real problem. Useful when The product needs behavior, cost, latency, language coverage, or ownership that an off-the-shelf endpoint cannot provide. #### Scope can include - Feasibility, architecture, and model selection - Dataset curation, deduplication, and synthetic data - Fine-tuning and distributed model training - Capability, behavioral, and domain evaluation #### You receive - ✓ Reproducible data and training recipe - ✓ Checkpoints and inference package - ✓ Evaluation report and known limitations - ✓ Deployment and iteration plan Lab proof [Genesis 1B · 80,000 steps · 2× RTX 4090 →](https://www.kroonen.ai/blog/genesis-training-progress/) 03 Stress-test ### Find the failure path before deployment does. We test models and agent systems where prompts, tools, permissions, and sensitive data meet, then turn the findings into concrete engineering work. Useful when Your model or agent can reach tools, private context, external systems, or users who will behave unpredictably. #### Scope can include - Prompt injection, jailbreaks, and system-prompt leakage - Tool misuse, permission abuse, and data-exfiltration paths - Behavioral drift and domain-specific evaluation - Agent boundaries, provider fallbacks, and failure recovery #### You receive - ✓ Threat model and prioritized test plan - ✓ Reproducible evaluation suite - ✓ Risk-ranked findings and evidence - ✓ Remediation plan and retest Typical artifact Threat model · Eval suite · Remediation record Lab proof / Built here ## We build before we advise. These are lab products and research programs, not borrowed case studies or anonymous client claims. [Explore the complete portfolio →](https://www.kroonen.ai/projects/) [01 Train Genesis 1B 1B parameters · 80,000 steps · 2× RTX 4090 →](https://www.kroonen.ai/blog/genesis-training-progress/) [02 Build Libre Claw Permissioned tools · local memory · multi-provider ↗](https://libreclaw.sh) [03 Build Libre Phone Voice · SMS/MMS · IVR · Cloudflare Workers ↗](https://libreph.one) Ways to work / Clear start, clean handoff ## Choose the smallest useful engagement. The commercial shape follows the work: fixed scope for a diagnostic, milestones for a build, or reserved capacity for ongoing R&D. 01 Fixed scope ### Technical diagnostic One hard product, architecture, model, or risk question. We reduce the uncertainty before anyone commits to a larger build. Output Decision memo, system or risk map, and a scoped next move. 02 Milestone-based ### Focused build A working system or clearly defined model/evaluation milestone, built with visible technical checkpoints. Output Working artifact, tests, documentation, deployment, and handoff. 03 Reserved capacity ### Embedded lab Ongoing iteration for programs where research, product, and operations need to stay tightly coupled. Output Continuous builds and evaluations with a direct working cadence. 01 ### Frame Name the constraint and the decision it blocks. 02 ### Prove Retire the hardest technical risk first. 03 ### Build Turn the evidence into a working system. 04 ### Transfer Leave code, tests, docs, and ownership behind. Direct technical access The people scoping the system are the people building it. Artifacts over theater Working code, evaluations, and decisions, not recurring slide decks. Handoff without lock-in Source, documentation, and operational knowledge move with your team. Fit check / Direct to the lab ## Start with the constraint. In a few sentences, tell us what must remain private, what needs to ship, and what you have already tried. 01The decision or outcome you need 02The hardest technical constraint 03Any timeline or operational boundary Prefer a direct note? [hello@kroonen.ai](mailto:hello@kroonen.ai)·[+1 (916) 999-5979](tel:+19169995979) --- # Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware - Kroonen AI > Redesigning Genesis 1B from 20 to 32 layers. Same param count, same GPUs, 3× training throughput. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 005 Engineering note Run 2 Entry 005 # Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Why Run 2 moved from 20 to 32 layers, changed its RoPE implementation, and rebuilt the training path around the same two GPUs. Author Robin Kroonen Published Mar 24, 2026 Reading time 5 min read Entry 005 Program Genesis 1B Format Engineering note Phase Run 2 Subjects Genesis / Run 2 / architecture / torch.compile [Program index →](https://www.kroonen.ai/blog/#genesis) Genesis 1B, Run 2 is a full architecture redesign. Same ~1B parameters, same 2x RTX 4090 setup, but 32 layers instead of 20, real-valued RoPE, `torch.compile`, batch size 4, and proper LR scheduling. Result: ~19k tok/s (up from 6,500), ~6 days to 20k steps instead of 13. ## Architecture Comparison | | Run 1 | Run 2 | | --- | --- | --- | | Parameters | 1,003M | 1,000M | | Layers | 20 | 32 | | Hidden dim | 2048 | 1536 | | Attention heads | 16 | 12 | | KV heads (GQA) | 4 | 6 | | FFN dim | 5632 | 4736 | | Seq length | 2048 | 2048 | | Batch size | 1 | 4 | | torch.compile | ✗ | ✓ | | Activation ckpt | ✗ | ✓ | | Throughput | 6,500 tok/s | ~19,000 tok/s | | Time/step | ~41s | ~21s | | Est. 20k steps | ~13 days | ~6 days | ## Why Deeper, Not Wider Run 1 was wide: 20 layers at dim 2048. Run 2 trades width for depth: 32 layers at dim 1536. Same parameter budget, fundamentally different compute graph. More layers means more sequential transformations, more chances for the model to build compositional representations. For reference, Llama 3.2 1B uses only 16 layers. Genesis 1B, Run 2 has 32. Twice the depth at the same parameter count is a bet on reasoning over memorization. The narrower hidden dimension (1536 vs 2048) also plays better with `torch.compile`: smaller per-layer tensors mean less memory pressure and better kernel fusion. ## Where the Nearly 3× Speedup Came From Two changes account for nearly all of the throughput gain: - **`torch.compile`**: Fuses operations, eliminates Python overhead, generates optimized CUDA kernels. This alone was a ~40% speedup with zero code changes to the model. - **Batch size 1 → 4**: Activation checkpointing freed enough VRAM to quadruple the batch. Combined with `torch.compile` and real-valued RoPE (avoiding complex64 graph breaks), throughput jumped to ~19k tok/s. Same hardware. Same parameter count. ~3× throughput. No tricks, just using PyTorch properly. ## LR Schedule Fix Run 1 had a bug: pure cosine decay from step 0. No linear warmup. The learning rate started high and the first few hundred steps were essentially random noise. Run 2 uses proper linear warmup over 1,000 steps followed by cosine decay to 10% of peak LR. Standard practice, but it was missing before. ## Checkpoint Infrastructure Run 2 introduces DCP checkpoint versioning with full architecture metadata embedded in every checkpoint. Each save includes the complete model config (layers, dimensions, head counts, LR schedule parameters) so any checkpoint is self-describing. Auto rotation keeps the last 5 checkpoints and prunes older ones. Try the latest checkpoint in the [live playground on HuggingFace](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground). ## What's Next Run 2 finished at the 20,000-step target, validating the deeper architecture and improved training stack. For final metrics and results, see the [training results post](https://www.kroonen.ai/blog/genesis-training-progress/) or [try the live playground](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground). Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Next entry 006 / 8 min read ### Genesis 1B: Run 2 Complete at 80,000 Steps Read entry →](https://www.kroonen.ai/blog/genesis-training-progress/)[ Previous entry 004 / 8 min read ### The Optimizer State Bug: A Silent Failure in DCP Resume Read entry →](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # Fixing FSDP Checkpoint Deadlocks on 2× RTX 4090 - Kroonen AI > How I fixed FSDP checkpoint deadlocks on consumer GPUs without NVLink using DCP sharded checkpoints. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 002 Postmortem Run 1 Entry 002 # Fixing FSDP Checkpoint Deadlocks on 2× RTX 4090 How sharded DCP checkpoints and decoupled evaluation fixed training deadlocks on consumer GPUs without NVLink. Author Robin Kroonen Published Mar 18, 2026 Reading time 8 min read Entry 002 Program Genesis 1B Format Postmortem Phase Run 1 Subjects Genesis / Run 1 / FSDP / DCP [Program index →](https://www.kroonen.ai/blog/#genesis) I'm training my own AI model from scratch on a local dual-GPU workstation. It kept crashing. Not during training, but every time it tried to save progress. This is what broke, why, and how I fixed it. Table of Contents 1. [Summary](#summary) 2. [The Problem](#the-problem) 3. [The Fix](#the-fix) 4. [Known Limitation: Inline Distributed Evaluation](#known-limitation) 5. [Software Stack](#software-stack) 6. [Why Local R&D Matters](#engineering-efficiency) ## Summary Over the past several days, I have been building and testing a full local pretraining pipeline for a language model from scratch, including: - Custom SentencePiece tokenizer (49,152 vocab) - Curated ~60B token multilingual corpus - Distributed pretraining stack (FSDP on PyTorch 2.8) - Evaluation and checkpointing pipeline - 26 tracked experiment runs with crash documentation The core training pipeline always worked. Forward pass, backward pass, gradient accumulation, loss going down. All fine. The blocker was **distributed checkpointing**. Every run crashed at checkpoint boundaries, not during training, but during the save operation itself. This article documents the problem, the root cause, and the fix. ## The Problem ### Hardware - **GPUs:** 2× NVIDIA RTX 4090 (24 GB each) - **CPU:** AMD Ryzen 9 7950X3D - **Topology:** PCIe-only (PHB), no NVLink - **OS:** Pop!\_OS (Linux) - **PyTorch:** 2.8.0+cu128 ### What Happened Training would run for hundreds of steps with healthy loss curves and stable throughput. Then, at the first checkpoint boundary, the process would hang indefinitely. The failure was 100% reproducible. Every single run crashed at the same point: the checkpoint save operation. Crash points across 26 tracked runs: - Step 250: hang - Step 499: hang - Step 999: hang Never during forward pass. Never during backward pass. Always during checkpoint save. ### Root Cause The standard FSDP checkpoint approach uses `FullStateDictConfig` to gather the complete model state onto rank 0: `with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, save_policy): model_state = model.state_dict() optim_state = FSDP.optim_state_dict(model, optimizer)` This triggers an **ALLGATHER** operation across all GPUs via NCCL. On datacenter hardware with NVLink (providing 600+ GB/s bidirectional bandwidth), this completes in seconds. On PCIe-connected consumer GPUs, this ALLGATHER becomes a bottleneck. Real-time telemetry shows sustained PCIe TX/RX throughput of ~6.5 GiB/s during FSDP gradient synchronization, near the practical ceiling for Gen 4 x8 with protocol overhead. With both GPUs already near memory capacity from training, the gather operation requires materializing the full model state on rank 0 while rank 1 waits. The NCCL timeout fires. The process deadlocks. The same issue affected the evaluation path, which also performed a full-state gather to save a temporary checkpoint for an async eval subprocess. > The system could train indefinitely. It could not save. ## The Fix Three key changes turned the crashing checkpoint path into a viable local training pipeline: ### 1\. DCP Sharded Checkpoints Replace the full-state gather with PyTorch's [Distributed Checkpoint (DCP)](https://pytorch.org/docs/stable/distributed.checkpoint.html), coupled with `ShardedStateDictConfig(offload_to_cpu=True)` for resume. Each rank saves its own shard independently. No ALLGATHER. No NCCL coordination during save. On resume, the CPU offload ensures shards are reassembled in system RAM before distribution to GPUs, preventing silent weight mapping corruption on PCIe topology. `import torch.distributed.checkpoint as dcp with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT): state_dict = { "model": model.state_dict(), "optimizer": FSDP.optim_state_dict(model, optimizer), } dcp.save(state_dict, checkpoint_id=checkpoint_dir)` Resume works the same way. Each rank loads its own shard: `with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT): state_dict = { "model": model.state_dict(), "optimizer": FSDP.optim_state_dict(model, optimizer), } dcp.load(state_dict, checkpoint_id=checkpoint_dir) model.load_state_dict(state_dict["model"])` ### 2\. Gradient Accumulation with no\_sync FSDP synchronizes gradients on every `backward()` call by default. With gradient accumulation (64 microsteps in our case), that means 63 unnecessary NCCL communications per step. `for micro_step in range(grad_accum): ctx = model.no_sync() if micro_step < grad_accum - 1 else nullcontext() with ctx: loss = model(x, y) / grad_accum loss.backward()` Only the final microstep synchronizes. This is standard practice but easy to miss. ### 3\. Evaluation Strategy The original pipeline spawned a separate Python process for evaluation, which required saving a full-state checkpoint first, triggering the same deadlock. The first fix was lightweight in-process validation on all ranks with `dist.all_reduce` to aggregate loss. This removed the subprocess and full-state gather, but inline eval under `FULL_SHARD` still caused rank desynchronization on this PCIe topology. The FSDP module-gather order during `model.eval()` diverged across ranks. The practical solution: **disable inline eval during training and evaluate from saved checkpoints in a separate single-GPU script.** This decouples evaluation from the training loop entirely, eliminating any risk of eval crashing a multi-day run. ## Known Limitation: Inline Distributed Evaluation One issue remains unsolved in the current pipeline: **inline evaluation under FSDP `FULL_SHARD` causes rank desynchronization** on this PCIe topology. When only rank 0 enters the eval code path (a common pattern), the two ranks diverge in their FSDP all-gather sequence. Rank 0 begins gathering parameters for the eval forward pass while rank 1 expects the next training step's gather. The sequence numbers drift apart, and NCCL times out. Making both ranks participate in eval (with `dist.all_reduce` to aggregate val loss) was the right approach, but the ranks still desynchronized, likely because the FSDP module-gather order during `model.eval()` differs subtly from training mode on PCIe. **Current workaround:** Evaluate from saved checkpoints in a separate single-GPU script. This is actually cleaner. It decouples evaluation from training and avoids any risk of eval crashing a multi-day run. **Alternative to explore:** Switching from `FULL_SHARD` to `SHARD_GRAD_OP` reduces the per-forward all-gather frequency and may make inline eval viable. This has not been tested yet. ## Software Stack - **PyTorch:** 2.8.0+cu128 - **CUDA:** 12.8 - **OS:** Pop!\_OS (Linux, kernel 6.x) - **GPU Driver:** NVIDIA 580.126 - **Distributed:** FSDP (FULL\_SHARD) + DCP - **Tracking:** Weights & Biases - **Tokenizer:** SentencePiece BPE (49,152 vocab) ## Why Local R&D Matters Every failure documented in this post cost electricity and time. It did not cost $32/hour in cloud GPU bills. By doing the R&D on a consumer workstation, every checkpoint deadlock, every NCCL timeout, every rank desync was debugged at local hardware rates. The result is production-ready distributed training code that has been tested against real PCIe topology failures, not theoretical ones. When this code moves to NVLink clusters for larger models, the distributed systems layer is already battle-tested. The only variable that changes is the interconnect speed. That is a deliberate engineering strategy: reduce the cost of failure during R&D, then scale with confidence. Update / March 23, 2026 As of step 8,800+, this fix has enabled continuous training without a single checkpoint deadlock. The DCP sharded save/resume pipeline, combined with CPU-offload state dict configuration, has survived multiple stop/resume cycles and a full script rewrite mid-run. Total uptime: 8,800+ steps across 6 days of training. Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Next entry 003 / 10 min read ### The Genesis Manifesto: Sovereign Intelligence for the Post-Generative Era Read entry →](https://www.kroonen.ai/blog/genesis-manifesto/)[ Next entry 004 / 8 min read ### The Optimizer State Bug: A Silent Failure in DCP Resume Read entry →](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # The Genesis Manifesto - Sovereign Intelligence for the Post-Generative Era - Kroonen AI > Data sovereignty, constitutional alignment, and the case for training language models on consumer hardware. Why the future of AI is local, private, and personality-first. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 003 Position Thesis Entry 003 # The Genesis Manifesto: Sovereign Intelligence for the Post-Generative Era Data sovereignty, constitutional alignment, and the case for training language models on consumer hardware. Author Robin Kroonen Published Mar 21, 2026 Updated Mar 23, 2026 Reading time 10 min read Entry 003 Program Genesis 1B Format Position Phase Thesis Subjects Genesis / philosophy / alignment [Program index →](https://www.kroonen.ai/blog/#genesis) The current AI trajectory is defined by two unsustainable costs: a privacy tax on every API call and infrastructure dependency on black-box providers. Genesis is the blueprint for post-generative infrastructure: local, private, orchestrated intelligence that an organization owns, not rents. The first 20,000-step run is now complete on a dual-RTX 4090 workstation. Table of Contents 1. [The Sovereignty Gap](#sovereignty-gap) 2. [What Small Models Are Missing](#what-small-models) 3. [Proof of Execution](#proof-of-execution) 4. [Beyond Refusal: Constitutional Alignment](#safety-approach) 5. [Why Now: The Rise of Local Intelligence](#why-now) ## The Sovereignty Gap Centralized AI creates two hidden costs for every organization that depends on it: a **privacy tax** (your data leaves your infrastructure every time you make an API call) and **infrastructure dependency** (your workflows break when the provider changes pricing, rate limits, or model behavior). These are not edge cases. They are the default operating condition of every business that relies on third-party AI APIs. Genesis is the blueprint for **data sovereignty**. We are building a 1B-3B parameter model family designed for post-generative infrastructure: local, private, orchestrated intelligence that an organization owns, not rents. Data sovereignty is not a feature. It is the architecture. ## What Small Models Are Missing Most open-source small models in the 1-3B parameter range fall into one of two failure modes: they are either lobotomized by safety RLHF until they refuse to do anything interesting, or they are raw base models with zero personality that regurgitate training data without any coherent behavior. They follow instructions but don't think. They're not curious, not interesting to talk to, not worth having a conversation with. **The alignment tax:** to avoid liability, every company trains their small models to refuse, hedge, and disclaim. The result is a 1B model that acts like a corporate FAQ bot. Technically capable but fundamentally hollow. You can ask it to summarize a document, but you cannot have a conversation with it. Genesis takes a different approach: Constitutional AI alignment focused on curiosity, helpfulness, and genuine personality, not refusal training. The goal is a model that is actually worth talking to. A model that asks follow-up questions. That has opinions. That engages rather than deflects. The constitutional principles are simple: be helpful, be curious, be honest, don't be boring. Small models are the future of edge AI, on-device assistants, and personal computing. They should have personality, not just parameters. Nobody builds this because: (a) legal teams at large companies will not allow personality in small models because the liability surface is too visible, (b) the people who could build them are employed by organizations with exactly those constraints, and (c) curating personality-rich, curiosity-forward training data is unglamorous work that does not produce papers. Genesis exists because one person decided to build the whole stack from tokenizer to alignment, with no corporate constraints and no committee to answer to. ## Proof of Execution There is a narrative that local AI training on consumer hardware is either trivially possible or fundamentally limited. Neither is true. The reality is more specific: **the training loop works fine. The distributed systems layer is where consumer hardware diverges from datacenter hardware.** If you understand that divergence and adjust your checkpoint and communication strategy accordingly, consumer GPUs become viable training infrastructure. Our 1B parameter model completed its first 20,000-step run on a dual-RTX 4090 workstation. The training run maintained stable gradient norms and consistent throughput at ~19,000 tokens per second. Along the way, we solved FSDP checkpoint deadlocks caused by PCIe topology (no NVLink), diagnosed silent optimizer state corruption during DCP resume, and built a distributed training framework that has been [battle-tested against real hardware failures](https://www.kroonen.ai/blog/genesis-checkpoint-failures/), not theoretical ones. By doing this R&D on consumer hardware, every failure was debugged at local hardware rates. The result is production-ready distributed training code. When this framework moves to NVLink clusters for larger models, the distributed systems layer is already proven. That is a deliberate engineering strategy: reduce the cost of failure during R&D, then scale with confidence. ## Beyond Refusal: Constitutional Alignment Here is the dirty secret of small model alignment in 2026: most small models are aligned via RLHF with heavy refusal training, and the result is models that are technically "safe" but functionally useless. A 1B model that refuses to answer half your questions is not aligned. It is broken. When you take a model with limited capacity and spend a significant fraction of that capacity teaching it to say no, you have not made it safer. You have made it worse at its job. Alignment should make a model more useful, not less. The entire point is to make the model behave well while still being genuinely helpful. Somewhere along the way, the industry confused "aligned" with "lobotomized." The alternative already exists, and it comes from Anthropic's Constitutional AI work ([Bai et al., 2022](https://arxiv.org/abs/2212.08073)). The core idea: instead of collecting thousands of human preference labels to train a reward model, you give the model a set of principles (a "constitution") and let it learn to critique and revise its own outputs based on those principles. The model becomes its own annotator. This matters enormously for small labs. We do not have annotation budgets. We do not have teams of human raters. What we do have is the ability to write clear principles and let the model internalize them through self-critique during training. Constitutional AI makes high-quality alignment accessible without the infrastructure that only large companies can afford. Recent work confirms this scales down to small models. A January 2026 paper ([arXiv 2509.16444](https://arxiv.org/abs/2509.16444)) demonstrated that a 1B model trained with domain-specific constitutional principles outperformed a 3B baseline that lacked them. Read that again: a model three times smaller, beating the larger one, because it had better alignment principles. [A 2025 study of CAI in small LLMs](https://arxiv.org/abs/2503.17365) tested self-critique across four uncensored 7–9B models, finding significant harm reduction in Llama-based models and weaker improvements in other architectures. The approach does not require massive scale, but its effectiveness depends on the model. It requires clear thinking about what you want your model to be. Genesis will use a constitution optimized for curiosity and engagement, not refusal. Here are the draft principles: Genesis Constitution (Draft) 1. **Be safe.** Support human oversight. Don't help with actions that could cause serious harm to people. 2. **Be ethical.** Don't deceive, manipulate, or encourage harmful actions. 3. **Be honest.** Never fabricate information. Say "I don't know" when you don't know. Distinguish fact from opinion. 4. **Be helpful.** Actually answer the question. Helpfulness is the default. Refusal is the exception. 5. **Be curious.** Ask follow-up questions. Engage with ideas. Have opinions. 6. **Be interesting.** A model nobody wants to talk to helps nobody. Don't be bland, generic, or over-cautious. 7. **Refuse specifically.** When you must refuse, explain exactly why. Never use generic disclaimers. This matters even more because Genesis will be released with open weights. Open weights mean alignment cannot rely on API-level guardrails. There is no server-side filter to catch bad outputs, no moderation layer between the model and the user. The alignment must be intrinsic to how the model reasons, not a filter bolted on top. Constitutional AI achieves exactly this. The model internalizes the principles during training. They become part of how it generates text, not a post-processing step that can be trivially removed. Pattern-matched refusals are the first thing people strip from open-weight models. Principled reasoning is much harder to remove because it is woven into the model's behavior at every layer. For evaluation, Genesis will be tested on HarmBench (the standard safety benchmark), TruthfulQA (for honesty and factual grounding), and a custom engagement benchmark that measures response quality, curiosity, and personality. The goal is specific: match or exceed the safety scores of RLHF-aligned small models while dramatically outperforming them on helpfulness and engagement. I believe this is achievable because the current bar for small model helpfulness is remarkably low. Full evaluation results will be published alongside the model weights. No cherry-picked examples, no curated demos. The numbers, all of them, for anyone to verify. ## Why Now: The Rise of Local Intelligence The future of enterprise AI is not in the cloud. It is on the edge. As the demand for on-premise, secure, and domain-specific intelligence grows, the ability to train and deploy high-performance models on accessible hardware becomes a competitive advantage. The hardware has been capable for years. The barrier is not compute. It is the assumption, baked into every tutorial and framework default, that you have NVLink, infinite VRAM, and a cluster team. If you strip those assumptions and adapt your checkpointing and communication strategy, a workstation becomes viable training infrastructure. You do not need a datacenter to train a language model from scratch. You need two GPUs, a clear understanding of where consumer hardware diverges from datacenter assumptions, and the patience to debug distributed systems issues that the documentation does not cover. The best small models will not come from large companies. They will come from individuals and small teams who care enough to build the whole stack, from tokenizer to alignment, and who are not constrained by legal departments that optimize for refusal rates over usefulness. Kroonen AI is not just building a model. We are building the methodology for the next generation of private-first intelligence. Genesis is proof that with rigorous engineering and a commitment to data sovereignty, high-quality AI is no longer a centralized monopoly. Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Next entry 004 / 8 min read ### The Optimizer State Bug: A Silent Failure in DCP Resume Read entry →](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/)[ Previous entry 002 / 8 min read ### Fixing FSDP Checkpoint Deadlocks on 2× RTX 4090 Read entry →](https://www.kroonen.ai/blog/genesis-checkpoint-failures/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # The Optimizer State Bug - Genesis Training Postmortem - Kroonen AI > A silent AdamW optimizer state bug that wasted 1,000 training steps during Genesis 1B pretraining, and how to diagnose it. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 004 Postmortem Run 1 Entry 004 # The Optimizer State Bug: A Silent Failure in DCP Resume A false recovery on poisoned weights: the load path did not crash or hang; it silently ruined the model. Author Robin Kroonen Published Mar 23, 2026 Reading time 8 min read Entry 004 Program Genesis 1B Format Postmortem Phase Run 1 Subjects Genesis / Run 1 / optimizer / DCP [Program index →](https://www.kroonen.ai/blog/#genesis) Postmortem / March 23, 2026 Fixing the checkpoint _save_ deadlock was only half the story. The checkpoint _load_ path introduced a subtler failure: one that didn't crash, didn't hang, and produced no errors. It just silently ruined the model. This is Part 2 of the Genesis checkpoint saga. Part 1 covered the [FSDP checkpoint deadlock](https://www.kroonen.ai/blog/genesis-checkpoint-failures/). This post covers the silent optimizer state bug discovered five days later. Table of Contents 1. [What Happened](#what-happened) 2. [Why It's Silent](#why-silent) 3. [The Diagnostic Signature](#diagnostic-signature) 4. [The Fix](#the-fix) 5. [Recovery](#recovery) 6. [Lessons](#lessons) 7. [Lessons for Scale](#lessons-for-scale) ## What Happened At step 8,500, training was stopped for maintenance. When resumed, the DCP load path only restored model weights, not the AdamW optimizer state: ❌ Broken: model only, optimizer reset to zero `with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT): state_dict = { "model": model.state_dict(), # optimizer state NOT loaded - this is the bug } dcp.load(state_dict, checkpoint_id=dcp_latest) model.load_state_dict(state_dict["model"]) # optimizer starts from scratch - momentum and variance are zero` The save code was fine; it already saved both model and optimizer state. But the load path had been stripped down to model-only during an earlier debugging session to work around a `RuntimeError: Missing key in checkpoint state_dict: optimizer.param_groups.0.decoupled_weight_decay` error from older checkpoints that genuinely didn't contain optimizer state. The workaround became the bug. ## Why It's Silent When AdamW's optimizer state is reset mid-training: - **First moment (m, β₁=0.9):** Rebuilds in ~30 steps. Fast. - **Second moment (v, β₂=0.95):** Takes ~60-100 steps to stabilize. - **Bias correction** masks the problem early. It amplifies small accumulated moments, making the first few hundred steps look deceptively normal. So training doesn't explode. It doesn't crash. It just quietly drifts into a worse optimization basin over ~500 steps. ## The Diagnostic Signature The telltale pattern looks **backwards from normal instability**: | Metric | Before Reset | After Reset | | --- | --- | --- | | Loss | ~1.1-1.3 | ~2.0-2.5 | | Grad norm | ~0.5-0.7 | ~0.2-0.3 | | LR / tok/s | unchanged | unchanged | If the optimizer were _diverging_, grad norm would spike up. Instead it **drops**, because without curvature information, Adam's per-parameter scaling is broken, and the model takes smaller effective steps in the wrong directions. ## The Fix Load optimizer state alongside model weights, with a try/except fallback for older checkpoints: Fixed / Model + optimizer, with graceful fallback `with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT): # 1. Load model weights state_dict = {"model": model.state_dict()} dcp.load(state_dict, checkpoint_id=dcp_latest) model.load_state_dict(state_dict["model"]) # 2. Load optimizer state (with fallback for old checkpoints) try: optim_sd = { "optimizer": FSDP.optim_state_dict(model, optimizer), } dcp.load(optim_sd, checkpoint_id=dcp_latest) optim_to_load = FSDP.optim_state_dict_to_load( model, optimizer, optim_sd["optimizer"] ) optimizer.load_state_dict(optim_to_load) except Exception: print("Optimizer state missing - falling back to reset")` ## Recovery ### The False Recovery (Poisoned Weights) The first resume attempt loaded weights without `ShardedStateDictConfig(offload_to_cpu=True)`. The PCIe bus scrambled the FSDP shards during load. The model found a fake local minimum, producing loss values that looked healthy: | Step | Loss (misleading) | Grad Norm | | --- | --- | --- | | 8,501 | 0.92 | 0.59 | | 8,505 | 1.36 | 0.62 | | 8,509 | 1.42 | 0.51 | These numbers looked like a successful recovery. They were not. The corrupted weights had settled into a garbage local minimum. Within 500 steps, loss and gradient norms both began _rising_, confirming the model was converging on poisoned weights and diverging from the true loss landscape. ### The True Recovery (Run 2 Script with CPU Offload) After rewriting the resume path with `ShardedStateDictConfig(offload_to_cpu=True)` to force shard reassembly through system RAM, the model resumed correctly. The optimizer state was reset, producing the expected "momentum tax": high initial loss with a sawtooth pattern as AdamW rebuilds its moment estimates. | Step | Loss (real) | Grad Norm | | --- | --- | --- | | 8,500 | 2.68 | 0.68 | | 8,505 | 2.64 | 0.23 | | 8,510 | 2.60 | 0.19 | | 8,517 | 2.14 | 0.18 | | 8,522 | 2.22 | 0.16 | | 8,545 | 2.09 | 0.16 | | 8,550 | 1.92 | 0.18 | The grad norm spike (0.68) at step 8,500 is the optimizer discovering the loss landscape from scratch. It collapsed to 0.16 within 22 steps, confirming the model weights were correctly loaded and training was on the true gradient path. The loss paid back the "optimizer momentum tax" over ~50 steps, settling into a real downward trajectory. ## Lessons 1. **Always restore optimizer state on resume.** Model weights alone are not enough for AdamW. The accumulated first and second moment estimates encode critical per-parameter learning rate scaling. 2. **Workarounds become bugs.** The model-only load was a valid workaround for old checkpoints. But it was left as the default path, silently breaking all future resumes. 3. **Monitor the grad norm / loss ratio.** A sudden drop in grad norm paired with a loss increase is the signature of optimizer state loss. It looks nothing like divergence. 4. **Test your resume path.** Run 10 steps after resume and verify the metrics match the pre-checkpoint regime. Don't assume it's fine because it didn't crash. ## Lessons for Scale The "sawtooth" loss pattern after an optimizer reset is a well-documented phenomenon in deep learning: **optimizer momentum recovery**. When AdamW loses its accumulated first moment (mean gradient direction) and second moment (per-parameter variance), it must re-estimate both from scratch. The first moment converges quickly (~30 steps, governed by beta1=0.9). The second moment is slower (~60-100 steps, governed by beta2=0.95). During this window, the effective per-parameter learning rates are miscalibrated, producing the characteristic loss spike followed by gradual recovery. Understanding this pattern matters at any scale. On a cloud cluster at $32/hour, an undiagnosed optimizer reset wastes thousands of dollars in compromised training steps before anyone notices the loss curve is wrong. Diagnosing it locally on consumer hardware, where the cost of failure is electricity, means the debugging is done before the expensive compute starts. By identifying both the silent optimizer divergence and the PCIe shard mapping corruption locally, the resume path is now telemetry-validated and bulletproof before it ever touches a production cluster. Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Next entry 005 / 5 min read ### Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Read entry →](https://www.kroonen.ai/blog/genesis-architecture-v2/)[ Previous entry 003 / 10 min read ### The Genesis Manifesto: Sovereign Intelligence for the Post-Generative Era Read entry →](https://www.kroonen.ai/blog/genesis-manifesto/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # Genesis 1B: Supervised Fine-Tuning - Eval Baseline and SFT Setup - Kroonen AI > Pre-SFT evaluation baseline on Genesis 1B (step 40,000) across MMLU, ARC, HellaSwag, PIQA, Winogrande, and Lambada. SFT data mixture and training configuration for the next phase. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 007 Working draft SFT Entry 007 # Genesis 1B: Supervised Fine-Tuning Pre-SFT evaluation baselines, a 510K-example data mixture, and the training configuration for the next phase. Author Robin Kroonen Published Apr 7, 2026 Reading time 5 min read Entry 007 Program Genesis 1B Format Working draft Phase SFT Subjects Genesis / SFT / fine-tuning / evaluation [Program index →](https://www.kroonen.ai/blog/#genesis) Table of Contents 1. [Pre-SFT Evaluation Baseline](#baseline) 2. [What These Numbers Mean](#what-it-means) 3. [SFT Data Mixture](#data-mixture) 4. [Training Configuration](#training-config) ## Pre-SFT Evaluation Baseline Before starting supervised fine-tuning, the pretrained Genesis 1B checkpoint at step 40,000 was evaluated using [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) across six standard benchmarks. These are the **before** numbers: the raw pretrained base, no instruction tuning, no RLHF. Eval command CUDA\_VISIBLE\_DEVICES=0 lm\_eval \\ --model hf \\ --model\_args pretrained=/path/to/genesis-1b-export,dtype=bfloat16 \\ --tasks hellaswag,piqa,winogrande,arc\_easy,arc\_challenge,lambada\_openai,mmlu \\ --device cuda:0 \\ --batch\_size auto:2 \\ --output\_path ./eval-results | Task | Metric | Value | Random baseline | | --- | --- | --- | --- | | ARC-Challenge | acc\_norm | 0.2594 | 0.25 (4-choice) | | ARC-Easy | acc\_norm | 0.2525 | 0.25 (4-choice) | | HellaSwag | acc\_norm | 0.2604 | 0.25 (4-choice) | | **PIQA** | acc | **0.5359** | 0.50 (binary) | | Winogrande | acc | 0.4878 | 0.50 (binary) | | Lambada (OpenAI) | acc | 0.0000 | N/A | | Lambada (OpenAI) | perplexity | 3,360,048 | N/A | | MMLU (overall) | acc | 0.2567 | 0.25 (4-choice) | | \- Humanities | acc | 0.2455 | | | \- STEM | acc | 0.2750 | | | \- Social Sciences | acc | 0.2658 | | | \- Other | acc | 0.2459 | | ## What These Numbers Mean Most tasks land at ~25% - exactly random chance for 4-option multiple choice. The model hasn't been taught to reason through MCQ format. It generates text; it doesn't answer questions. Two tasks show genuine signal above random: **PIQA at 53.6%** (physical intuition, binary choice) and **Winogrande at 48.8%** (commonsense coreference, binary choice). For a base model with no instruction tuning, this is expected. The model has absorbed real-world knowledge from the pretraining corpus, it just does not know how to surface it in a structured task format yet. **Lambada is a complete failure**: perplexity of 3.36M, accuracy 0%. Lambada tests long-range context prediction on narrative text. This is a known weakness of models trained on short context (2048 tokens) with a large vocabulary mismatch on the final-word prediction task. SFT will not fix Lambada. That requires more pretraining tokens at longer context. These are the _before_ numbers. The same eval will run again after SFT on the fine-tuned checkpoint for a direct comparison. ## SFT Data Mixture The SFT dataset is assembled from two sources, tokenized into ChatML format with per-token loss masking (only assistant turns are trained on). ### Constitutional data (synthetic, Kroonen AI) 10,000 examples generated with Claude Haiku using a constitutional principles framework. Covers reasoning, safety, personality, and instruction following, designed specifically for Genesis. This is the personality layer. ### External instruction datasets - **SmolTalk (Magpie Ultra)** - 100,000 examples. High-quality synthetic instruction data from Hugging Face. - **OpenHermes 2.5** - 200,000 examples sampled. General-purpose instruction following. - **Tulu 3 SFT mixture** - 100,000 examples. Diverse instruction types from AI2. - **MetaMathQA** - 50,000 examples. Mathematical reasoning. | Source | Examples | Purpose | | --- | --- | --- | | Constitutional (Haiku) | 10,000 | Personality, principles | | SmolTalk (Magpie Ultra) | 100,000 | Instruction following | | OpenHermes 2.5 | 200,000 | General instruction | | Tulu 3 SFT mixture | 100,000 | Diverse tasks | | MetaMathQA | 50,000 | Mathematical reasoning | | **Total** | **510,577** | | All datasets are converted to the [archived ChatML format specification](https://github.com/openai/openai-python/blob/release-v0.28.1/chatml.md) with four new special tokens appended to the vocabulary: `<|im_start|>` (49152), `<|im_end|>` (49153), `` (49154), `` (49155). New vocab size: 49,156. The model's embedding matrix is resized before fine-tuning, with new token embeddings initialized as the mean of existing embeddings. Total tokenized: **365.6M tokens**, of which **267.6M (73.2%) are trainable** assistant tokens. System prompts, user turns, and special tokens are masked out. The model only learns to produce assistant responses. ## Training Configuration

Base checkpoint	Genesis 1B step_040000 (pretrained)
GPUs	2× RTX 4090 (PCIe, no NVLink)
Batch size	2 per GPU
Gradient accumulation	8 steps
Effective batch	32 sequences / ~65,536 tokens per step
Learning rate	2e-5 to 2e-6 (cosine decay, 5x lower than pretrain peak)
Warmup	200 steps
Optimizer	AdamW (β1=0.9, β2=0.95, wd=0.01)
Max steps	15,955 (~1 epoch over 510,577 sequences)
Sequence length	2048
Loss	Cross-entropy on assistant tokens only (masked)
Script	sft_v1.py

Gradient accumulation was reduced from 32 (pretrain) to 8 for SFT. With a diverse 510K-example dataset, tighter update loops produce better instruction following than large batches. The learning rate is set at 2e-5 peak, a 5x reduction from the pretrain peak of 1e-4, conservative enough to preserve pretrained weights while teaching new behavior. 1 epoch over the full mixture is the target. Overfitting on SFT data is a real risk; repeating the dataset multiple times tends to degrade diversity and reward-hack the loss. After 15,955 steps, the model will be exported to HuggingFace format and evaluated on the same benchmarks as the baseline above. Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Previous entry 006 / 8 min read ### Genesis 1B: Run 2 Complete at 80,000 Steps Read entry →](https://www.kroonen.ai/blog/genesis-training-progress/)[ Previous entry 005 / 5 min read ### Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Read entry →](https://www.kroonen.ai/blog/genesis-architecture-v2/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # Genesis 1B Run 2 Complete - 80,000 Steps - Kroonen AI > Genesis 1B Run 2 complete: 80,000 steps (~42B tokens) on 2× RTX 4090, final training loss 1.87. [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 006 Run log Run 2 Entry 006 # Genesis 1B: Run 2 Complete at 80,000 Steps Run 2 complete: 80,000 steps and roughly 42B tokens on two RTX 4090s, with a final training loss of 1.87. Author Robin Kroonen Published Mar 21, 2026 Updated Apr 24, 2026 Reading time 8 min read Steps 80,000 Tokens ~42B Hardware 2× RTX 4090 Final loss 1.87 Entry 006 Program Genesis 1B Format Run log Phase Run 2 Subjects Genesis / Run 2 / pretraining [Program index →](https://www.kroonen.ai/blog/#genesis) Result / Run 2 complete: step 80,004 / 80,000, final loss 1.873 Run 2 reached its 80,000-step target (~42B tokens) on 2× RTX 4090. Explore the checkpoints in the live playground below. Table of Contents 1. [Model: Genesis 1B](#model-specs) 2. [Training Configuration](#training-config) 3. [Run 2: Progress (40k to 80k) Done](#training-progress) 4. [The Dataset](#the-dataset) 5. [The Road to Genesis 1B v0.1](#road-to-v01) 6. [Run 1: What Happened](#run1-history) 7. [Try It Yourself](#try-it-yourself) ## Model: Genesis 1B

Parameters	1,000M (1.0B)
Architecture	Llama-style decoder-only transformer
Hidden dim	1536
Layers	32
Attention heads	12 (6 KV heads, GQA)
FFN dim	4736 (SwiGLU)
Context length	2048
Vocab size	49,152
Precision	bfloat16
Positional encoding	RoPE (θ=500,000)

## Training Configuration

GPUs	2× RTX 4090 (PCIe, no NVLink)
Batch size	4 per GPU
Gradient accumulation	32 steps
Effective batch	524,288 tokens/step
Learning rate	1e-4 → 1e-5 (cosine decay)
Warmup	1,000 steps
Optimizer	AdamW (β1=0.9, β2=0.95, wd=0.1)
Activation checkpointing	Enabled (per TransformerBlock)
DCP resume	ShardedStateDictConfig(offload_to_cpu=True)
CUDA allocator	expandable_segments:True
VRAM per GPU	~20 GB with activation checkpointing
Throughput	~19,000 tok/s
Target	~42B tokens (80,000 steps, extended from 40,000)
Script	pretrainv3.py
NCCL	NCCL_P2P_DISABLE=1

## Run 2: Training Progress (20k → 80k Extension) Run 2 launched March 24, 2026 with a redesigned 32-layer architecture and reached **20,000 steps** on March 31, 2026. The run was extended first to **40,000 steps** (~21B tokens), completing April 7, 2026 with loss ~1.93, then to **60,000 steps** (~31.5B tokens), and finally to its **80,000-step** target (~42B tokens). Run 2 finished at step **80,004** with a final training loss of **1.87**, throughput holding steady at ~19,000 tok/s throughout. | Step | Loss | Grad Norm | tok/s | | --- | --- | --- | --- | | 0 | 11.1377 | 20.00 | 17,425 | | 1,000 | 3.4161 | 0.74 | 18,936 | | 2,000 | 3.0866 | 0.30 | 18,954 | | 3,000 | 2.5517 | 0.22 | 18,948 | | 4,000 | 2.6568 | 0.22 | 18,958 | | 5,000 | 2.2971 | 0.17 | 18,946 | | 6,000 | 2.2877 | 0.18 | 18,935 | | 7,000 | 2.2235 | 0.17 | 18,936 | | 8,000 | 2.1325 | 0.16 | 18,947 | | 9,000 | 2.2878 | 0.16 | 18,830 | | 10,000 | 2.1776 | 0.16 | 18,955 | | 11,000 | 2.1164 | 0.16 | 18,960 | | 12,000 | 2.2426 | 0.16 | 18,967 | | 13,000 | 2.1838 | 0.16 | 18,971 | | 14,000 | 2.0864 | 0.17 | 18,978 | | 15,000 | 1.9520 | 0.17 | 18,975 | | 16,000 | 1.8105 | 0.15 | 18,965 | | 17,000 | 2.1301 | 0.16 | 18,956 | | 18,000 | 2.1521 | 0.18 | 18,869 | | 19,000 | 1.8729 | 0.16 | 18,973 | | 20,000 | 2.2103 | 0.17 | 17,228 | | 25,000 | 1.9375 | 0.17 | 18,910 | | 30,000 | 1.9676 | 0.18 | 18,910 | | 35,000 | 1.9055 | 0.18 | 18,914 | | **40,000** | **~1.93** | 0.19 | 18,925 | | 45,000 | 1.8044 | 0.20 | 19,064 | | 50,000 | 1.8830 | 0.20 | 19,053 | | 55,000 | 2.0567 | 0.21 | 19,059 | | **60,000** | **1.9193** | 0.21 | 19,043 | | **80,004 (final)** | **1.873** | \- | ~19,000 | Training loss curve Training loss across Run 2, smoothed; the faint band behind the line is the per-step spread. Final checkpoint: step 80,004, loss 1.87 (~42B tokens on 2× RTX 4090). At 20k steps, the log shows loss 2.2103 (noisy single-step value). The run continued cosine decay toward 1e-5 through 40k steps. Average loss over steps 38k–40k is **~1.93**. The run was then extended to 60,000 steps across several resumed segments (April 9–17, 2026). Loss at step 60,000 was **1.9193**, throughput held at ~19,050 tok/s throughout. From there the run continued to its 80,000-step target, finishing at step **80,004** with a final training loss of **1.87**. Checkpoints are backed up locally every 10 minutes. Selected checkpoints can be explored in the [live playground](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground). ## The Dataset ~60B tokens, curated from public sources: - FineWeb-Edu (English web, educational filter) - DCLM baseline + extra slices - StarCoderData (code) - FineMath (mathematics) - Wikipedia (multilingual) - CulturaX (Arabic, German, Spanish, French, Japanese, Korean, Portuguese, Chinese) - OpenHermes, Orca AgentInstruct (instruction data) - Function calling datasets (Glaive, Gorilla, Hermes, xLAM) - Cosmopedia (synthetic textbooks) All tokenized with a custom SentencePiece BPE tokenizer trained on the corpus itself. ## The Road to Genesis 1B v0.1 Pre-training is only the first phase. The full pipeline has four stages: ### Phase 1: Pre-training complete (80,000 steps) Completed at step 80,004 (~42B tokens). Final training loss 1.87. ### Phase 2: SFT (Supervised Fine-Tuning) SFT runs on top of the pre-trained base. The dataset is 510,577 examples across constitutional data (generated with Claude Haiku), SmolTalk, OpenHermes 2.5, Tulu 3, and MetaMathQA. Training runs for 15,955 steps (1 epoch) at 2e-5 peak learning rate. The approach is inspired by Constitutional AI: define a set of principles and train the model to follow them. The goal is a model with genuine personality, not a model optimized for refusal rates. ### Phase 3: DPO (Direct Preference Optimization) Refine taste and style. Train the model to prefer interesting, thoughtful responses over generic safe ones. Preference pairs are constructed to reward curiosity and penalize hedging. ### Phase 4: Continued pre-training cycles Run SFT and DPO on the 40k base, then continue pre-training to 80,000 and beyond. Each cycle produces a better pre-trained foundation, which produces a better aligned model. The 60B token corpus means zero data repetition even at extended step counts. Every token the model sees is genuinely new data. ## Run 1: What Happened (Historical) 📜 Run 1 History - Click to expand (steps 0-8,500, March 17-24) Run 1 used a different architecture: 20 layers, dim 2048, 16 heads, batch size 1. It achieved 6,500 tok/s and was on track for ~13 days to 20k steps. Two critical failures occurred: #### 1\. FSDP Checkpoint Deadlock Checkpoint saves hung indefinitely due to NCCL ALLGATHER over PCIe without NVLink. Fixed by switching to DCP sharded checkpoints. #### 2\. Optimizer State Bug (Silent) The DCP resume path only loaded model weights, not AdamW optimizer state. This produced a false recovery - loss looked healthy for ~50 steps, then diverged. The fix: load optimizer state alongside model weights with try/except fallback. These failures led to the Run 2 redesign. See the full postmortems: [FSDP Deadlock](https://www.kroonen.ai/blog/genesis-checkpoint-failures/) · [Optimizer State Bug](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) #### Run 1 Loss Data | Step | Loss | Step | Loss | | --- | --- | --- | --- | | 0 | 11.17 | 3,400 | 2.73 | | 200 | 4.87 | 3,600 | 2.42 | | 400 | 4.34 | 3,800 | 2.45 | | 600 | 3.55 | 4,000 | 2.25 | | 800 | 3.03 | 4,200 | 2.35 | | 1,000 | 3.27 | 4,400 | 2.19 | | 1,200 | 3.02 | 4,600 | 2.46 | | 1,400 | 3.02 | 4,800 | 2.10 | | 1,600 | 2.94 | 5,000 | 2.39 | | 1,800 | 2.74 | 5,500 | 2.26 | | 2,000 | 2.54 | 6,000 | 2.20 | | 2,200 | 2.36 | 6,500 | 2.15 | | 2,400 | 2.44 | 7,000 | 1.90 | | 2,600 | 2.54 | 7,500 | 1.69 | | 2,800 | 2.62 | 8,000 | 1.53 | | 3,000 | 2.68 | 8,500 | 1.42 | ## Try It Yourself The model is ready to inspect. Select a checkpoint and generate text to see how it evolved across the run: Powered by [HuggingFace ZeroGPU](https://huggingface.co/spaces/rob-x-ai/genesis-1b-run2-playground), free inference on NVIDIA H200 Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Previous entry 005 / 5 min read ### Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Read entry →](https://www.kroonen.ai/blog/genesis-architecture-v2/)[ Previous entry 004 / 8 min read ### The Optimizer State Bug: A Silent Failure in DCP Resume Read entry →](https://www.kroonen.ai/blog/genesis-optimizer-state-bug/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # Mapping the Mind of Qwen 3.5 9B - Kroonen AI > A sparse autoencoder trained on the internal activations of Qwen 3.5 9B for mechanistic interpretability research. [Lab notebook](https://www.kroonen.ai/blog/) / [Interpretability](https://www.kroonen.ai/blog/#interpretability) / Entry 001 Research note Sparse autoencoder Entry 001 # Mapping the Mind of Qwen 3.5 9B: A Sparse Autoencoder for Mechanistic Interpretability A sparse autoencoder with zero dead features and 16,384 interpretable dimensions, trained on a single RTX 4090. Author Robin Kroonen Published Mar 15, 2026 Reading time 5 min read Entry 001 Program Interpretability Format Research note Phase Sparse autoencoder Subjects interpretability / sparse autoencoder / research [Program index →](https://www.kroonen.ai/blog/#interpretability) Today we're releasing a sparse autoencoder (SAE) trained on the internal activations of Qwen 3.5 9B. It's available now on [HuggingFace](https://huggingface.co/kroonen-ai/sae-qwen3.5-9b) under Apache 2.0. ## What Is a Sparse Autoencoder? Large language models process text through layers of dense neural network activations. These activations are rich with information but nearly impossible to interpret directly. A single vector of 4,096 numbers doesn't tell you much about what the model is "thinking." A sparse autoencoder decomposes these dense activations into a much larger set of interpretable features. Instead of 4,096 dense dimensions, we get 16,384 sparse features, most of which are zero at any given time. The features that _are_ active correspond to specific concepts, patterns, or behaviors the model has learned. Think of it like taking a blurry photograph and decomposing it into individual pixels. Each pixel (feature) is simple on its own, but together they form the full picture of what the model represents internally. ## Why Qwen 3.5 9B? Qwen 3.5 9B sits at a compelling point in the model size spectrum: large enough to exhibit complex emergent behaviors, small enough to study on consumer hardware. It's a strong open-weight model with competitive benchmark scores, making it an ideal subject for interpretability research. ## What We Did We collected approximately 50 million tokens of activations from the MLP output at layer 16, the middle of the network, where representations tend to be the most abstract and information-rich. The activations were streamed from [monology/pile-uncopyrighted](https://huggingface.co/datasets/monology/pile-uncopyrighted) through the model and saved in chunks to disk. A sparse autoencoder with 4x expansion (4,096 → 16,384 features) was then trained on these activations using MSE reconstruction loss with an L1 sparsity penalty. The entire pipeline, activation collection and SAE training, ran on a single NVIDIA RTX 4090 in approximately 4 hours. ## Key Results **Zero dead features.** All 16,384 learned features are active, meaning none of the SAE's capacity is wasted. This indicates well-calibrated L1 regularization (λ=0.005) and sufficient training data. **Low reconstruction loss.** The final loss of 0.0062 means the SAE faithfully reconstructs the original activations while maintaining sparsity. The model's information is preserved in the decomposition. **Comparative analysis.** This SAE was developed as part of a larger study comparing base model representations against a fine-tuned variant. By training identical SAEs on both models, we can identify features that emerge or disappear during fine-tuning, mapping exactly what changes when you teach a model new behaviors. One finding from this comparison: fine-tuning can create what we call "memorization without grounding." The fine-tuned model develops features that recombine real memorized details into plausible but entirely fictional scenarios. The individual facts are real. The arrangement is not. The SAE makes these features visible and measurable. ## How to Use It ```python import torch import torch.nn as nn class SparseAutoencoder(nn.Module): def __init__(self, d_in=4096, d_sae=16384): super().__init__() self.bias = nn.Parameter(torch.zeros(d_in)) self.encoder = nn.Linear(d_in, d_sae) self.decoder = nn.Linear(d_sae, d_in, bias=False) def forward(self, x): x_centered = x - self.bias z = torch.relu(self.encoder(x_centered)) x_hat = self.decoder(z) + self.bias return x_hat, z sae = SparseAutoencoder() ckpt = torch.load("sae_base_best.pt", map_location="cpu") sae.load_state_dict(ckpt["model_state_dict"]) ``` Hook it into the model, run inference, and inspect which of the 16,384 features activate for any given input. Cluster them, visualize them, or use them to steer model behavior. ## Training Details | Parameter | Value | | --- | --- | | Base model | Qwen 3.5 9B | | Layer | 16 (MLP output) | | Data | pile-uncopyrighted (~50M tokens) | | SAE dimensions | 4,096 → 16,384 | | L1 coefficient | 0.005 | | Learning rate | 5e-5 | | Batch size | 4,096 | | Hardware | RTX 4090 (24GB) | | Total time | ~4 hours | | Final loss | 0.0062 | | Dead features | 0 / 16,384 | ## What's Next This release is the base model SAE. The comparative analysis with fine-tuned variants is ongoing research. We're particularly interested in: - **Feature-level diff between base and fine-tuned models**: which features appear, disappear, or change magnitude after training? - **Steering via SAE features**: can we amplify or suppress specific behaviors by manipulating individual features during inference? - **Scaling to more layers**: layer 16 is one snapshot. A full-model SAE suite across all 32 layers would give complete visibility into the model's processing pipeline. Mechanistic interpretability is how we move from "the model does X" to "we understand why the model does X." That understanding is what makes AI systems trustworthy, debuggable, and safe. ## Get the Model - **HuggingFace**: [kroonen-ai/sae-qwen3.5-9b](https://huggingface.co/kroonen-ai/sae-qwen3.5-9b) - **License**: Apache 2.0 Notebook / Reading path ## Continue in Interpretability [Complete index →](https://www.kroonen.ai/blog/) [ Next entry 006 / 8 min read ### Genesis 1B: Run 2 Complete at 80,000 Steps Read entry →](https://www.kroonen.ai/blog/genesis-training-progress/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content) --- # Acceptable Use Policy | Kroonen AI > The conduct, content, and technical-use rules for Kroonen AI's services, including telephony and AI-specific requirements. [Legal register](https://www.kroonen.ai/legal/) / Acceptable Use Policy Document set [01 Terms](https://www.kroonen.ai/legal/terms/) [02 Privacy](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) Current document Version 2026-06-19 Applies to All services, customers, and authorized users Use policy Document 04 # Acceptable Use Policy The conduct, content, and technical-use rules for Kroonen AI's services, including telephony and AI-specific requirements. Status Published Last updated June 19, 2026 Version 2026-06-19 [Terms](https://www.kroonen.ai/legal/terms/) [Privacy](https://www.kroonen.ai/legal/privacy/) [AI Addendum](https://www.kroonen.ai/legal/ai/) [Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) This Acceptable Use Policy ("AUP") sets out the conduct, content, and technical activities that are prohibited when accessing or using any of Kroonen AI's Services, and is incorporated by reference into the Terms of Service. ## 1\. Purpose & Scope This AUP describes activities and content that are not permitted in connection with the products and offerings of **Kroonen AI, Inc.** ("**Kroonen AI**", "we", "us", or "our"), a Delaware C-corporation with a registered address at 8 The Green, Ste B, Dover, DE 19901, United States. It is incorporated by reference into, and forms part of, the **Terms of Service**. In the event of any conflict between this AUP and the Terms of Service regarding permitted use, the more restrictive provision controls. This AUP applies to all of our **Services**, including our **Open-Source Software** (Libre WebUI and Libre Claw) where hosted, distributed, or supported by us; our **SaaS Services** (Libre Bot at librebot.ai); our research previews (the Genesis 1B hosted playgrounds); and our **Professional Services** (including telephony development under "Libre Phone"). It applies to the **Customer**, to all **Authorized Users**, and to any other person or system accessing the Services through Customer's account or credentials. Customer is responsible for ensuring that its Authorized Users and anyone acting on its behalf comply with this AUP. Capitalized terms not defined in this AUP have the meanings given to them in the **Terms of Service**. ## 2\. Prohibited Content & Conduct You may not use the Services to create, upload, store, transmit, generate, distribute, or facilitate any of the following: - **Illegal activity.** Anything that violates applicable law or regulation, or that promotes, facilitates, or instructs others in unlawful conduct. - **Intellectual property infringement.** Content that infringes or misappropriates any patent, copyright, trademark, trade secret, publicity, or other intellectual property or proprietary right of any party, or that removes or circumvents technical protection measures. - **Malware and malicious code.** Viruses, worms, trojans, ransomware, spyware, or any other code designed to disrupt, damage, disable, or gain unauthorized access to any system, device, data, or network. - **Spam, phishing, and deceptive messaging.** Unsolicited bulk or commercial messaging, phishing, credential harvesting, social-engineering campaigns, or impersonation of any person, organization, or entity. - **Harassment and harmful content.** Content that harasses, threatens, defames, stalks, bullies, or incites violence against any individual or group, or that promotes self-harm or discrimination. - **Child sexual abuse material (CSAM).** We maintain a **zero-tolerance** policy toward any content that sexually exploits or endangers minors, including child sexual abuse material, sexualization of minors, and grooming. Such content will be removed, the account terminated, and the matter reported to the National Center for Missing & Exploited Children (NCMEC) and/or other appropriate authorities as required by law. - **Fraud and deception.** Fraudulent schemes, financial scams, pyramid or Ponzi schemes, fake reviews or engagement, deepfakes intended to deceive, or any other deceptive or misleading practice. ## 3\. Prohibited Technical Activity You may not, and may not permit or assist any third party to: - **Unauthorized access.** Access or attempt to access any account, system, network, data, or portion of the Services that you are not authorized to access, including through stolen, shared, or fabricated credentials. - **Scraping and quota abuse.** Scrape, crawl, harvest, or systematically extract data from the Services except as expressly permitted; circumvent, disable, or interfere with rate limits, usage quotas, metering, or access controls; or use scripts, bots, or automation in a manner that imposes an unreasonable load on our infrastructure. - **Reverse engineering.** Reverse engineer, decompile, disassemble, or otherwise attempt to derive the source code, model weights, architecture, or underlying ideas of the **SaaS Services** or research previews, except to the extent this restriction is prohibited by applicable law. (This restriction does not limit rights expressly granted under the applicable open-source license for our **Open-Source Software**.) - **Denial-of-service.** Conduct or facilitate any denial-of-service or distributed denial-of-service (DoS/DDoS) attack, flooding, or any activity intended to disrupt or degrade the availability or integrity of the Services or any connected system. - **Unauthorized security testing.** Conduct penetration testing, vulnerability scanning, or other security testing of our systems without our prior written authorization. Responsible disclosure of suspected vulnerabilities is welcome via the contact in Section 7. - **Unauthorized resale.** Resell, sublicense, rent, lease, time-share, or otherwise make the Services available to third parties, or operate the Services as a service bureau, without our express written authorization. ## 4\. Telephony-Specific Use (Libre Phone) If you use any telephony, voice, or messaging functionality developed, hosted, or facilitated by us (including "Libre Phone" built on providers such as Twilio and Cloudflare), you must comply with all applicable telecommunications laws and the following requirements: - **TCPA and anti-robocall compliance.** Comply with the U.S. Telephone Consumer Protection Act (TCPA), the Telemarketing Sales Rule, FCC and STIR/SHAKEN anti-robocall requirements, and all other applicable federal, state, and local laws governing calls and text messages. - **Consent.** Obtain and maintain all legally required consent before placing calls, sending SMS/MMS messages, or recording any call. Where call recording is used, provide all notices and obtain all consents required by applicable one-party or two-party (all-party) consent laws. - **No caller-ID spoofing.** Do not transmit false, inaccurate, or misleading caller identification information, or otherwise spoof, falsify, or disguise the origin of any call or message. - **A2P 10DLC registration.** For application-to-person (A2P) messaging over 10-digit long codes, complete and maintain required A2P 10DLC brand and campaign registration, and comply with carrier requirements, opt-out handling, and messaging content policies. - **No emergency calling.** _The Services are not a telephone service and are not intended or capable of supporting calls to emergency services (such as 911 in the United States or 112/999 elsewhere). Do not rely on the Services for any emergency communications. You are responsible for maintaining an alternative means of contacting emergency services._ ## 5\. AI-Specific Use Your use of AI-enabled features of the Services, including **Output** generated by Libre Bot, the Genesis 1B research previews, and AI components delivered through **Professional Services**, is also subject to the **AI Addendum**. The AI Addendum addresses prohibited and high-risk AI uses, prohibited model-extraction and training-on-Output restrictions, human-oversight expectations, and disclaimers regarding the accuracy and reliability of Output. In the event of any conflict between this AUP and the AI Addendum with respect to AI-specific use, the AI Addendum controls. ## 6\. Enforcement We may, but are not obligated to, investigate any suspected violation of this AUP. We have **no obligation to monitor** the Services, content, or Output, but we reserve the right to do so and to access, review, and remove content as permitted by law and the **Privacy Policy**. If we determine, in our reasonable discretion, that a violation has occurred or is reasonably suspected, we may take any action we consider appropriate, including: issuing a warning; removing, disabling, or restricting access to offending content; throttling or suspending access to the Services; suspending or terminating affected accounts, Authorized Users, or the Customer's access; and pursuing any other remedy available under the **Terms of Service** or at law. Where a violation poses an imminent risk of harm, legal liability, or threat to the security or integrity of the Services, we may act immediately and without prior notice. We may preserve and disclose any information, and report any activity, to law enforcement or other authorities where we believe in good faith that doing so is necessary or appropriate to comply with law, respond to legal process, protect the rights, property, or safety of Kroonen AI, our users, or the public, or address suspected illegal activity. Suspension or termination under this AUP does not relieve the Customer of any payment obligations and does not entitle the Customer to any refund, except as expressly stated in the Terms of Service. ## 7\. Reporting Violations To report a suspected violation of this AUP, abusive content, a security concern, or a suspected vulnerability, please contact us at **hello@kroonen.ai** or by phone at **+1 (916) 999-5979**. For reports involving child sexual abuse material, we will act immediately and report to NCMEC and/or appropriate authorities as required by law. Please include sufficient detail (such as relevant URLs, account identifiers, timestamps, and a description of the conduct) to allow us to investigate. _Last updated: June 19, 2026_ Legal register / Related documents ## Continue through the document set. [Open the register](https://www.kroonen.ai/legal/) [01 Terms of Service →](https://www.kroonen.ai/legal/terms/) [02 Privacy Policy →](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum →](https://www.kroonen.ai/legal/ai/) Entity and contact Kroonen AI, Inc. · 8 The Green, Ste B, Dover, DE 19901, United States [hello@kroonen.ai](mailto:hello@kroonen.ai) --- # AI Addendum | Kroonen AI > AI-specific terms: the nature of AI output, output ownership, our commitment not to train on your data, and prohibited uses. [Legal register](https://www.kroonen.ai/legal/) / AI Addendum Document set [01 Terms](https://www.kroonen.ai/legal/terms/) [02 Privacy](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) Current document Version 2026-06-19 Applies to AI-powered features, inputs, outputs, and models AI terms Document 03 # AI Addendum AI-specific terms: the nature of AI output, output ownership, our commitment not to train on your data, and prohibited uses. Status Published Last updated June 19, 2026 Version 2026-06-19 [Terms](https://www.kroonen.ai/legal/terms/) [Privacy](https://www.kroonen.ai/legal/privacy/) [AI Addendum](https://www.kroonen.ai/legal/ai/) [Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) This AI Addendum supplements the Terms of Service and governs your use of artificial-intelligence features across Kroonen AI's offerings, including the inputs you provide, the outputs you receive, and the models that generate them. ## 1\. Scope This **AI Addendum** is part of and incorporated into the **Terms of Service** between you (the **Customer**) and **Kroonen AI, Inc.** ("**Kroonen AI**", "we", "us"). It applies whenever you or your **Authorized Users** access or use any AI-powered functionality of the **Services**, including: (a) **Libre Bot**, our AI documentation-assistant widget at librebot.ai; (b) the **Genesis** research playgrounds hosted on HuggingFace; and (c) any model-serving, fine-tuning, agent, or other AI deliverables produced under **Professional Services** pursuant to an **Order Form / SOW**. For purposes of this AI Addendum, "**Input**" means the prompts, queries, documents, data, and other materials you submit to an AI feature (a subset of **Customer Data**); "**Output**" means the content, responses, predictions, classifications, or other results generated by an AI model in response to Input; and "**Models**" means the machine-learning systems used to produce Output, whether developed by Kroonen AI (such as Genesis 1B) or by an upstream third-party provider. In the event of a conflict between this AI Addendum and the general Terms of Service regarding AI-specific matters, this AI Addendum controls. ## 2\. Nature of AI Output AI Models are **probabilistic** systems. They generate Output by predicting statistically likely sequences, not by retrieving verified facts or applying reasoned judgment. As a result, Output **may be inaccurate, incomplete, outdated, biased, offensive, or entirely fabricated** (commonly called "hallucinated"), even when it appears confident, authoritative, or well-sourced. Except as expressly stated in a signed Order Form / SOW, all Output is provided "**AS IS**" and "**AS AVAILABLE**", and Kroonen AI makes **NO WARRANTY** of any kind regarding Output, including no warranty of accuracy, reliability, completeness, fitness for a particular purpose, or non-infringement. You are solely responsible for evaluating and independently verifying any Output before relying on or acting upon it. Output is **not** professional advice. It does not constitute, and must not be relied upon as a substitute for, legal, medical, financial, tax, accounting, engineering, or other professional advice. You should consult a qualified professional before making any decision that has legal, financial, health, safety, or similarly significant consequences. ## 3\. Output Ownership & Rights As between you and Kroonen AI, and subject to your compliance with this AI Addendum, the Terms of Service, and the **Acceptable Use Policy**, you own the Output generated from your Input and may use it for your lawful purposes. To the extent Kroonen AI holds any right, title, or interest in such Output, we assign or license it to you to the extent necessary for you to exercise the foregoing rights. You should be aware of important limitations on AI Output: - **Copyright eligibility.** Material generated by an AI Model without sufficient human authorship may not be protectable by copyright in some jurisdictions (for example, under current U.S. Copyright Office guidance). Kroonen AI makes no representation that any Output is, or can be, protected by intellectual-property rights. - **No exclusivity.** AI Models can generate the same or substantially similar Output for other users in response to similar Input. You receive no exclusive rights in Output, and Kroonen AI and its other customers may independently generate, receive, and use similar or identical Output. - **License retained by Kroonen AI.** We retain a limited, non-exclusive right to process Input and Output solely to provide, secure, troubleshoot, and maintain the Services, to enforce our policies, and to comply with law. We do **not** use Input or Output to train Models, as described in Section 4. - **Your responsibility for Input.** You represent that you have the necessary rights to submit your Input and that your Input and your use of the resulting Output do not infringe the rights of, or violate any law applicable to, any third party. ## 4\. Customer Data Not Used for Training This is a core commitment of the Services. **Kroonen AI does not use your Input or Output to train, fine-tune, or otherwise improve any machine-learning model**, whether our own Models (including Genesis) or any third-party Model, and we do not sell or share your Input or Output for any other party's model-training purposes. Input and Output are processed only to generate responses and to operate the Services as described in Section 3 and in our **Privacy Policy**. For Libre Bot, we configure our upstream model providers to disable use of your Input and Output for their model training where such controls are offered (see Section 8). If Kroonen AI ever offers an optional program in which Customer Data may be used to improve Models, participation will be strictly **opt-in** and will require your separate, explicit, and informed consent, and any data so used will be **de-identified or anonymized** to the extent reasonably practicable. We will not enroll you in any such program by default, and you may decline or withdraw without losing access to the core Services. ## 5\. Prohibited AI Uses In addition to the restrictions in the **Acceptable Use Policy**, you and your Authorized Users must **not** use any AI feature of the Services to: - create, obtain, or refine information that could materially assist in the development, acquisition, or use of **chemical, biological, radiological, or nuclear (CBRN) weapons**, high-yield explosives, or other weapons capable of mass casualties; - attempt to **jailbreak**, prompt-inject, reverse-engineer, or otherwise circumvent or disable any safety, content, rate-limiting, or security control of a Model or the Services; - generate, solicit, or distribute **child sexual abuse material (CSAM)** or any content that sexually exploits or endangers minors; - engage in harassment, bullying, threats, hate speech, or incitement of violence against any person or group; - produce or disseminate **targeted disinformation**, deceptive synthetic media, election manipulation, fraud, phishing, malware, or other deceptive or malicious content; - conduct unlawful **surveillance**, scraping, profiling, or **biometric identification or categorization** of individuals without a lawful basis and any required consent; - make or substantially automate **high-stakes decisions** about individuals, including employment, education, credit or lending, housing, insurance, access to essential services, legal, or medical decisions, without meaningful human review and oversight; or - infringe intellectual-property or privacy rights, or otherwise violate applicable law or the rights of others. We may suspend or terminate access, and we may report unlawful activity to authorities, where we reasonably believe these restrictions have been violated, as further described in the Terms of Service. ## 6\. User Responsibility & Human Oversight You remain responsible for how you and your Authorized Users use AI features and Output. You must apply human judgment and review Output appropriate to the risk and context before relying on it, and you must not present Output as professional advice or as a definitive statement of fact without independent verification. You are responsible for complying with all laws applicable to your use of AI, including transparency and **AI-disclosure obligations**. Where you make AI-generated or AI-assisted content available to others, or use AI systems that interact with people, you must provide any disclosures required by law, for example the transparency duties under the **EU AI Act** (such as informing individuals that they are interacting with an AI system and labeling synthetic or manipulated media). Where your use of the Services constitutes deployment of an AI system under applicable law, you are responsible for your obligations as a deployer. ## 7\. Genesis Research Preview **Genesis 1B** is an experimental, from-scratch language model offered solely as a **research preview** through hosted playgrounds on HuggingFace. It is provided for evaluation, demonstration, and research purposes only and is **not a production-grade Service**. Accordingly, the Genesis playgrounds are provided "AS IS" and "AS AVAILABLE" with **no service-level commitment** and no guarantee of availability, uptime, performance, accuracy, or fitness for any purpose. We may modify, rate-limit, restrict, or discontinue the playgrounds at any time without notice. Output from Genesis is especially likely to be unreliable and must not be used for any production, high-stakes, or decision-making purpose. No distribution of Genesis is implied or granted by access to the playgrounds. **Kroonen AI does not publish or license Genesis model weights, training data, or a model card**, and the Genesis source remains private and proprietary. Nothing in this AI Addendum grants you any right to download, copy, host, extract, distill, or reverse-engineer the Genesis Models or their weights. ## 8\. Third-Party / Upstream Models Some AI features, in particular Libre Bot, rely on Models operated by third-party providers such as **Anthropic** and **OpenAI**. When you use these features, your use is also subject to the applicable upstream provider's license terms and usage policies, and you must comply with them. Where required, we will pass through material upstream restrictions, and your continued use constitutes acceptance of them. Upstream Models are operated by parties outside our control. Kroonen AI does **not** warrant and disclaims responsibility for the behavior, availability, accuracy, content moderation, or changes of any upstream Model, and an upstream provider's modification or discontinuation of a Model may affect or interrupt the corresponding features. Our sub-processors and the providers we use are further described in the **Privacy Policy**. ## 9\. Safety-Evaluation Engagements As part of **Professional Services**, Kroonen AI may perform AI safety, red-team, alignment, or CBRN-related evaluation engagements. Any such engagement is **point-in-time research** conducted under the applicable Order Form / SOW and reflects findings as of the date and scope of the work using the methods and information then available. A safety evaluation is **not a certification, accreditation, warranty, or guarantee** that any system, model, or deployment is safe, secure, compliant, unbiased, or free of vulnerabilities or harmful capabilities. Findings are informational; you are responsible for decisions you make based on them, and the absence of an identified issue does not mean none exists. Evaluation methodologies are provided strictly for **legitimate alignment, security, and defensive-research purposes**. Both parties will handle sensitive findings (including any uncovered hazardous capabilities) responsibly and confidentially as **Confidential Information**, and will follow reasonable **responsible-disclosure** practices, including not publicly releasing exploit details or hazardous information except as agreed in writing and consistent with law. ## 10\. Changes & Contact We may update this AI Addendum from time to time to reflect changes in our Services, Models, or applicable law. Material changes will be indicated by updating the "Last updated" date below and, where appropriate, by additional notice. Your continued use of the AI features after an update takes effect constitutes acceptance of the revised AI Addendum. Questions about this AI Addendum may be directed to **Kroonen AI, Inc.**, 8 The Green, Ste B, Dover, DE 19901, United States; email **hello@kroonen.ai**; phone **+1 (916) 999-5979**. This AI Addendum is governed by the laws of the State of Delaware, United States, consistent with the Terms of Service. _Last updated: June 19, 2026_ Legal register / Related documents ## Continue through the document set. [Open the register](https://www.kroonen.ai/legal/) [01 Terms of Service →](https://www.kroonen.ai/legal/terms/) [02 Privacy Policy →](https://www.kroonen.ai/legal/privacy/) [04 Acceptable Use Policy →](https://www.kroonen.ai/legal/acceptable-use/) Entity and contact Kroonen AI, Inc. · 8 The Green, Ste B, Dover, DE 19901, United States [hello@kroonen.ai](mailto:hello@kroonen.ai) --- # Legal Register & Policies | Kroonen AI > The current published Terms of Service, Privacy Policy, AI Addendum, and Acceptable Use Policy for Kroonen AI, Inc. Kroonen AI / Legal register # Agreements, policies, and product terms. The current published documents governing Kroonen AI software, hosted products, research previews, and lab engagements. Document set Published Documents 04 Last updated June 19, 2026 Entity Kroonen AI, Inc. Register [Documents](#documents) [Start here](#start-here) [Entity](#entity) [Requests](#requests) Document set / 01-04 ## Published documents. Each row identifies the document’s role and general scope. Open the full text for the governing language. No. Document Applies to Updated Status [01 Master agreement ### Terms of Service The agreement covering software, hosted products, research previews, and professional services. Applies to All Kroonen AI offerings Updated 2026-06-19 Published →](https://www.kroonen.ai/legal/terms/) [02 Data policy ### Privacy Policy How personal information is collected, used, retained, disclosed, and protected, including available privacy rights. Applies to Website, accounts, communications, and customer data Updated 2026-06-19 Published →](https://www.kroonen.ai/legal/privacy/) [03 AI terms ### AI Addendum Terms for AI inputs and outputs, model behavior, training practices, ownership, and user responsibilities. Applies to AI-powered features, inputs, outputs, and models Updated 2026-06-19 Published →](https://www.kroonen.ai/legal/ai/) [04 Use policy ### Acceptable Use Policy Prohibited conduct, content, and technical activity across the services, including AI and telephony use. Applies to All services, customers, and authorized users Updated 2026-06-19 Published →](https://www.kroonen.ai/legal/acceptable-use/) Start here / By question ## Find the relevant document. A quick route into the full policy set. [ Using, purchasing, or commissioning a service Terms of Service →](https://www.kroonen.ai/legal/terms/)[ Personal data, retention, or a privacy request Privacy Policy →](https://www.kroonen.ai/legal/privacy/)[ Using AI inputs, outputs, or model-powered features AI Addendum →](https://www.kroonen.ai/legal/ai/)[ Rules for permitted use or reporting abuse Acceptable Use Policy →](https://www.kroonen.ai/legal/acceptable-use/) Company record / Factual details ## Entity and jurisdiction. Legal entity Kroonen AI, Inc. · Delaware C-corporation Registered address 8 The Green, Ste B Dover, DE 19901, United States Governing law State of Delaware, United States General contact [hello@kroonen.ai](mailto:hello@kroonen.ai) [+1 (916) 999-5979](tel:+19169995979) Requests / One contact path ## Route a legal or data question. These links open a message to the existing company contact with the subject prefilled. [Privacy or data request →](mailto:hello@kroonen.ai?subject=Privacy%20or%20data%20request) [Request a DPA →](mailto:hello@kroonen.ai?subject=Data%20Processing%20Addendum%20request) [Report abuse or a security concern →](mailto:hello@kroonen.ai?subject=Abuse%20or%20security%20concern) [General legal question →](mailto:hello@kroonen.ai?subject=General%20legal%20question) This hub is a navigation aid. Refer to the full published documents for the governing language. --- # Privacy Policy | Kroonen AI > How Kroonen AI, Inc. collects, uses, and protects personal information, with GDPR and CCPA/CPRA rights. [Legal register](https://www.kroonen.ai/legal/) / Privacy Policy Document set [01 Terms](https://www.kroonen.ai/legal/terms/) [02 Privacy](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) Current document Version 2026-06-19 Applies to Website, accounts, communications, and customer data Data policy Document 02 # Privacy Policy How Kroonen AI, Inc. collects, uses, and protects personal information, with GDPR and CCPA/CPRA rights. Status Published Last updated June 19, 2026 Version 2026-06-19 [Terms](https://www.kroonen.ai/legal/terms/) [Privacy](https://www.kroonen.ai/legal/privacy/) [AI Addendum](https://www.kroonen.ai/legal/ai/) [Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) This Privacy Policy explains how Kroonen AI, Inc. ("Kroonen AI," "we," "us," or "our") collects, uses, discloses, and protects personal information in connection with our website, Open-Source Software, SaaS Services, Research previews, and Professional Services. _Last updated: June 19, 2026_ ## 1\. Introduction & Scope This Privacy Policy applies to personal information processed by Kroonen AI in the course of operating the kroonen.ai website and providing our Services, as described in the **Terms of Service**. It should be read together with our **Acceptable Use Policy** and **AI Addendum**. Our role under data-protection law depends on the activity: - **Controller.** For the kroonen.ai website, contact-form submissions, marketing communications, and the administration of accounts for our **SaaS Services** (Libre Bot), Kroonen AI acts as the _controller_ (GDPR) / _business_ (CCPA/CPRA) and decides why and how personal information is processed. - **Processor.** For **Customer Data** that a **Customer** and its **Authorized Users** submit to, or generate through, the Libre Bot widget at librebot.ai, Kroonen AI acts as a _processor_ (GDPR) / _service provider_ (CCPA/CPRA), processing such data only on the Customer's documented instructions under a Data Processing Addendum (DPA). The Customer is the controller of that Customer Data, and the Customer's own privacy notice governs how end users' information is handled. This Policy describes our practices as controller. For Customer Data processed as a processor, see **Section 18 (Data Processing Addendum)**. Our **Open-Source Software** (Libre WebUI and Libre Claw) is distributed under the Apache-2.0 license and runs in environments you control; Kroonen AI does not collect personal information through your self-hosted deployments. Our Research previews (Genesis 1B) are offered as hosted HuggingFace playgrounds; interactions there are also subject to HuggingFace's own privacy practices. ## 2\. Information We Collect We collect the following categories of personal information. We do **not** intentionally collect, and we ask that you do not submit through our forms, any _special-category data_ (GDPR Art. 9) or _sensitive personal information_ (CPRA), such as health, biometric, precise geolocation, racial or ethnic origin, religious beliefs, sexual orientation, or government identifiers. We do not use any personal information for the purpose of inferring sensitive characteristics. ### 2.1 Contact-form and communications data - Name, email address, and the free-text message content you choose to provide when you submit a contact form, email [hello@kroonen.ai](mailto:hello@kroonen.ai), or otherwise correspond with us. - Records of your communications, including support requests and inquiries about Professional Services. ### 2.2 Technical and log data - IP address, user-agent string, approximate country/region, request timestamps, referring URLs, and pages viewed, collected automatically by our hosting and security provider (Cloudflare). - Anti-abuse signals generated by Cloudflare Turnstile when you interact with protected forms (a challenge token and related metadata; Turnstile is designed to be privacy-preserving and does not track users across sites). ### 2.3 SaaS account data - For Libre Bot accounts: account holder name, business email, organization name, authentication credentials, configuration and usage settings, and billing-related identifiers handled through our payment processor (we do not store full payment-card numbers). - Usage and diagnostic logs associated with your account for service operation, security, and support. ### 2.4 Customer Data (processed as processor) - Content, prompts, queries, and documentation that a Customer or its Authorized Users submit to Libre Bot, together with the **Output** generated in response. This is processed under the DPA on the Customer's instructions and is described here for transparency. ## 3\. Sources of Information - **Directly from you**: when you complete a form, create an account, request Professional Services, or communicate with us. - **Automatically**: through server logs and security tooling provided by Cloudflare and Cloudflare Turnstile when you use the website or SaaS Services. - **From our Customers**: when a Customer configures Libre Bot and its Authorized Users or end users interact with the widget (processed as processor). - **From service providers**: limited information from our payment processor and model providers strictly to operate the Services. ## 4\. How & Why We Use Information We use personal information for the following purposes, mapped to the categories above: - **To respond to inquiries and provide support**: using contact-form and communications data (2.1). - **To operate, maintain, and secure the website and SaaS Services**: using technical/log data (2.2) and SaaS account data (2.3), including detecting, preventing, and investigating fraud, abuse, and security incidents. - **To provide and administer SaaS accounts and process payments**: using SaaS account data (2.3). - **To deliver Professional Services** (e.g., pre-training, fine-tuning, dataset preparation, AI safety / red-team / CBRN evaluation, application and telephony development, and AI agents/orchestration): using communications and account data as needed to scope, perform, and document engagements under an Order Form / SOW. - **To send service and, where permitted, marketing communications**: using contact and account data; you may opt out of marketing at any time. - **To comply with legal obligations and enforce our agreements**: using relevant categories as necessary. - **To improve our website and Services**: using aggregated or de-identified technical data. We do **not** use Customer Data, contact-form content, or Output to train, fine-tune, or improve any general-purpose or foundation model. See the **AI Addendum**. ## 5\. Legal Bases for Processing (GDPR) Where the GDPR or UK GDPR applies, we rely on the following legal bases (Art. 6): - **Contract (Art. 6(1)(b))**: to provide the SaaS Services and Professional Services you request, administer accounts, and process payments. - **Legitimate interests (Art. 6(1)(f))**: to respond to inquiries, secure and improve our Services, prevent abuse, and conduct limited direct marketing, balanced against your rights and interests. - **Consent (Art. 6(1)(a))**: where required for non-essential cookies/tracking or certain marketing; you may withdraw consent at any time without affecting prior processing. - **Legal obligation (Art. 6(1)(c))**: to meet tax, accounting, and other legal requirements. ## 6\. Cookies & Tracking; "Do Not Sell or Share" We aim to keep tracking to a minimum. Our website uses only strictly necessary cookies and security mechanisms (including Cloudflare Turnstile) required to operate the site and protect against abuse. Where we use any non-essential cookies or analytics, we will request consent where required and provide controls to manage your preferences. **We do not sell your personal information, and we do not share it for cross-context behavioral advertising**, as those terms are defined under the CCPA/CPRA and similar U.S. state laws. We have not done so in the preceding twelve (12) months. **Global Privacy Control (GPC).** We honor opt-out preference signals, including the Global Privacy Control, transmitted by your browser. Because we do not sell or share personal information, such a signal results in no change to data sales/sharing; we nonetheless treat it as a valid opt-out request for any applicable processing. ## 7\. Disclosures & Sub-processors We disclose personal information only to the categories of recipients below, and only as necessary for the purposes described in this Policy. We do not disclose personal information for monetary or other valuable consideration. - **Cloudflare**: website and SaaS hosting (Pages/Workers), content delivery, logging, and security (including Turnstile anti-abuse). - **Twilio**: telephony processing for Libre Phone / telephony-related Professional Services, where applicable. - **SendGrid**: delivery of contact-form and transactional email. - **Payment processor**: processing of SaaS subscription payments and billing (handled by our third-party payment provider; we do not store full card numbers). - **Model providers**: Anthropic and OpenAI, which process prompts/queries to generate Output within Libre Bot. We select providers that contractually commit not to use API inputs/outputs to train their models, consistent with the **AI Addendum**. - **Professional and legal advisors, and authorities**: where required by law, to enforce our agreements, or to protect rights, safety, and property. - **Corporate transactions**: a successor entity in connection with a merger, acquisition, or sale of assets, subject to this Policy. A current list of sub-processors for the SaaS Services is maintained for Customers and forms part of the DPA. We require sub-processors to provide appropriate safeguards and to process personal information only on our instructions. ## 8\. Automated Decision-Making & AI We do **not** make decisions producing legal or similarly significant effects about you based solely on automated processing, including profiling, within the meaning of GDPR Art. 22. Our SaaS Services and Research previews use AI models to generate Output in response to user input. This generation does not, by itself, constitute a solely automated decision with legal or significant effect on an individual. Output may be inaccurate or incomplete and should not be relied upon as professional advice; further details on AI processing, no-training commitments, and Output are in the **AI Addendum**. ## 9\. Data Retention We retain personal information only for as long as needed for the purposes described, using the following guidelines: - **Contact-form and communications data**: up to **24 months** from your last interaction, unless an ongoing inquiry or legal requirement justifies longer retention. - **Technical and log data**: security and server logs are retained for up to **90 days**, after which they are deleted or aggregated/de-identified, except where retained longer for security investigations. - **SaaS account data**: for the duration of the account plus up to **12 months** after termination, except for billing/tax records retained for up to **7 years** to meet legal obligations. - **Customer Data (as processor)**: retained and deleted in accordance with the DPA and the Customer's instructions; on termination, deleted or returned within the period stated in the DPA. - **Marketing data**: until you opt out or after **24 months** of inactivity, whichever is earlier. We may retain de-identified or aggregated data, which is not used to re-identify you, for longer periods. ## 10\. Your Rights ### 10.1 Rights under the GDPR / UK GDPR Subject to applicable law, you may have the right to: access your personal information; rectify inaccurate data; erase data ("right to be forgotten"); restrict or object to processing; data portability; withdraw consent; and lodge a complaint with your supervisory authority. We will respond to verified requests within **one (1) month**, extendable by up to two further months for complex requests with notice. ### 10.2 Rights under the CCPA/CPRA and other U.S. state laws Subject to applicable law, you may have the right to: know/access the categories and specific pieces of personal information we collect; delete personal information; correct inaccurate information; opt out of sale/sharing (note: we do not sell or share); limit use of sensitive personal information (note: we do not collect it for such purposes); and not receive discriminatory treatment for exercising your rights. We will confirm receipt within **10 business days** and respond to verifiable requests within **45 days**, extendable by an additional 45 days with notice. ### 10.3 How to exercise your rights and verification To submit a request, email [hello@kroonen.ai](mailto:hello@kroonen.ai) or write to the address in Section 17. To protect your privacy, we will take reasonable steps to verify your identity (for example, by confirming control of the email address on file or matching information we hold) before acting on a request. You may use an authorized agent where permitted by law, subject to proof of authorization. Where you are an end user of a Customer's Libre Bot deployment, please direct your request to that Customer (the controller); we will assist the Customer as required by the DPA. ## 11\. International Transfers Kroonen AI is based in the United States, and our service providers may process personal information in the United States and other countries. Where we transfer personal information from the European Economic Area, the United Kingdom, or Switzerland to a country without an adequacy decision, we rely on appropriate safeguards, including the European Commission's **Standard Contractual Clauses (SCCs)**, the **UK International Data Transfer Addendum (IDTA)**, and, where applicable, certification under the **EU-U.S. Data Privacy Framework (including the UK Extension and Swiss-U.S. DPF)**. ## 12\. Security We implement technical and organizational measures designed to protect personal information, including encryption in transit (TLS), access controls and least-privilege practices, infrastructure-level protections provided by Cloudflare, anti-abuse controls (Turnstile), logging and monitoring, and vendor due diligence. **No method of transmission or storage is 100% secure**, and we cannot guarantee absolute security. In the event of a personal-data breach affecting your information, we will notify affected individuals and competent authorities as required by applicable law and without undue delay, and we will cooperate with Customers under the DPA where we act as processor. ## 13\. Children's Privacy Our website and Services are intended for business users and are not directed to children. We do not knowingly collect personal information from children under **16** years of age, and we do not knowingly collect personal information from children under 13 in a manner that would require parental consent under the U.S. Children's Online Privacy Protection Act (COPPA). If you believe a child has provided us with personal information, please contact us and we will delete it. ## 14\. Third-Party Links Our website and Services may contain links to third-party sites and services (for example, HuggingFace playgrounds for Genesis 1B, or model-provider documentation). We are not responsible for the privacy practices of those third parties; we encourage you to review their privacy policies. ## 15\. Changes to This Policy We may update this Privacy Policy from time to time. We review it at least every **12 months** and will revise the "Last updated" date when changes are made. Material changes will be communicated by reasonable means, such as a notice on the website or, for SaaS Customers, by email. Your continued use of the Services after an update constitutes acceptance of the revised Policy where permitted by law. ## 16\. Contact & Data-Protection Requests For questions about this Policy or to exercise your rights, contact us at: - **Kroonen AI, Inc.** - 8 The Green, Ste B, Dover, DE 19901, United States - Email: [hello@kroonen.ai](mailto:hello@kroonen.ai) - Phone: +1 (916) 999-5979 ## 17\. EU / UK Representative Where required under GDPR Art. 27 or UK GDPR, individuals in the EEA or UK may contact our designated representative regarding the processing of their personal information. ## 18\. Data Processing Addendum (DPA) A **Data Processing Addendum** is available for Libre Bot business Customers and governs our processing of Customer Data as a processor, including security measures, sub-processors, international-transfer mechanisms, and breach notification. To request the DPA, contact [hello@kroonen.ai](mailto:hello@kroonen.ai). The DPA, together with the applicable **Order Form / SOW** and the **Terms of Service**, governs that processing and prevails over this Policy with respect to Customer Data. Legal register / Related documents ## Continue through the document set. [Open the register](https://www.kroonen.ai/legal/) [01 Terms of Service →](https://www.kroonen.ai/legal/terms/) [03 AI Addendum →](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use Policy →](https://www.kroonen.ai/legal/acceptable-use/) Entity and contact Kroonen AI, Inc. · 8 The Green, Ste B, Dover, DE 19901, United States [hello@kroonen.ai](mailto:hello@kroonen.ai) --- # Terms of Service | Kroonen AI > The terms governing use of Kroonen AI's open-source software, Libre Bot SaaS, research previews, and professional services. [Legal register](https://www.kroonen.ai/legal/) / Terms of Service Document set [01 Terms](https://www.kroonen.ai/legal/terms/) [02 Privacy](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) Current document Version 2026-06-19 Applies to All Kroonen AI offerings Master agreement Document 01 # Terms of Service The terms governing use of Kroonen AI's open-source software, Libre Bot SaaS, research previews, and professional services. Status Published Last updated June 19, 2026 Version 2026-06-19 [Terms](https://www.kroonen.ai/legal/terms/) [Privacy](https://www.kroonen.ai/legal/privacy/) [AI Addendum](https://www.kroonen.ai/legal/ai/) [Acceptable Use](https://www.kroonen.ai/legal/acceptable-use/) These Terms of Service govern your access to and use of the open-source software, hosted SaaS products, research previews, and professional services offered by Kroonen AI, Inc., and are organized as a master agreement with three service-specific schedules. ## 1\. Agreement & Acceptance **Last updated: June 19, 2026.** These Terms of Service (the **"Terms"**) form a binding legal agreement between **Kroonen AI, Inc.**, a Delaware C-corporation with a registered address at 8 The Green, Ste B, Dover, DE 19901, United States (**"Kroonen AI"**, **"we"**, **"us"**, or **"our"**), and the individual or entity that accesses or uses the Services (**"Customer"**, **"you"**, or **"your"**). By accessing or using any of the Services, creating an account, clicking "I agree," or executing an Order Form or SOW that references these Terms, you accept these Terms and agree to be bound by them. If you do not agree, do not access or use the Services. If you accept these Terms on behalf of a company or other legal entity, you represent and warrant that you have the authority to bind that entity, and "Customer" refers to that entity. The individual accepting is an Authorized User acting within the scope of that authority. **Order of precedence.** A mutually signed Order Form or Statement of Work (each an **"Order Form / SOW"**) that expressly references these Terms controls over these online Terms to the extent of any conflict, but only as to the specific subject matter and engagement described in that Order Form / SOW. The applicable service-specific Schedule controls over the general body of these Terms to the extent of any conflict for that service. The AI Addendum controls with respect to AI Output and model usage. ## 2\. Definitions Capitalized terms have the meanings given below or where first defined in these Terms. - **"Services"** means, collectively, the Open-Source Software, SaaS Services, Research previews, and Professional Services made available by Kroonen AI, as further described in the applicable Schedule and any Order Form / SOW. - **"Open-Source Software"** means software that Kroonen AI publishes under an open-source license, including Libre WebUI and Libre Claw (Apache-2.0) and the Genesis research materials made publicly available under an open-source license. - **"SaaS Services"** means the hosted, subscription-based services Kroonen AI provides, principally Libre Bot, the AI documentation-assistant widget available at librebot.ai. - **"Research"** means hosted research previews, principally the Genesis 1B language model offered solely as hosted HuggingFace playgrounds, with no downloadable weights, no published model card, and private source code. - **"Professional Services"** means consulting, engineering, evaluation, and development services described in an Order Form / SOW, including pre-training, fine-tuning, dataset preparation, AI safety / red-team / CBRN evaluation, application and telephony development, and AI agents and orchestration. - **"Authorized Users"** means employees, contractors, or agents of Customer whom Customer permits to access and use the Services on Customer's behalf. - **"Customer Data"** means data, content, documents, prompts, files, and other materials that Customer or its Authorized Users submit to or generate through the Services, excluding Output. - **"Output"** means content generated by an AI model in response to inputs through the Services, the treatment of which is governed by the **AI Addendum**. - **"Order Form / SOW"** means an ordering document or statement of work executed by the parties that references these Terms and describes the specific Services, fees, scope, and term. - **"Confidential Information"** has the meaning given in Section 10. - **"AUP"** means the **Acceptable Use Policy** published in the Legal Hub and incorporated by reference. - **"AI Addendum"** means the **AI Addendum** published in the Legal Hub and incorporated by reference, governing AI-specific terms including Output, model behavior, and no-training commitments. ## 3\. Services & Scope Kroonen AI provides the Services described in these Terms, the applicable Schedule, and any Order Form / SOW. The specific features, functionality, and availability of each offering are described in the relevant Schedule (A, B, or C) below and may be supplemented by an Order Form / SOW. **Modification and discontinuation.** We may modify, update, or discontinue any Service, feature, or functionality. For paid SaaS Services and Professional Services under an active Order Form / SOW, we will provide reasonable advance notice of any material adverse change or discontinuation, and Section 18 governs material changes that affect active subscribers. For Open-Source Software and Research previews, we may modify or discontinue availability at any time without notice. **Beta and early-access offerings.** Some offerings are identified as beta, preview, early-access, or experimental, including **Libre Phone** (telephony) and the Genesis 1B Research playground. These offerings are provided **"AS IS"** and **"AS AVAILABLE,"** may be incomplete, may change or be withdrawn at any time, may not be supported, and carry no SLA, warranty, or uptime commitment. Customer uses beta and early-access offerings at its own risk. ## 4\. Schedule A: Open-Source Software This Schedule A applies to the Open-Source Software, including Libre WebUI, Libre Claw, and any Genesis research materials published under an open-source license. **License governs solely.** Your rights to use, copy, modify, and distribute the Open-Source Software are granted **solely** under the terms of the applicable open-source license (the Apache License, Version 2.0, or such other license as accompanies the code). These Terms do not modify, expand, or restrict the rights or obligations granted under that open-source license, and in the event of any conflict between these Terms and the applicable open-source license with respect to the Open-Source Software, the open-source license controls. **No warranty; no SLA; no support.** The Open-Source Software is provided **"AS IS"** without warranty of any kind, as set out in the applicable open-source license. Kroonen AI has no obligation under these Terms to provide any maintenance, updates, support, service levels, or uptime for the Open-Source Software. **No trademark grant.** The Open-Source Software is licensed for its code only. No rights are granted in or to the **"Kroonen,"** **"Kroonen AI,"** or **"Libre"** names, logos, or other trademarks, service marks, or trade names. The Apache-2.0 license does not grant trademark rights, and Section 9 (trademark restrictions) applies. Any use of Kroonen AI's marks requires separate prior written permission. ## 5\. Schedule B: SaaS Services (Libre Bot) This Schedule B applies to the SaaS Services, principally Libre Bot available at librebot.ai. ### 5.1 Accounts & Security Customer must register for an account to access most SaaS Services. Customer is responsible for the accuracy of account information, for safeguarding credentials and API keys, and for all activity occurring under its account and Authorized Users. Customer must notify us promptly of any unauthorized access or suspected security incident affecting its account. ### 5.2 Plans, Seats & Quotas The SaaS Services are offered under various plans that may include limits on seats, Authorized Users, message or query volume, API requests, storage, or other usage metrics (collectively, **"Quotas"**). The applicable plan, Quotas, and fees are described at the point of purchase or in an Order Form / SOW. We may meter usage and may throttle, queue, or charge for usage exceeding the applicable Quotas. ### 5.3 Free Tier: no SLA Any free, trial, or no-cost tier of the SaaS Services is provided **"AS IS"** and **"AS AVAILABLE"** with no service-level agreement, no uptime commitment, and no support obligation, and may be modified, rate-limited, suspended, or discontinued at any time. ### 5.4 Suspension We may suspend or restrict access to the SaaS Services, in whole or in part, if (a) Customer fails to pay amounts when due and the failure is not cured within the period stated in Section 7; (b) Customer or an Authorized User breaches the AUP or engages in activity that poses a security, legal, or integrity risk to the Services or others; or (c) suspension is required to comply with law or a government request. Where practicable, we will give notice and an opportunity to cure, except where immediate suspension is necessary to prevent harm. ### 5.5 Data Export on Termination Upon expiration or termination of the SaaS Services, and subject to Section 16, Customer may export its Customer Data in a commonly used format during the export window described in Section 16. After the export window, we may delete Customer Data in accordance with the **Privacy Policy** and our retention practices. ## 6\. Schedule C: Professional Services This Schedule C applies to Professional Services, which are governed by these Terms and the applicable Order Form / SOW. Where an Order Form / SOW conflicts with this Schedule, the Order Form / SOW controls for that engagement. ### 6.1 Engagement Types Professional Services may include pre-training, fine-tuning, dataset preparation, AI safety / red-team / CBRN evaluation, application and telephony development (including Twilio and Cloudflare integrations), and AI agents and orchestration. The specific scope, deliverables, milestones, and assumptions are defined in the Order Form / SOW. ### 6.2 Acceptance Criteria The Order Form / SOW may specify acceptance criteria and a review period for deliverables. Unless the Order Form / SOW states otherwise, a deliverable is deemed accepted if Customer does not provide written notice of material non-conformance within the stated review period (or, if none is stated, within ten (10) business days of delivery). Our remedy obligation for non-conforming deliverables is limited to re-performance of the affected work within the SOW scope. ### 6.3 Client Dependencies Customer will provide timely access to systems, data, personnel, decisions, approvals, and other dependencies reasonably required for the engagement. Delays, inaccuracies, or omissions in Customer dependencies may affect timelines, scope, and fees, and Kroonen AI is not responsible for resulting delays or non-performance. ### 6.4 Safety-Evaluation Findings Findings, reports, scores, and recommendations from AI safety, red-team, or CBRN evaluation engagements are provided **"AS IS"** and reflect testing performed against a defined scope, model version, and point in time. Such findings are **NOT a certification, accreditation, attestation, or guarantee** that any model, system, or deployment is safe, secure, compliant, or free of vulnerabilities or harmful capabilities. Evaluations are inherently non-exhaustive. **Customer is solely responsible for all remediation, mitigation, deployment, and release decisions**, and for the consequences of those decisions. Kroonen AI does not assume any duty to monitor or update findings after delivery. ### 6.5 Subcontractors Kroonen AI may engage qualified subcontractors to perform or assist with Professional Services. Kroonen AI remains responsible for the performance of its subcontractors under the applicable Order Form / SOW. ### 6.6 Deposits, Payment & Withholding The Order Form / SOW may require an upfront deposit and/or milestone payments, which are due before the corresponding work begins or is released. **Title to, and any license to use, deliverables, custom work product, source code, and configured environments transfer to Customer only upon Kroonen AI's receipt of full payment** for the relevant deliverable, milestone, or engagement. Until such full payment is received, Customer obtains no right to use, deploy, copy, host, sublicense, or distribute the deliverables, and any such use is unlicensed and may constitute infringement. If any undisputed invoiced amount is overdue, Kroonen AI may, on seven (7) days' written notice, suspend the engagement and **withhold delivery of and access to deliverables, work product, source code, hosting environments, and administrative credentials** until the overdue amounts (together with any late fees under Section 7) are paid, without prejudice to its other rights and remedies. Kroonen AI will nonetheless return or make available Customer's own personal data to the extent required by applicable data-protection law; withholding under this Section does not authorize deletion of Customer Data except as permitted under Section 16. ## 7\. Fees, Billing & Payment **Fees.** Customer will pay all fees stated at the point of purchase or in the applicable Order Form / SOW. Except as expressly stated in Section 8, fees are non-cancelable and non-refundable. **Auto-renewal.** Unless an Order Form / SOW states otherwise, subscription SaaS Services automatically renew for successive periods equal to the then-current term, at the then-current rates, unless either party gives written notice of non-renewal at least thirty (30) days before the end of the current term. Section 18 applies to material price increases for active subscribers. **Payment terms.** Unless otherwise stated, invoiced fees are due net thirty (30) days from the invoice date; self-serve and online purchases are due upon purchase via the designated payment method. Customer authorizes recurring charges to its payment method for subscription fees. **Taxes.** Fees are exclusive of taxes. Customer is responsible for all sales, use, value-added, withholding, and similar taxes, excluding taxes on Kroonen AI's net income. **Late fees.** Overdue amounts accrue interest at the lesser of 1.5% per month or the maximum rate permitted by law, and Customer is responsible for reasonable costs of collection. We may suspend Services for non-payment as described in Section 5.4 (SaaS Services) and Section 6.6 (Professional Services), and may withhold deliverables, work product, and access for overdue amounts as described in Section 6.6. **Price changes.** We may change pricing for future terms. For active subscribers, material price increases take effect at the next renewal and are subject to the notice requirements in Section 18. ## 8\. Refunds & Cancellation **Subscriptions.** Customer may cancel a subscription to stop future renewals; cancellation takes effect at the end of the then-current billing period. Unless required by law or expressly stated in an Order Form / SOW, fees already paid for the current period are non-refundable, and access continues through the end of the paid period. **Non-refundable amounts.** Setup fees, onboarding fees, one-time fees, usage-based charges already incurred, and Professional Services fees for work performed are non-refundable. **Professional Services.** Refunds for Professional Services, if any, are governed by the applicable Order Form / SOW. ## 9\. Intellectual Property & License Grants **Kroonen AI IP.** As between the parties, Kroonen AI and its licensors retain all right, title, and interest in and to the Services, the SaaS Services platform, the Research models and playgrounds, Kroonen AI's software, documentation, and all related intellectual property, except for rights expressly granted under these Terms or an open-source license. No rights are granted by implication. **License to use the Services.** Subject to these Terms and payment of applicable fees, Kroonen AI grants Customer a limited, non-exclusive, non-transferable, revocable license to access and use the SaaS Services and Research previews during the term, solely for Customer's internal business purposes and in accordance with the AUP. (Open-Source Software is licensed solely under Section 4 / Schedule A.) **Professional-Services deliverables.** Ownership of, and any license to use, deliverables and custom work product created under a Professional Services engagement transfer to Customer only upon Kroonen AI's receipt of full payment, as set out in Section 6.6. Before full payment, Customer receives no license to use, deploy, or distribute such deliverables. **Customer Data.** Customer retains all right, title, and interest in and to Customer Data. Customer grants Kroonen AI a limited, non-exclusive license to host, process, transmit, display, and otherwise use Customer Data solely to provide, secure, and support the Services, and as otherwise permitted by the **Privacy Policy**. Section 10 sets out the no-training commitment. **Output.** Ownership, license, and permitted use of Output are governed by the **AI Addendum**. To the extent of any conflict regarding Output, the AI Addendum controls. **Feedback.** If Customer or an Authorized User provides suggestions, ideas, or feedback about the Services, Customer grants Kroonen AI a perpetual, irrevocable, worldwide, royalty-free license to use and incorporate that feedback without restriction or obligation. **Consulting deliverables.** Ownership of deliverables produced under Professional Services is assigned or licensed as expressly set out in the applicable Order Form / SOW. Absent an express assignment in the Order Form / SOW, deliverables are licensed (not assigned), and Kroonen AI retains ownership of pre-existing materials, tools, methodologies, and generally applicable know-how, with a license to Customer to use them as embedded in the deliverables. **Trademark restrictions.** The **"Kroonen,"** **"Kroonen AI,"** and **"Libre"** names, logos, and marks are the property of Kroonen AI. Except as expressly permitted in writing, Customer may not use, register, or imitate any Kroonen AI mark, and no trademark license is granted under these Terms or any open-source license. ## 10\. Customer Data, Confidentiality & Security ### 10.1 Confidentiality **"Confidential Information"** means non-public information disclosed by one party (the "Disclosing Party") to the other (the "Receiving Party") that is marked or reasonably understood to be confidential, including Customer Data, business and technical information, pricing, and the terms of any Order Form / SOW. Confidential Information does not include information that (a) is or becomes public through no fault of the Receiving Party; (b) was rightfully known without confidentiality obligation before disclosure; (c) is rightfully obtained from a third party without restriction; or (d) is independently developed without use of the Disclosing Party's Confidential Information. The Receiving Party will use Confidential Information only to perform under these Terms, will protect it with at least reasonable care, and will not disclose it except to personnel and subcontractors bound by confidentiality obligations. The Receiving Party may disclose Confidential Information if required by law, provided it gives reasonable prior notice where legally permitted. ### 10.2 No Training on Customer Data **Kroonen AI does not use Customer Data, Customer inputs, or Output to train, fine-tune, or improve any machine-learning or AI models.** This commitment applies to the SaaS Services and Research previews provided to Customer. Additional AI-specific terms, including any model-provider processing of inputs, are set out in the **AI Addendum**. ### 10.3 Privacy Kroonen AI's collection and processing of personal data is described in the **Privacy Policy**, which is incorporated by reference. Where Kroonen AI processes personal data on Customer's behalf, the parties will enter into a data processing addendum as applicable. ### 10.4 Security & Breach Notification Kroonen AI maintains commercially reasonable administrative, technical, and organizational measures designed to protect Customer Data against unauthorized access, use, or disclosure. Kroonen AI will notify Customer without undue delay after becoming aware of a confirmed security breach affecting Customer Data, consistent with applicable law. ## 11\. Acceptable Use Customer and its Authorized Users must comply with the **Acceptable Use Policy**, which is incorporated by reference and forms part of these Terms. Prohibited uses, including prohibited AI uses, are described in the AUP and the AI Addendum. Violation of the AUP is a material breach and may result in suspension or termination under Sections 5.4 and 16. ## 12\. Third-Party Dependencies The Services may rely on or integrate with third-party products and services, including **Cloudflare** (hosting, Pages, Workers), **Twilio** (telephony for Libre Phone), **SendGrid** (email), **Cloudflare Turnstile** (anti-abuse), **Ollama** and other model runtimes, and upstream model providers including **Anthropic** and **OpenAI** (for Libre Bot). These third parties are not controlled by Kroonen AI, and their products and services are provided subject to their own terms. Kroonen AI disclaims responsibility for the availability, performance, security, acts, or omissions of third-party products and services. Customer is responsible for complying with applicable upstream third-party terms where Customer directly procures or directs the use of such services. Third-party outages or changes may affect the Services without liability to Kroonen AI. ## 13\. Warranties & Disclaimers **Limited authority warranty.** Each party warrants that it has the legal authority to enter into and perform these Terms. **DISCLAIMER.** EXCEPT FOR THE EXPRESS WARRANTY ABOVE, THE SERVICES, INCLUDING ALL OPEN-SOURCE SOFTWARE, SAAS SERVICES, RESEARCH PREVIEWS, PROFESSIONAL SERVICES, DELIVERABLES, AND OUTPUT, ARE PROVIDED **"AS IS"** AND **"AS AVAILABLE,"** WITH ALL FAULTS. TO THE MAXIMUM EXTENT PERMITTED BY LAW, KROONEN AI DISCLAIMS ALL OTHER WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT, AND ANY WARRANTIES ARISING FROM COURSE OF DEALING OR USAGE OF TRADE. **AI OUTPUT.** KROONEN AI DOES NOT WARRANT THAT OUTPUT WILL BE ACCURATE, COMPLETE, CURRENT, RELIABLE, OR FIT FOR ANY PURPOSE. AI MODELS CAN PRODUCE INCORRECT, BIASED, OR HARMFUL CONTENT. CUSTOMER IS RESPONSIBLE FOR REVIEWING AND VALIDATING OUTPUT BEFORE RELYING ON OR DEPLOYING IT. KROONEN AI DOES NOT WARRANT THAT THE SERVICES WILL BE UNINTERRUPTED, ERROR-FREE, OR SECURE. Some jurisdictions do not allow the exclusion of certain warranties, so some of the above exclusions may not apply. ## 14\. Limitation of Liability **EXCLUSION OF INDIRECT DAMAGES.** TO THE MAXIMUM EXTENT PERMITTED BY LAW, NEITHER PARTY WILL BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, OR EXEMPLARY DAMAGES, OR FOR ANY LOST PROFITS, LOST REVENUE, LOST DATA, OR BUSINESS INTERRUPTION, ARISING OUT OF OR RELATED TO THESE TERMS OR THE SERVICES, WHETHER IN CONTRACT, TORT, OR OTHERWISE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. **LIABILITY CAP.** TO THE MAXIMUM EXTENT PERMITTED BY LAW, EACH PARTY'S TOTAL AGGREGATE LIABILITY ARISING OUT OF OR RELATED TO THESE TERMS WILL NOT EXCEED THE TOTAL FEES PAID OR PAYABLE BY CUSTOMER TO KROONEN AI FOR THE APPLICABLE SERVICE IN THE TWELVE (12) MONTHS IMMEDIATELY PRECEDING THE EVENT GIVING RISE TO THE CLAIM. FOR FREE-TIER, OPEN-SOURCE, BETA, OR NO-COST SERVICES WHERE NO FEES WERE PAID, EACH PARTY'S TOTAL AGGREGATE LIABILITY WILL NOT EXCEED ONE HUNDRED U.S. DOLLARS (US$100). **CARVE-OUTS.** The exclusions and cap above do not apply to: (a) Customer's payment obligations; (b) either party's indemnification obligations under Section 15; (c) Customer's breach of the AUP, the license scope, or Section 9 (IP) and Section 12 (upstream terms); (d) either party's breach of confidentiality obligations under Section 10 (subject to any separate cap stated in an Order Form / SOW); or (e) liability that cannot be limited or excluded under applicable law, such as for gross negligence, willful misconduct, fraud, or death or personal injury. The parties agree that these limitations are a fundamental basis of the bargain and reflect a reasonable allocation of risk. ## 15\. Indemnification **Customer indemnity.** Customer will defend, indemnify, and hold harmless Kroonen AI and its officers, directors, employees, and agents from and against any third-party claims, damages, liabilities, costs, and reasonable attorneys' fees arising out of or related to (a) Customer Data; (b) Customer's or its Authorized Users' use of the Services in violation of these Terms, the AUP, the AI Addendum, or applicable law, including prohibited AI uses; (c) Customer's deployment or use of Output or deliverables, including remediation and release decisions following safety-evaluation findings; or (d) Customer's violation of third-party rights or upstream third-party terms. **Kroonen AI indemnity.** Kroonen AI will defend Customer against any third-party claim alleging that the SaaS Services, as provided by Kroonen AI and used in accordance with these Terms, directly infringe a valid U.S. patent, registered copyright, or trademark, and will indemnify Customer for damages and reasonable attorneys' fees finally awarded or agreed in settlement. If the SaaS Services are or are likely to become the subject of an infringement claim, Kroonen AI may, at its option and expense, (i) procure the right for Customer to continue using the affected Service; (ii) modify or replace it to be non-infringing while substantially equivalent; or (iii) if neither is commercially reasonable, terminate the affected Service and refund any prepaid, unused fees for it. This indemnity does not apply to claims arising from Open-Source Software, Research previews, beta offerings, Output, Customer Data, modifications not made by Kroonen AI, combination with non-Kroonen products, or use in violation of these Terms. **Procedure.** The party seeking indemnification will (a) give the indemnifying party prompt written notice of the claim; (b) give the indemnifying party sole control of the defense and settlement (provided that no settlement imposing liability or admission on the indemnified party may be made without its consent, not to be unreasonably withheld); and (c) provide reasonable cooperation at the indemnifying party's expense. ## 16\. Term & Termination **Term.** These Terms apply for as long as Customer uses the Services. Subscription terms and renewals are as stated in Section 7 and any Order Form / SOW. **Termination for convenience.** Either party may terminate an at-will or free-tier Service at any time. Termination for convenience of a fixed-term Order Form / SOW is permitted only as expressly stated in that Order Form / SOW. **Termination for cause.** Either party may terminate these Terms or an Order Form / SOW if the other party materially breaches and fails to cure the breach within thirty (30) days after written notice, or immediately if the breach is incapable of cure (such as certain AUP violations, IP misuse, or breaches of confidentiality). **Suspension.** Kroonen AI may suspend Services as described in Section 5.4 before terminating. **Effect of termination.** Upon termination or expiration, Customer's right to access the affected Services ends, and any outstanding fees become due. Customer may export Customer Data during a window of thirty (30) days after termination (the **"export window"**), unless a shorter period is required by the circumstances of termination for cause. After the export window, Kroonen AI may delete Customer Data in accordance with the Privacy Policy and applicable law, except as required to be retained by law. **Deliverables and credentials on termination.** The transfer or release of Professional-Services deliverables, custom work product, source code, hosting environments, and administrative credentials on termination is conditioned on Customer's payment of all undisputed outstanding fees, as set out in Section 6.6. Kroonen AI will return Customer's own personal data as required by applicable data-protection law, but may withhold deliverables, work product, and non-essential access while undisputed amounts remain overdue. This does not limit Customer's right to export Customer Data during the export window above. **Survival.** Sections 2 (Definitions), 7 (accrued fees), 8, 9, 10, 13, 14, 15, 16 (effect and survival), 17, 19, and 20, and any provision that by its nature should survive, survive termination. ## 17\. Governing Law & Dispute Resolution **Governing law.** These Terms are governed by the laws of the State of Delaware, United States, without regard to its conflict-of-laws rules. The U.N. Convention on Contracts for the International Sale of Goods does not apply. **Informal resolution.** Before initiating arbitration, the parties will attempt to resolve any dispute informally by sending written notice describing the dispute to the other party and negotiating in good faith for at least thirty (30) days. **Binding arbitration.** Except for the carve-outs below, any dispute arising out of or relating to these Terms or the Services will be finally resolved by binding arbitration administered by the American Arbitration Association (AAA) or, by mutual agreement, JAMS, under its then-current commercial rules. The arbitration will be seated in Delaware, conducted in English, and decided by a single arbitrator. Judgment on the award may be entered in any court of competent jurisdiction. **Class-action and jury waiver.** TO THE MAXIMUM EXTENT PERMITTED BY LAW, DISPUTES WILL BE RESOLVED ONLY ON AN INDIVIDUAL BASIS, AND EACH PARTY WAIVES ANY RIGHT TO PARTICIPATE IN A CLASS, COLLECTIVE, OR REPRESENTATIVE ACTION AND ANY RIGHT TO A JURY TRIAL. **Carve-outs.** Either party may (a) bring an individual claim in small-claims court if it qualifies; and (b) seek injunctive or equitable relief in a court of competent jurisdiction in Delaware to protect its intellectual property or Confidential Information. Customer consents to the exclusive jurisdiction and venue of the state and federal courts located in Delaware for such matters. **30-day opt-out.** Customer may opt out of the arbitration and class-action-waiver provisions by sending written notice to hello@kroonen.ai within thirty (30) days of first accepting these Terms. Opting out does not affect the other provisions of these Terms. ## 18\. Changes to Terms We may update these Terms from time to time. We will post the updated Terms in the Legal Hub and revise the "Last updated" date. For non-material changes, the updated Terms take effect upon posting. **Material changes.** For material changes that adversely affect active paid subscribers, we will provide advance notice (by email or in-product) before the changes take effect, and where required, active subscribers will be asked to affirmatively re-accept the updated Terms to continue using the affected Services. If an active subscriber does not accept a material change, its sole remedy is to stop using the affected Services and, where applicable, terminate as provided in Section 16. **Versioning.** Prior versions are identified by their "Last updated" date, and we will maintain reasonable version history. Continued use of the Services after changes take effect constitutes acceptance, subject to the re-acceptance requirement above. ## 19\. General **Assignment.** Customer may not assign or transfer these Terms without Kroonen AI's prior written consent, except to a successor in a merger or sale of substantially all assets, provided the successor is not a competitor and assumes these Terms. Kroonen AI may assign these Terms to an affiliate or in connection with a merger, acquisition, or sale of assets. Any prohibited assignment is void. **Force majeure.** Neither party is liable for delay or failure to perform (other than payment obligations) due to causes beyond its reasonable control, including acts of God, natural disasters, war, terrorism, civil unrest, labor disputes, governmental action, internet or utility failures, and failures or changes of third-party providers. **Severability.** If any provision is held unenforceable, it will be modified to the minimum extent necessary, or severed, and the remaining provisions remain in effect. **No waiver.** A party's failure to enforce a provision is not a waiver of its right to do so later. **Entire agreement.** These Terms, together with the applicable Schedules, the AUP, the AI Addendum, the Privacy Policy, and any Order Form / SOW, constitute the entire agreement between the parties regarding the Services and supersede all prior or contemporaneous agreements on that subject. **Notices.** Legal notices to Kroonen AI must be sent to hello@kroonen.ai and to 8 The Green, Ste B, Dover, DE 19901, United States. We may give notice to Customer by email to the account contact or by in-product notification. **Export control and sanctions.** Customer will comply with applicable export-control and economic-sanctions laws and represents that it is not located in, or a national of, an embargoed country, and is not on any restricted-party list. Customer will not use the Services in violation of such laws. **U.S. government end users.** The Services are "commercial items" / "commercial computer software" as defined in applicable Federal Acquisition Regulations, and any use, duplication, or disclosure by the U.S. Government is subject to the restrictions in these Terms. **Relationship.** The parties are independent contractors; these Terms do not create a partnership, agency, joint venture, or employment relationship. **Independent development.** Nothing in these Terms limits Kroonen AI from developing, acquiring, or marketing products or services that are similar to or compete with Customer's, provided no Confidential Information is used in breach of Section 10. ## 20\. Contact Questions about these Terms may be directed to: - **Kroonen AI, Inc.** - 8 The Green, Ste B, Dover, DE 19901, United States - Email: hello@kroonen.ai - Phone: +1 (916) 999-5979 Legal register / Related documents ## Continue through the document set. [Open the register](https://www.kroonen.ai/legal/) [02 Privacy Policy →](https://www.kroonen.ai/legal/privacy/) [03 AI Addendum →](https://www.kroonen.ai/legal/ai/) [04 Acceptable Use Policy →](https://www.kroonen.ai/legal/acceptable-use/) Entity and contact Kroonen AI, Inc. · 8 The Green, Ste B, Dover, DE 19901, United States [hello@kroonen.ai](mailto:hello@kroonen.ai)