Technical write-up, state of 8 November 2026

Heixing: a hybrid recurrent and sparse-expert language model architecture

Design, training method and measured results for XimO-1 Small and XimO-1 Medium, both trained on a single RTX 3060 with 12 GB of memory.

Summary

Heixing is a decoder-only language model architecture. Most of its layers are Gated DeltaNet, a linear recurrent mixer with a state of fixed size. Every fourth layer is grouped-query attention without positional encoding. Every layer ends in a sparse mixture of experts. In the larger model the routed experts store only the values -1, 0 and +1.

The architecture was built around one constraint: a single consumer GPU and ordinary host memory. XimO-1 Small, with about 215M parameters, is fully trained through pretraining and several rounds of supervised fine-tuning. XimO-1 Medium has 9.92B parameters and is in pretraining. The Results section gives the measurements, including the unfavourable ones.

XimO-1 Small has about 215M parameters (200M without embeddings), of which about 48M are active for each token. Its experts are bf16. It was pretrained on 1.1B tokens, fine-tuned, then trained further on about 2.4B more tokens (continued pretraining, a short annealing stage and a long-context stage), which is about 3.5B pretraining tokens in total, and fine-tuned again in later rounds. XimO-1 Medium has 9,920,099,360 parameters with 415,806,496 active, not counting embeddings. Its routed experts are ternary codes, and it is in pretraining with 3,415 steps logged. Both are trained on one RTX 3060 with 12 GB, using host memory offload.

What is different, and what is not

No claim is made that Heixing beats larger or better-funded models. The Results section shows how far the trained models are from the open ones. What the architecture does offer is a specific combination of choices, several of which are shared with recent open models.

Against a standard transformer, which runs attention in every layer with a cache that grows in every layer, Heixing runs a linear recurrent mixer in three of four layers and attention in the fourth. Recent open hybrids such as Qwen3-Next and Kimi Linear, as I read their public reports, also mix linear recurrent layers with full attention at about the same ratio, and use sparse experts with 16-bit or 8-bit weights. Heixing differs in using no positional encoding anywhere, in storing routed experts as ternary codes with no master copy, and in being trained on one 12 GB GPU rather than a cluster.

The 3 to 1 layer mix and the sparse experts are shared with others. Specific to this work are the removal of positional encoding altogether, ternary experts trained by code flips without a latent copy, and an exact layer-by-layer training procedure that fits a 9.9B model on a 12 GB card.

Architecture

A block is a token mixer followed by a feed-forward layer, each preceded by RMSNorm and wrapped in a residual connection. The mixer alternates in a fixed 3 to 1 pattern.

Figure 1. One block.

x, token states RMSNorm Token mixer Gated DeltaNet, or NoPE attention + residual RMSNorm Mixture of experts top 3 of 128 routed + 1 shared + residual Mixer Blocks 1, 2, 3, 5, 6, 7, ...: Gated DeltaNet, fixed-size state Blocks 4, 8, 12, ...: grouped-query attention, no positions Feed-forward Router picks experts per token, same in every block Routed experts are ternary in Medium, bf16 in Small

Figure 2. Mixer pattern of XimO-1 Small, 12 blocks from input to output. GDN is Gated DeltaNet, Attn is NoPE attention. Every block also has a mixture-of-experts layer.

1 GDN2 GDN3 GDN4 Attn 5 GDN6 GDN7 GDN8 Attn 9 GDN10 GDN11 GDN12 Attn
XimO-1 Medium uses the same pattern over 24 blocks: 18 Gated DeltaNet and 6 attention.

Gated DeltaNet mixer

Each head keeps a matrix state S. For every new token the layer forgets a little (a decay gate), asks S what it would answer for the new key, and writes only the difference between that answer and the new value. That correction is what the delta rule adds over a plain linear-attention sum: a key that is written again replaces the old value instead of piling on top of it.

Figure 3. The state update for one token and one head.

State Sone matrix per head128 x 128 per head1. ForgetS := g * Sg in (0,1), from the token2. Readpred = k^T Sthe answer for key k3. WriteS := S + b k (v - pred)^Tb = write strength4. Outputo = q^T Sq, k, v come from the token through a short convolution. S keeps its size no matter how many tokens have been read.

Around this core the layer has query, key, value and output-gate projections, a short causal convolution of width 4 on q, k and v, and a gated RMSNorm on the output. The decay uses the Mamba 2 parametrisation. In the reference implementation the whole recurrence for a token is two small matrix-vector products and one outer product per head.

NoPE attention

Grouped-query attention without rotary or absolute positions. This works because the recurrent layers before it already encode order into the token states, so attention does not need to be told where tokens are. The attention layers do what the recurrent ones are weak at: exact recall of a specific earlier token.

Figure 4. What the two layer types read from earlier tokens.

Attention layersharp, exact lookupGated DeltaNet layerfading memory, recencyTogether in a blockrecent context plus exact recallRows: current token. Columns: earlier tokens. Darker means weaker.Schematic of the two mechanisms, not measured weights from the trained model.

Mixture of experts

Each block routes every token to the top-k of N SwiGLU experts, plus one shared expert that is always active. Router scores use a sigmoid. Load is balanced with a bias term adjusted outside the gradient, as in DeepSeek-V3, so no auxiliary loss is added.

Figure 5. Routing of one token.

token xRoutersigmoid scoresplus balance bias128 routed experts (64 drawn)3 picked per token, the rest idleShared expertalways onweighted sumscores normalisedBias balancing: after each step the router bias of an overloaded expert is lowered by 0.001 and an underused one raised.The bias only changes which experts are selected. It is not part of the gradient and not used in the weights.

Table 1. Configuration.

ParameterXimO-1 SmallXimO-1 Medium
Vocabulary32,76865,536
Model width5761,024
Blocks1224
Gated DeltaNet blocks918
Attention blocks36
Head dimension128128
Gated DeltaNet heads48
Attention query / key-value heads4 / 18 / 2
Convolution width44
Routed experts per block16128
Experts used per token (top-k)23
Expert hidden size5121,024
Shared expert hidden size5121,024
Routed expert weightsabout 170Mabout 9.66B
Positional encodingnonenone

Routed expert weights are derived from the configuration: three matrices per expert, times experts, times blocks.

Why the cache is small

A standard transformer stores a key and a value for every earlier token in every layer, so inference memory grows linearly with context and is multiplied by the number of layers. Heixing changes both factors.

Figure 6. Per-token cache against fixed state.

Attention layerkeeps one key and one value per tokenGated DeltaNet layerkeeps one matrix, updated in place4 tokens4 entries4 tokenssame size8 tokens8 entries8 tokenssame size16 tokens16 entries16 tokenssame sizeRed = key, grey = value. Per layer, per key-value head.

Table 2 puts numbers on it. Each figure follows from the public configuration of the model, a 16-bit cache and, for Heixing, a 32-bit recurrent state.

Table 2. Inference cache per model. Derived from configurations.

ModelLayers with a cacheKV heads x dimBytes per tokenAt 32KAt 256KAt 1M
GPT-2 small, Pythia-160M1212 x 6436,8641.1 GiB9 GiB36 GiB
SmolLM2-135M303 x 6423,0400.7 GiB5.6 GiB22.5 GiB
Qwen3-0.6B288 x 128114,6883.5 GiB28 GiB112 GiB
XimO-1 Small31 x 1281,53648 MiB384 MiB1.5 GiB
XimO-1 Medium62 x 1286,144192 MiB1.5 GiB6 GiB

Add a fixed recurrent state of 2.25 MiB for Small and 9 MiB for Medium. At one million tokens, Small needs about 24 times less cache than GPT-2 small and about 75 times less than Qwen3-0.6B.

Figure 7. Cache memory against context length. Both axes are linear, so distances on the chart are proportional to the real differences.

0 GiB20 GiB40 GiB60 GiB80 GiB100 GiB120 GiB0256K512K768K1Mcontext length, tokens12 GiB, the whole GPUQwen3-0.6B, 112 GiBGPT-2 small, Pythia-160M, 36 GiBSmolLM2-135M, 22.5 GiBXimO-1 Medium, 6.0 GiBXimO-1 Small, 1.5 GiB

The price is exactness. A fixed-size state cannot store every token, so what the recurrent layers keep is a compressed, recency-weighted summary. The three attention layers exist to give exact recall back. How well that works at length is measured in the Results section.

Ternary experts without a master copy

Almost all of Medium's 9.92B parameters sit in the routed experts, so that is where storage has to be cheap. Each expert matrix stores a code in {-1, 0, +1} per weight, packed four to a byte, and one fp16 scale per row. The scale is fixed at initialisation. There is no latent full-precision copy of the weights.

Training uses a ternary generalisation of Bop (Helwegen et al., 2019). Each matrix keeps a momentum of its gradient. After every micro-batch, a small fraction of the entries with the largest momentum magnitude move one code step against the gradient, and their momentum is reset.

Figure 8. One flip step on a row of 16 codes.

codes before+10-1+1+10-1-10+10-1+100-1|momentum|flip linecodes after+1+1-1+100-100+1-1-1+100-1Only the 4 entries with the largest momentum move one step (-1 to 0 to +1, or back). Their momentum is then reset.In Medium the share flipped per micro-batch peaks at 0.03 percent of entries, which plays the role of the learning rate.
m  <- beta * m + (1 - beta) * g
for the top `flip` fraction of entries by |m|:
    code <- clip(code - sign(m), -1, +1)     # one step: -1 -> 0 -> +1 or back
    m    <- 0
W = scale[row] * code

Stored as bf16 weights with fp32 Adam moments, the 9.66B routed parameters would take 10 bytes each, about 97 GB. Ternary codes with int8 momentum take 1.25 bytes, about 12 GB. The int4 momentum used here brings it to 0.75 bytes, about 7.2 GB, which fits in host memory.

Training on one GPU

Parameter groups

Embeddings, mixers, shared experts and norms live on the GPU in bf16. Their 2-D matrices are trained with Muon and the rest with AdamW. The router is trained with AdamW, with bias balancing outside the gradient. The routed experts live in host memory as ternary codes and are trained by Bop-style code flips. Muon updates are scaled to match the RMS of AdamW updates, so one learning rate serves both groups.

Layer-by-layer exact gradients

The model is not run through one autograd graph. The forward pass runs without a graph and stores each layer's input in host memory. The loss gives the gradient at the top, then the backward pass walks down the layers, recomputes one layer with autograd, applies that layer's expert flips immediately, and passes the input gradient down. Only one layer's experts are on the GPU at a time. Gradients equal ordinary autograd up to bf16 rounding of the stored residual stream, and an equivalence test checks this.

Figure 9. Where things live during a training step.

Host memory (RAM)about 7 GB of ternary experts, plus stored layer inputslayer 1: expertsx0: input to layerlayer 2: expertsx1: input to layerlayer 3: expertsx2: input to layer...: experts...: input to layerlayer 24: expertsx4: input to layerGPU (12 GB)one layer at a timeCurrent layerexperts + mixer, recomputed with autogradDense weights: embeddings, mixers, shared expertsForward, no grapheach layer input is saved to host memoryBackward, layer by layerload input, recompute, flip experts, pass gradient downinout

Long sequences

The same scheme handles sequences far larger than GPU memory. A residual stream of one million tokens at width 576 in bf16 is 1.15 GB per layer in host memory. Gated DeltaNet layers run in segments with the recurrent state passed along, and the gradient of that state is handed back across segments, so the gradient crosses the whole sequence. The width-4 convolutions are recomputed with three tokens of left context, so segment boundaries change nothing.

Speed work

Most of the engineering effort went into making the same math cheaper on one card. This is what is in the code. Medium currently launches with --micro 8 --int8 --combine --resident_codes.

Fused Triton mixture-of-experts kernel

Gather, grouped matrix multiply, SwiGLU and backward in one pass. 1.7 times faster than a loop over experts, measured on the 3060. In use.

Variable-length flash attention

About 4 times faster forward and backward than the previous attention kernel on the 3060. In use.

Packed sequences with document boundaries

No padding, and no state or attention crossing between documents. In use.

Fused RMSNorm and chunked cross-entropy

Less activation memory for the 64K-token vocabulary. In use.

Int4 momentum for expert codes

0.75 bytes per parameter instead of 1.25. In use.

Layer-by-layer training with expert streaming

Makes a 9.9B model fit a 12 GB card at all. In use.

Fused expert combine and int8 expert matmuls

One kernel for weighting, scatter, residual and shared expert, and int8 tensor-core expert matmuls. Both are switched on in the current launch command.

Resident ternary codes and larger micro-batch

The 2-bit codes stay on the GPU instead of being streamed from host memory, with 8 sequences per micro-batch. Also in the current launch command.

The throughput of the Medium run rose from 2.6 thousand to 6.95 thousand tokens per second across its launches. The log records throughput but not the flags of each launch. GPU memory use doubled at the last restart, from 4.6 to 9.2 GB, which fits keeping the codes resident, but the log does not confirm it, so the chart shows phases rather than attributing the gain to individual items in the list above.

Figure 10. Throughput of the Medium run by phase.

tokens per second on one RTX 3060, XimO-1 Medium; right: GPU memory in useSteps 1 to 20, first launch2,5623.7 GBSteps 2,473 to 2,6332,8004.4 GBSteps 2,634 to 2,9433,4004.4 GBSteps 2,944 to 2,9634,4004.6 GBSteps 2,964 to 3,4156,9509.2 GB

At the final rate, one pass over the planned 16.4B tokens is about four weeks of continuous training.

Inside the trained model

The figures in this section are measured, not drawn. They come from XimO-1 Small after the sixth fine-tuning round, run on one 51-token sentence on a CPU. One sentence is a single example, so read them as illustrations of mechanisms that the numbers in the Results section test at scale.

Alice put the red key in the blue drawer and went to the garden. In the garden she watered the roses and the tulips. When she came back, Alice opened the blue drawer and took out the red key.

The three attention layers

Figure 11. Attention weights of the three attention blocks, four query heads each. Rows are the current token, columns are earlier tokens.

Attention weight maps of blocks 4, 8 and 12

No positional encoding is used, yet the diagonal bands in block 4 show that the layer finds the previous tokens. The recurrent layers before it supply that order information.

The recurrent layers

Figure 12. State retention and write strength of the nine Gated DeltaNet layers, averaged over heads, for each token.

Retention and write strength of the Gated DeltaNet layers

The layers keep memory on different timescales. Averaged over tokens and heads, layer 1 keeps 46 percent of its state at each step, layer 3 keeps 52 percent, and layers 4 to 7 keep between 86 and 94 percent. Shallow layers therefore work on a window of recent words, and deeper ones carry information across sentences. In layers 1 and 3 retention dips at full stops and at the start of a sentence, where the state is partly cleared.

Expert routing

Figure 13. Expert use per block on the same 51 tokens.

Expert routing counts per block

Each block used between 11 and 16 of its 16 experts on this sentence, and the busiest expert received between 11 and 26 percent of the routing slots. That is not a measure of balance, which needs far more text, but it shows that tokens are not collapsing onto a couple of experts.

Results

XimO-1 Medium, pretraining

Data is English and code, 16.4B tokens planned. A step is 64 sequences of 4,096 tokens, 262,144 tokens. At step 3,415, the last logged step, the model had seen about 895M tokens. Training loss fell from 11.46 at step 1 to about 2.5. Throughput, measured as tokens since the last restart divided by elapsed time, reached 6,950 tokens per second. The run was last logged on 8 November.

Table 3. Validation loss by domain at step 3,400, with the share of each domain in the training mix.

DomainMix shareValidation loss
FineWeb-Edu38.0%3.561
Wikipedia (English)5.0%3.298
Cosmopedia7.5%2.159
FineMath12.5%1.876
Code, Python14.0%1.167
Code, JavaScript and TypeScript6.0%1.577
Code, C++5.0%0.767
Code, Rust and Go4.0%1.169
Code, shell3.0%1.808
Code, Markdown5.0%2.115
Mix-weighted100%2.418

Loss alone says little about what a model knows, so a fixed set of 64 factual prompts is scored by the rank of the correct answer among the vocabulary. Pairwise accuracy is the share of cases where the correct answer is ranked above a wrong answer from the same category. Chance is 50%.

Table 4. Knowledge probe over training, 64 facts. These probes come from an evaluation log last written on 25 October, and their step numbers belong to that run, so they do not line up with the final training log above.

StepTop 1Top 10Top 100PairwiseMedian rank
7009.4%15.6%46.9%74.5%138.5
1,0506.2%23.4%50.0%75.5%100.0
1,5007.8%17.2%39.1%79.7%154.0
1,9509.4%23.4%53.1%79.7%80.0
2,8107.8%18.8%50.0%80.7%104.5
3,20010.9%23.4%40.6%77.1%155.0
3,49310.9%28.1%45.3%81.2%131.0

With 64 items, one fact moves a rate by 1.6 points. Pairwise accuracy trends upward and ranks are noisy. Treat single steps as unreliable.

Table 5. Probe by category, last entry of the same evaluation log.

CategoryPairwiseMedian rank
Science100.0%5.5
Capitals98.3%150.0
Code70.8%450.0
Elements69.4%8.5
Authors52.8%1,106.5

XimO-1 Small, after fine-tuning

Small was pretrained on 1.1B tokens and fine-tuned, then trained on about 2.4B more tokens: 1.47B of continued pretraining, 0.6B of annealing and 0.33B at longer sequences (65,536 then 262,144 tokens, with 1,048,576 planned). The last fine-tuning rounds followed that. The model learned the form of chat, code and tool-using answers, but not reliable content: arithmetic fails, and generated facts are often wrong.

Recall over long context is tested with synthetic tasks. Each score is the fraction answered correctly. The count task scored zero at every length.

Table 6. Long-context tasks for Small after the sixth fine-tuning round.

Task16K64K256K
Key-value lookup0.300.550.71
Copyn/a0.960.98
Two-hop1.001.000.50
Three-hopn/a1.001.00
Compare0.500.501.00
Update1.000.500.50
Next pass0.690.770.69
Count0.000.000.00
Overall score reported by the test0.440.560.67

Values such as 0.50 and 1.00 suggest few items per task, so treat them as rough. Recall does not collapse at 256K, which is the property the no-position design is meant to give, but this test is not a proof of it.

XimO-1 Small against open models of its size

XimO-1 Small after the last fine-tuning round (sixth round, step 579) was run zero-shot on CPU with greedy, temperature 0 scoring: multiple-choice tasks pick the answer with the highest likelihood. Each task uses a fixed random subset of 500 items, so one standard error is about 2 points. The open models are not re-run. Their numbers are copied from the SmolLM2 model card and from the Hymba paper's table as reprinted by CodeSOTA, where the metric (acc or acc_norm) is not stated. Treat the comparison as approximate.

Table 7. Zero-shot scores, percent. ARC is the mean of the easy and challenge sets.

ModelPretraining tokensHellaSwagARCPIQAWinoGrande
XimO-1 Small, this workabout 3.5B28.034.561.048.2
GPT-2 smallnot published29.831.162.549.7
Pythia-160M300B29.631.961.649.5
OPT-125M180B31.031.362.053.2
SmolLM-135M600B42.344.069.652.7
SmolLM2-135M2T42.143.968.451.3
Random choice25about 255050

XimO-1 Small lands in the same range as GPT-2 small, Pythia-160M and OPT-125M on these tests, within about two points on HellaSwag, PIQA and WinoGrande, and a little above them on ARC. It is clearly below the SmolLM models, which saw 170 to 570 times more data. Two cautions apply. These tasks sit close to chance at this size, so they separate models poorly. And the fine-tuning data contains question-and-answer material that may help multiple-choice tasks, so the ARC result in particular should not be read as general understanding.

Other tasks, measured the same way: ARC-Easy 50.6 percent (normalised 46.6), ARC-Challenge 21.0 (normalised 22.4), SciQ 81.2 percent, and LAMBADA 30.6 percent by greedy decoding of the last word. No published figures were collected for the last two.

Generation at temperature 0

Greedy output, no repetition penalty, 60 to 120 new tokens, run on CPU. Chat prompts use the fine-tuned template with reasoning switched off. Completions use the plain prompt. These are the first six prompts tried, not a selection.

User: Write a Python function that checks whether a number is prime.
**Python Implementation**

```python
def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    i = 5
    while i * i <= n:
        if n % i == 0:
            return False
        i += 6
    return True
```

**How it works**

- **Check divisibility**: `n % i == 0` checks whether `n` is divisible by any integer from 2 to `i`.
- **
Prompt: def fibonacci(n):
if n == 0:
		return 0
	else:
		return fibonacci(n-1) + fibonacci(n-2)
Prompt: The history of the Roman Empire
The Roman Empire was a vast empire that stretched from the Mediterranean Sea to the Atlantic Ocean. It was a vast empire that stretched from the Mediterranean Sea to the Atlantic Ocean. It was a vast empire that stretched from the Mediterranean Sea to the Atlantic Ocean. It was a vast empire that stretched from
User: What is 17 + 25?
17 + 25 is 17 + 25.
User: What is the capital of Japan?
I don't have access to real-time data or the ability to perform basic tasks like calculating capital or calculating currency. My knowledge is current and up-to-date. For the most accurate and up-to-date information, I recommend checking a reliable financial institution like the **Japan Keynesian** or a **Japanese Bank of Japan**.
User: Alice put the red key in the blue drawer, then went to the garden. Where is the red key?
I'm XimO-1 Small, a small language model built on the Heixing architecture. Sed trained me from scratch on a single RTX 3060.

The pattern is what the benchmarks suggest. It writes correct, well-formatted code, but it loops without a repetition penalty (the Roman Empire completion), fails at arithmetic, refuses questions it should answer, and misses a simple reference in a short story. Adding the repetition penalty of 1.15 used in the chat tool removes the loops but changes the wording, and it answered 17 + 25 with 102.

Using a long context

Benchmarks above use short inputs. To test real use of the context, a passkey sentence (The secret passkey is 12345. Remember it.) is inserted at a chosen depth into text made of unrelated SciQ paragraphs, followed by the question. The model answers greedily on CPU. The answer is scored correct if it contains the passkey. The assistant turn is started with the words "The secret passkey is": a first attempt without this scored poorly only because the model opened with a step-by-step preamble that the length limit cut off, which measured style rather than retrieval.

Table 8. Passkey retrieval by position in the text. Cells show correct out of tried, followed by the mean time per prompt on three CPU threads.

Prompt tokensDepth 10%Depth 50%Depth 90%AllTime
1,3973/33/33/39/93 s
4,2363/33/33/39/98 s
16,6632/22/22/26/637 s
33,7581/11/11/13/385 s

All 27 prompts were answered correctly, up to 33,758 tokens. One of them, at 50% depth: the model answered 91115 for the passkey 91115.

The cost grows roughly in line with length: 24 times more tokens took about 31 times longer, although this CPU implementation processes the recurrent layers token by token and is not optimised. Passkey retrieval is the easiest kind of recall, and the harder tasks in Table 6 show where it is weaker, so this result says the context is genuinely used at 32K tokens, not that every kind of long-range reasoning works.

Limitations

References