Cut Your Token Usage by Up to 36% (Sponsored)
AI coding agents can generate code quickly, but CI checks often happen after the agents finish their work. Sonar Vortex changes this pattern. Sonar Vortex operates inside the agent’s coding loop, giving agents architectural context before they write and verifying their output in real time as they produce it. Internal testing found 36% lower token consumption and 92% fewer defects. Thinking Machines released a model called Inkling on July 15, 2026. A few interesting points made in its introduction are as follows:
Thinking Machines, founded by Mira Murati (the ex-CTO of OpenAI), describes its mission as building AI that extends human will and judgment. The company lists four directions of work, which are training strong models, building tools that let people customise models with their own knowledge, developing interfaces that widen the communication channel between people and machines, and publishing research on how models are made [3]. Before Inkling, the company shipped Tinker, a service for fine-tuning open models [4]. Inkling is the company’s first model trained from scratch [1]. The weights sit on Hugging Face under an Apache 2.0 license [2], so anyone can download them and retrain the model on their own data. In this article, we will work through the various choices Thinking Machines made while building Inkling. Here is what we will cover:
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies. The diagram below shows where each of these five things sits inside the model.
GroundworkLet us first understand four key terms that are really important to make sense of the architecture:
SparsityInkling separates the cost of storing a model from the cost of running it. This is the reason a model this large is affordable to use. In an ordinary transformer, the feed-forward step in each layer is a single network, and every token passes through all of it. If that network holds 5 billion parameters, then every token processed involves all 5 billion. Inkling replaces that single network with 256 smaller ones, called experts. For each token, a selection step picks six of the 256. Only those six run, and the remaining 250 sit idle for that token while handling other tokens instead [2]. This design pattern is called Mixture of Experts, and Thinking Machines states that their version largely follows the approach published by DeepSeek [1][8]. See the diagram below:
Let’s understand this via a simple example:
Inkling does this at scale. The total across the whole model is 975 billion parameters, and the count involved in processing any single token is roughly 41 billion [1]. That is about 4 percent of the model running at a time. The hardware requirements make things clearer. A checkpoint is the saved file holding all the trained parameters. Inkling’s full-precision checkpoint needs at least 2 TB of combined GPU memory, which the model card gives as eight NVIDIA B300 cards or sixteen H200 cards [2]. Thinking Machines also talks about a quantised checkpoint. This means the same parameters are stored with less numerical precision, in roughly the way that writing 3.14 instead of 3.14159265 uses less space at the cost of some accuracy. That version needs around 600 GB and fits on four B300 cards [2]. What Makes an FDE Role Credible? (Sponsored)
Strong candidates are skeptical of vague forward deployed engineer postings, and the title alone won’t earn their trust. The free State of FDE Jobs 2026 Report explains what candidates look for, how the market is evolving, and how employers can make these roles easier to understand. Hiring? You can also bring your openings to forwarddeployedengineer.com, the focused jobs board for forward-deployed engineers. RoutingPicking six experts out of 256 sounds simple, but it is not. Here’s how the selection works:
This approach makes one specific type of failure more probable. To understand the failure, consider what happens across millions of training steps. Suppose, by luck, expert 47 gets picked slightly more often than average early on. It receives more tokens, so it receives more gradient updates, so it improves faster than its neighbours. Since it is better, the router scores it higher. Because it scores higher, it gets picked even more. If we run that loop long enough, we can end up with a 256-expert layer where perhaps 20 experts handle nearly everything and the other 236 stay underdeveloped. This is called routing collapse. In other words, while we paid to store 256 experts, we got the capability of just 20. There is a second cost too. Since experts are usually spread across different machines, a machine holding four popular experts can become a bottleneck while its neighbours idle [9]. The traditional fix for this adds a penalty to the training objective that grows when expert usage is uneven. This means training the model on two things at once: predicting the next token correctly, and keeping expert usage balanced. The trouble is that these two goals produce gradients pointing in different directions. The prediction gradient might say “increase this parameter,” while the balance penalty says “decrease it.” One of them wins, and either way the model is being pulled away from what you actually wanted. If we set the penalty strength high, the text quality degrades. If we set it too low, the experts collapse anyway [9]. Thinking Machines uses a method introduced by Wang and colleagues, which was also adopted by DeepSeek [1][8][9]. This method removes the conflict entirely by keeping a separate bias value for each expert, which is just a small number added to that expert’s score. The trick is where the bias gets applied.
Let us walk through a token arriving at a layer in this setup. Consider that the router produces these sigmoid scores for four of the experts. Expert 47 has been overloaded recently, so its bias has drifted down to −0.15. This drops its selection score below expert 88’s, and expert 88 takes the slot instead. The bias changed which expert got picked. When expert 88’s output is combined into the final result, it is weighted by 0.80, its original router score, with the bias left out. In other words, the bias affects selection only. It never touches the weighting, and it is updated by a simple counting rule that runs outside backpropagation entirely. The main goal is to nudge down busy experts while giving a chance to the quieter experts.
The result is that expert usage stays balanced, and the training objective receives no competing gradient at all. One point to note here is that two additional experts run on every single token regardless of routing [2]. These are called shared experts, and they hold the general-purpose processing that nearly every token needs. Thinking Machines states that the scores of the six selected routed experts and the two shared experts are normalised together before being used as weights [1], which means all eight are scaled to a common range and contribute proportionally. In other words, eight experts run per token in total. AttentionSparsity handles the cost of the feed-forward step. The attention step carries a separate cost that grows far faster, and Inkling manages it by having most layers examine very little. For every token, the model computes three sets of numbers:
Every token’s query is compared against every earlier token’s key. Strong matches produce high scores, and those scores determine how much of each earlier token’s value gets pulled into the current position. This is how the word “it” in a sentence ends up connected to the noun it refers to. Since every token compares itself against every earlier token, the number of comparisons grows with the square of the sequence length. For example:
Inkling supports a context window of one million tokens [2], and a trillion comparisons per layer across 66 layers is far beyond what any reasonable amount of hardware can deliver. This is where a sliding-window layer restricts each token to a fixed number of recent tokens rather than everything before it. If the window is 1,000, then token number 500,000 compares itself against tokens 499,000 through 500,000 and stops there. That is 1,000 comparisons instead of 500,000, and the total for the whole sequence grows in a straight line rather than as a square. This raises the obvious question that if every layer sees only a small window, how does information from page one of a long document ever reach page four hundred? Inkling alternates between sliding-window layers and full-attention layers at a ratio of 5:1 [1]. Integration notes published by the vLLM project put concrete numbers on the split, describing the 66 layers as 55 sliding-window layers and 11 full-attention layers [7]. Information from far away travels through those eleven full layers. For example, picture a fact at token 200 that matters for a prediction at token 900,000. In layer 6, the first full-attention layer, the representation at position 900,000 can reach back and pick up that fact directly. From there, it rides forward within the local representations, is refreshed at layer 12, again at layer 18, and so on. The long-range path exists on roughly one layer in six, and the other five handle nearby context at a fraction of the price. Inkling also uses 8 key-value heads [1]. Attention normally runs several times in parallel with different query, key, and value sets, and each parallel copy is called a head. Sharing a smaller number of key and value sets across many query heads reduces the memory needed while generating text. See the diagram below that tries to show this setup in a simplistic manner:
There is one practical consequence of this approach. Long-context models often handle the general content of a large document well, but can still miss one specific detail buried in the middle. PositionDealing with a million tokens brings up another question: how does the model represent where each token sits in the sequence? Thinking Machines picked an older technique over the current standard. This is due to the lengths the model never saw during training. The comparison between queries and keys involves no information about order. For example, both “the cat bit the man” and “the man bit the cat” contain the same five tokens, so without a position signal, the attention step produces the same comparisons for both. Something has to tell the model that “cat” came before “bit” in one case and after it in the other. Almost every recent open model uses Rotary Position Embedding, shortened to RoPE. Each token’s query and key are treated as points that get rotated by an angle proportional to that token’s position in the sequence. Token 1 gets a small rotation, token 500 gets a much larger one, and so on. The usefulness of this shows up when two tokens are compared. Since both were rotated by amounts tied to their own positions, the comparison between them depends on the difference between those two rotations, which tells how far apart they are. However, those rotation angles were only ever encountered at positions the model actually trained on. If training used sequences up to 32,000 tokens, then every angle the model learned to interpret came from that range. If we ask it about position 900,000, the angle involved falls outside anything it has experience with. A whole family of techniques exists specifically to stretch RoPE into ranges beyond its training data. Inkling uses a relative scheme in the style of Shaw and colleagues [1][10]. Rather than encoding where each token sits, this approach learns a value for each distance between two tokens and adds that value directly to the comparison score. For example, tokens at positions 5 and 9 are 4 apart. Tokens at positions 500,005 and 500,009 are also 4 apart. A relative scheme treats both pairs identically, because 4 is 4 wherever it occurs. Distances beyond some cutoff, say anything more than 128 apart, all share the same learned value, so a pair 900,000 tokens apart uses a value the model has seen countless times during training. Nothing has to be extrapolated. The vLLM notes describe the implementation as a learned relative-position term added to the attention scores before they are converted into weights [7]. Thinking Machines states that this performed better and extrapolated better to longer sequences than RoPE in their testing [1].
ConvolutionsInkling adds one small operation named convolution to handle a job that attention would otherwise have to learn from scratch. A convolution here means combining each position in the sequence with a few positions immediately before it. Inkling uses a window of four [7], so the numbers at position 100 get mixed with the numbers at positions 97, 98, and 99, using a small set of learned weights. Position 101 does the same with 98, 99, and 100. It is a cheap, fixed, strictly local operation.
To understand better, consider the two tokens that make up “New York.” The second token means something quite different on its own than it does following the first. Attention can learn to make that connection, and it has to learn it, because attention starts as a general comparison mechanism with no built-in preference for nearby tokens over distant ones. A convolution supplies that local mixing directly through its structure, without any training required to discover that neighbours matter. Thinking Machines places these convolutions at two kinds of locations:
The effect is that immediate context arrives pre-mixed, and attention can spend its capacity on the connections that genuinely require learning. MultimodalityEverything so far concerns text moving through the model. Images and audio have to get in first, and Inkling accepts both without a separately trained encoder standing in front of it. Most multimodal models attach three trained components to a language model:
However, in the case of Inkling, Sound arrives as a mel spectrogram, which is a standard way of representing audio as a grid of numbers. Frequency bands run down one axis, short slices of time run across the other, and each cell holds a loudness value. A three-second clip becomes a grid of a few hundred columns. The dMel method then rounds each of those loudness values to one of a fixed set of levels, in the same way you might round 0.73 to 0.7 [1][11]. That is the entire conversion. No separate audio model needs training beforehand, because rounding numbers requires no training. In the case of images, an image is cut into square patches measuring 40 by 40 pixels [1]. A 400 by 400 pixel image therefore becomes 100 patches. Each patch passes through a small four-stage network called an hMLP stem, which combines the pixels within that patch and processes each patch independently of every other one [1][12]. The paper describing this reports that it adds under one percent to compute compared with the simplest possible alternative [12]. Both then pass through a lightweight conversion layer and join the text tokens in a single sequence, processed by the same 66 layers we have been talking about [1]. Thinking Machines states that these multimodal components were trained from scratch on general-domain data [1], meaning they learned alongside the rest of the model rather than arriving pretrained. See the diagram below:
One point of confusion can be that the launch announcement labels this as an encoder-free architecture [1], while the model card describes images as being encoded via a hierarchical patch encoder [2]. Both descriptions are accurate. Encoder-free here means there is no large, separately pretrained encoder network, rather than meaning there is no processing at all. Moreover, this design predates Inkling. Thinking Machines described the same arrangement two months earlier for their real-time interaction system, using dMel for audio, 40 by 40 patches through an hMLP for images, and a lightweight conversion layer, with every component trained together from scratch [5]. EffortHow long the model reasons before answering is adjustable, and it was trained into the model rather than requested through wording. This impacts the model’s benchmark numbers Effort is a number between 0 and 1. The documented presets run from 0 for none, through 0.1 for minimal and 0.2 for low, then 0.7 for medium, 0.9 as the default, and 0.99 at the top [6]. The spacing between those values is uneven, which is a hint that the number represents a learned response rather than a token allowance. Mechanically, the setting arrives as text. Before the conversation begins, a system message stating the effort level is inserted ahead of everything else [6]. Reasoning models produce a stretch of working out before their final answer. This working-out phase costs tokens like everything else. Thinking Machines trained Inkling’s response to the effort setting during reinforcement learning, a training stage where the model produces complete attempts at tasks and receives a score for each attempt, then adjusts toward whatever scored well. During that stage, Thinking Machines varied the effort message across attempts while also adjusting the cost charged per token generated:
Across many attempts, the connection between the message and the profitable length was learned.
Here are some key points about limits:
Sweeping the effort setting across its range produces a curve of score against tokens generated. On the coding benchmark Terminal Bench 2.1, Inkling reaches the same score as NVIDIA’s Nemotron 3 Ultra while producing roughly a third as many tokens [1]. TradeoffsEach of the design decisions we have looked at has trade-offs. Let us look at some of them:
Thinking Machines states that other models available today, both open and closed, are stronger overall [1]. However, considering the availability for fine-tuning from day one, and the quantised checkpoint that brings the hardware requirements down, it is clear that Inkling has been built to be taken and adapted to the company’s use case. ConclusionLet us look at the key ideas we’ve encountered while understanding Inkling’s design and architectural decisions:
References
|
Post a Comment