What Is Together Compiler? The Beginner's Guide to AI That Retunes Its Own Inference

CodexFuse Team · 2026-11-05T11:03:46.127Z · 22 min read min read

What Is Together Compiler? The Beginner's Guide to AI That Retunes Its Own Inference

Together Compiler is Together AI's agent-driven system that keeps re-optimizing AI inference while the service is running, and this guide answers the question what is Together Compiler in plain terms. Together AI announced it on November 4, 2026, one day before this guide was written, as the first project of a larger initiative called Polymorphic Inference. Instead of tuning an inference engine once and freezing it, Together Compiler sends a swarm of coding agents to rewrite the model's forward pass, measures each candidate, ships the winner, and starts over whenever traffic changes shape.

The idea matters because inference is where most AI spending now happens. Every time you ask a chatbot a question, an inference engine runs the model to produce the answer, and the speed of that run decides both the cost per answer and how long you wait. This guide explains what Together Compiler does, how its loop works, what its speed claims actually mean, and how to get access while it is still in limited beta.

What Is Together Compiler, Exactly?

Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model's forward pass. A forward pass is one trip through the model when it processes a request, and the code that performs it decides how fast each answer comes back. Together Compiler takes a server launch command, a traffic profile, and a hardware configuration as its inputs, then lets agents propose faster versions of that code while a persistent engine measures every candidate.

The winning code ships into the serving engine, and when the traffic profile drifts, the loop runs again.

In short, Together Compiler explained in plain terms: it is a pit crew for a running AI service. A traditional compiler tunes the engine once before the race, while Together Compiler keeps a crew of agents working on the car while it is still on the track. The system is already running in production across many of Together AI's most popular serverless endpoints, according to the company, and it serves trillions of tokens per day across the fleet.

If you run open models yourself, you will know the other half of the job: getting the model file onto your machine safely. Before loading a downloaded checkpoint, compare its hash against the publisher's value, and the SHA-256 hash generator can produce the hash you need to compare. For a machine to run it on, VPS hosting gives you a server you can rebuild in minutes if an experiment goes wrong.

One honest note up front: every speed figure in this guide is vendor-reported by Together AI, with no independent validation published as of November 5, 2026. The guide explains the idea and the evidence, not a tool most readers can try today, because Together Compiler is in limited beta and self-serve access is still coming.

What Is Polymorphic Inference?

Polymorphic Inference is Together AI's name for inference that changes shape as models, traffic, and hardware change. The word comes from Greek for "many shapes," and it describes the company's answer to a fast-moving field. The old era, which Together AI calls Static Inference, used fixed engine configurations, one model, and one hardware platform.

The new era assumes nothing stays still.

Three trends drive the change, according to the launch post. First, new models arrive constantly: Together AI reports that 80% of traffic on OpenRouter goes to models released in the last three months, and all of the top ten most popular open-source models in use today use different architectures. Second, traffic itself shifts hour by hour, which the next section covers in detail.

Third, hardware keeps changing, with new GPUs and accelerators arriving every year.

Together Compiler is the first project inside this Polymorphic Inference initiative. It is the practical half of the vision: while Polymorphic Inference describes the goal, Continuous Compilation is the mechanism that gets there, and Together Compiler is the system that runs it. The rest of this guide follows that mechanism step by step.

What Is Together Compiler Continuous Compilation?

Continuous Compilation is Together AI's name for the fix to a simple problem: a configuration tuned once goes stale. A typical vLLM, SGLang, or InferenceX setup launches with a static, hyperparameter-tuned engine configuration, and a configuration optimized for one setting is not necessarily best for another. When the traffic changes, the tuning that was perfect last week quietly becomes the wrong answer.

Together AI's own production traffic shows how fast that drift happens. The company reports three patterns from its fleet, and each one breaks a fixed configuration in a different way.

Those numbers are large. The 97K versus 70K gap is about 39% more context in one region than another. The burst from 18K to 33K is an 83% jump in three days, and the gradual shift from 10K to 18K is an 80% rise over two weeks.

A kernel tuned for short prompts wastes time on long ones, and a layout built for one region misfires in another.

Random test data makes the problem worse. Together AI reports that kernels tuned on random inputs activated 13% to 38% more mixture-of-experts (MoE) experts than real traffic did, which means the tuning optimized for a workload that never actually arrives. In one striking case, an agent-written top-k kernel took 20 microseconds on random data but up to 62 milliseconds on real data at 70K context, which is 3,100 times slower.

Continuous Compilation answers this by tuning against the real traffic profile, not a synthetic guess.

The company also learned that giant fused megakernels are not the universal answer. After putting its first megakernels into production, Together AI concluded that megakernel fusion "is actually not the right design goal for modern Polymorphic Inference," because MoE architectures and dynamic workloads larger than batch size 1 need different techniques. The fix is not a bigger kernel, but a loop that keeps finding the right technique for the current traffic.

How Does Together Compiler Work? The 5-Step Loop

Together Compiler reframes compiler optimization as an agentic search problem. Instead of a human specialist hand-writing kernels for one fixed setup, a swarm of agents searches for faster forward passes, and a fast engine judges every candidate on real traffic. The loop has five steps, and it repeats for as long as the service runs.

1. Feed it the setup. The loop starts with three inputs: the server launch command, a traffic profile describing real requests, and the hardware configuration it will run on.
2. Pick a starting layout. A cost model ranks tensor, expert, data, and context parallel layouts before the agents start, mixing spec-sheet numbers, microbenchmarks, and measured forward passes. On Kimi K3, it found a production layout with up to 24% higher tokens per second (TPS), according to Together AI.
3. Let the agents rewrite the forward pass. A swarm of parallel agents proposes faster versions of the forward pass. They communicate through a message board and keep experiments queued while other agents reason or use tools, so the search never idles waiting on one agent.
4. Measure every candidate without restarting. A Tiered Persistent Engine hot-reloads each candidate and measures it on real traffic distributions, then keeps the winners and drops the rest.
5. Ship the winner and watch for drift. The best configuration and forward-pass source are integrated directly into the serving engine. When the traffic profile shifts, the loop recompiles.

The agent half of the loop will feel familiar to anyone who has used a coding agent. The agents read raw engine source code, write new forward-pass code, and learn from the measured results of their own experiments. If you want to see how that kind of agent works from the user's side, read how to use Claude Code, which covers the same loop of propose, run, measure, and revise at a smaller scale.

Agents also need a way to talk to the tools around them, and what MCP over MOQT is explains the protocol family that connects agents to external systems.

Before a winning candidate can serve real users, it passes verification tests for KV-cache contamination and output quality. Those checks run after the optimization budget is exhausted, when the final source code is prepared for integration into the serving engine. A faster forward pass that corrupts cached results or changes answers is not a win, so verification is a gate, not an afterthought.

What Is the Tiered Persistent Engine?

The Tiered Persistent Engine is the part of Together Compiler that makes the agent loop affordable. Most agent optimization loops restart the whole inference engine, such as vLLM, SGLang, or TensorRT-LLM, for every experiment. Together AI measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM, which means every experiment would begin with more than twelve minutes of waiting before a single measurement.

The persistent engine starts once and never restarts during optimization. Agents submit a candidate forward pass over RPC, and the server swaps the forward function pointer and re-captures the CUDA graphs. The launch post's startup comparison puts the status-quo sequence at 467 seconds to serving versus 9.0 seconds with Together Compiler's setup, which is about 52 times faster.

Those figures cover engine startup during optimization, not individual request latency, so they should never be read as requests getting 52 times faster.

The two-tier design is the reason the loop can test so many candidates. The fast tier screens cheaply, the slow tier confirms carefully, and together they tested 8.1 times more candidates than a single slow tier would have managed. The headline figure for the mechanism is 252 times higher agent experiment throughput than an engine-restart baseline, according to Together AI, and that throughput is what lets a swarm of agents explore widely instead of testing a handful of guesses.

How Fast Is Together Compiler, Really?

The headline claim is that Together Compiler delivers up to 2.6 times higher TPS than the best published InferenceX configurations, measured on SemiAnalysis' InferenceX AgentX workload, in under 24 hours per setting, using at most 2 nodes per model. That figure is vendor-reported by Together AI, and it is the best case across the tested settings, not the typical one. Across all settings, the reported gains run from about 4% to 2.6 times, so the honest reading is a wide range with a strong ceiling.

The batch-size-1 P90 TPS table below shows the per-model results Together AI published. P90 means the 90th percentile of per-user throughput, so it reflects what most users actually experience rather than a single best run.

Read the table from the bottom up and the picture changes. The weakest reported case, GLM 5.3 on AMD's MI355X, moved only from 463 to 480 TPS, which is about a 4% gain. The strongest, MiniMax M3.1 Flash on a B200, jumped from 633 to 1,111 TPS, which is about a 76% gain.

Most settings sit between those extremes, and the 2.6 times headline comes from a different comparison, the best result across the full InferenceX workload, rather than any single row of this table.

Other vendor-reported figures fill out the range. Together AI reports 1.3 times to 3 times higher TPS versus highly optimized open-source baselines across Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash, which is a wider baseline than the InferenceX-specific 2.6 times figure, so the two ranges should not be mixed into one claim. On the small end, Llama 1B ran at over 2,100 TPS and up to 1.7 times the speed of hand-written megakernels, including Together AI's own ThunderKittens and Mirage, and was fastest at every output length tested on NVIDIA H100 and B200 hardware.

The models tested were Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B. The hardware tested was NVIDIA B200, B300, GB300, and H100, plus AMD MI355X and AWS Trainium2, with each run using at most 2 nodes per model. The baselines were the leading open configurations as of October 30, 2026.

One disclosed exception matters: the Trainium2 runs used a synthetic multi-turn dataset, because the implementation did not yet support higher concurrencies within the available memory, so those numbers should not be read as production-traffic results.

Two cautions belong with every number above. First, no independent party had replicated these results as of November 5, 2026, and at least one news outlet, oossa.com, explicitly flagged that the results are company-reported with no independent validation. Second, Together AI also reports 1.5 times to 6.4 times speedups on 8 scientific models covering weather, biology, time-series, and partial differential equations, with bit-exact outputs, which is a different workload family from chatbot inference and should not be quoted as chatbot speed.

How Does Together Compiler Compare With Other Agentic Optimizers?

Together Compiler is not the only system that uses agents to optimize GPU code, but the paper positions it as the only one that optimizes a full forward pass inside a live serving engine. The comparison below comes from the paper, as reported by MarkTechPost, and it shows how different the scopes are.

The contrast is scope, not just speed. Cursor plus NVIDIA's system and AMD's GEAK optimize individual kernels, which is like tuning one instrument, while Together Compiler optimizes the whole forward pass inside the engine, which is like tuning the orchestra. Apple's RSI-Inference Bench is a benchmark that scores agents rather than a system that ships optimizations.

GEAK is open source and Apple's bench is available under the Apple Sample Code License, while Together Compiler itself has no public code release, so readers cannot download it and try it the way they could with GEAK.

The hardware range also differs. Together Compiler was tested on NVIDIA B200, B300, GB300, and H100, AMD MI355X, and AWS Trainium2. The Cursor plus NVIDIA system ran on B200 only, GEAK on AMD Instinct MI300X, and the Apple bench on H100 and B200.

A system tested on six hardware platforms faces a harder generalization test than one tuned for a single GPU, which is part of why the Polymorphic Inference framing matters.

What Did the Agents Learn? The Bitter Lessons

The most surprising finding in the paper is what Together AI calls its bitter lessons. The team expected that giving agents expert-curated strategies would beat letting them explore freely. The opposite happened.

Naive agents told simply to make the forward pass "as fast as possible" beat the expert-curated strategies, and an expert-curated knowledge base produced 25% to 31% lower speedup after 12 hours. After 10 trials, ensemble scores were 6.2 times for the naive agents versus 4.9 times for the expert-guided ones, which is about 1.27 times better for the naive approach.

The lesson changed how the system feeds agents. Together AI now gives agents raw engine source code, not an intermediate representation, because the agents found optimizations in the real code that the curated abstraction hid. That choice echoes a pattern in agent research: the model's own exploration can beat a human's pre-built map when the search space is large and poorly understood, provided the measurement loop is honest.

Honest measurement is the key qualifier. The agents only learned because the persistent engine measured each candidate on real traffic distributions, with verification gates before shipping. Without that measurement, naive exploration would be guessing.

The bitter lesson is not that expertise is useless, but that expert guidance can narrow the search too early when the terrain itself is still being mapped.

Who Is Already Using Together Compiler?

The system is already running in production on many of Together AI's top serverless endpoints, according to the launch post, so readers using Together AI's hosted models may already be benefiting from it without knowing. The company says it serves trillions of tokens per day across its fleet, which is a vendor claim about the whole fleet rather than a measured property of Together Compiler alone, but it frames the scale at which the loop operates.

The named customer proof point is Phonic, a voice AI company. Its co-founder Moin Nadeem is quoted in the press release as saying, "Running Together Compiler cut our median time to first speech by nearly 50%, and it happened overnight." That figure is a vendor-supplied customer quote, not an independent measurement, and it describes a median time to first speech, which is a voice-product metric, not a general inference speed claim. It is still the clearest real-world result published so far: a voice product got noticeably faster after one overnight optimization run.

For a voice product, time to first speech is the delay before the user hears anything, so a near-50% cut is something users would feel directly. For a text chatbot, the equivalent would be time to first token. The Phonic result suggests the loop's value shows up fastest where latency is user-facing, which matches the design: the loop optimizes the exact traffic profile a customer actually serves, not a generic benchmark.

How Do You Get Access to Together Compiler?

As of November 5, 2026, Together Compiler is available to end customers in limited beta, and the launch post's invitation is to "reach out if you want early access." There is no self-serve signup today. The company says that soon, customers will be able to use Together Compiler to optimize their dedicated endpoints for their unique workloads at the click of a button, and the press release adds that customers can join the limited beta to use it on their own dedicated deployments, with self-serve access coming soon.

That means the honest answer for most readers is: not yet, unless you are already a Together AI customer who can ask for beta access. There is no pricing published anywhere in the sources fetched for this guide, so no price is stated or estimated here. There is also no public code repository for Together Compiler itself, so it cannot be downloaded or self-hosted the way an open-source tool could be.

While you wait, the safe way to experiment with agent-written code is a sandbox, where a bad experiment cannot touch your real system. If you want to try that pattern, read how to use E2B, which covers running code in isolated sandboxes. The same discipline applies inside Together Compiler: the persistent engine measures candidates in isolation, and verification gates keep a bad candidate out of production.

Is Together Compiler the Same as together.compile?

No. together.compile is a different product with a similar name, and the two should never be confused. together.compile is an extension of ThunderKittens that generates an optimized kernel stack once at startup with a single function call, with no changes to model code required.

Together Compiler is a continual, agent-driven loop that keeps re-optimizing a live inference deployment as traffic changes. In short: together.compile optimizes kernels once at startup, while Together Compiler keeps retuning the running service.

The together.compile results are still worth knowing as context. Applied to Hedra's Omnia video model, it accelerated generation of 200 frames by 25%. In production Flux Kontext benchmarks, server startup plus generating 51 images across 17 resolutions completed in 329 seconds with together.compile, versus 558 seconds with torch.compile, which is a 41% improvement.

Those are startup-time results for a different product, and they belong in this section only so readers who saw the similar name can tell the two apart.

What Comes Next for Together AI Inference Optimization?

The launch post teases a research roadmap that extends the same idea in several directions. First is self-serve Continuous Compilation, which would let customers optimize their dedicated endpoints at the click of a button. Second is shape-shifting speculative decoding, which would adapt the decoding strategy as traffic changes.

Third is the ThunderAgent Router, which would route work across heterogeneous deployments. Fourth is Together Post-Training, and fifth is TogetherLink, described as workload-aware model routing.

Each item on that list follows the same logic: stop freezing decisions and keep adapting them. Routing, decoding, training, and model selection are all choices that today get made once and then go stale, just like kernel tuning did. If Together Compiler proves the loop works for the forward pass, the natural next step is to apply the same loop to every decision around it.

The broader claim in the launch post is larger still. Together AI describes Together Compiler as the first step toward what it calls recursive self-improving inference, in which AI systems optimize their own infrastructure. That is the vendor's own characterization, not an established fact, and it should be read as ambition rather than achievement.

The concrete achievement so far is narrower and more useful: a loop that measures real traffic, lets agents propose faster code, verifies the winners, and ships them without downtime.

For readers, the practical takeaway is simple. Inference speed is no longer a fixed property of a model, and it is no longer set once at launch. It is a moving target that depends on the model, the traffic, and the hardware, and systems like Together Compiler exist because the target moves faster than human specialists can retune it by hand.

As of November 5, 2026, the only way to watch that idea play out is from the outside, since the beta is limited and the results are vendor-reported. The guide will age quickly, and the facts above carry that as-of date for exactly that reason.

Sources

Every statistic and fact in this guide comes from a source fetched on November 5, 2026. The guide covers continuous compilation AI inference as Together AI defines it. Benchmark figures are vendor-reported by Together AI, with no independent validation published as of that date.

• Together AI, Polymorphic Inference: self-tuning inference for open models, published November 4, 2026.
• Together AI, Together Compiler: Continual Compilation for AI Inference (research paper), dated November 4, 2026.
• MarkTechPost, Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster, published November 5, 2026.
• Business Wire, Together AI Launches Together Compiler for Continual Optimization of Open Source AI Models, published November 4, 2026.
• Unite.AI, Together AI Brings Continuous Compilation to Production Inference, November 4-5, 2026.
• Together AI, Key research and product announcements at the AI Native Conf, published March 5, 2026, updated September 10, 2026.
• oossa.com, Together AI says its compiler speeds up AI inference, November 4, 2026.
• TokenPost, Together AI Compiler Cuts AI Engine Setup Time to 9 Seconds, November 5, 2026.

Frequently Asked Questions

What is Together Compiler in simple terms?

Together Compiler is Together AI's system that keeps making AI inference faster while the service is running. Coding agents rewrite the model's forward pass, a persistent engine measures each version on real traffic, the best version ships into the live service, and the loop repeats whenever traffic changes. It was announced on November 4, 2026, and it is in limited beta as of November 5, 2026.

How much faster does Together Compiler make inference?

Together AI reports up to 2.6 times higher tokens per second than the best published InferenceX configurations, but that is the best case, not the typical one. Across the published settings, gains run from about 4% to 2.6 times. For example, MiniMax M3.1 Flash on an NVIDIA B200 went from 633 to 1,111 TPS, about a 76% gain, while GLM 5.3 on AMD MI355X went from 463 to 480 TPS, about a 4% gain.

All figures are vendor-reported, with no independent validation as of November 5, 2026.

Is Together Compiler the same as together.compile?

No. together.compile, announced at the AI Native Conf in March 2026, generates an optimized kernel stack once at startup with a single function call. Together Compiler, announced November 4, 2026, is a continual agent-driven loop that keeps re-optimizing a live inference deployment as traffic changes.

The names are similar, but the products solve different problems.

Can I use Together Compiler today?

Not through self-serve. As of November 5, 2026, Together Compiler is in limited beta for Together AI customers, and the company asks interested users to reach out for early access. Self-serve access for dedicated endpoints is coming soon, according to the vendor, and no pricing has been published.

Is Together Compiler open source?

No. Together AI has not published a public code repository for Together Compiler, and none of the sources fetched for this guide describe it as open source. Related systems in the same space, such as AMD's GEAK, are open source, but Together Compiler itself is a hosted product inside Together AI's cloud.