Prefill, Cache, Sample, and Train

Clock cycles explain when a request is scheduled; this page explains where the remote computation of a request actually goes. In a typical "sample + train" loop, remote GPU time splits into three phases — prefill, sample, and train — and prefilling cache is the server-side reuse optimization for prefill. Understanding these four concepts helps you estimate job cost and locate performance bottlenecks.

Like clock cycles, these concepts describe server-side computation behavior rather than any specific SDK parameter. Refer to the model list for exact billing rules.

Four kinds of computation on one pipeline

A typical agentic RL iteration contains the following stages:

1. Prefill: the model processes the entire prompt and builds the KV cache in preparation for the first generated token.
2. Sample: tokens are decoded one by one until the completion is finished.
3. The local program assembles trajectories into trio.Datum objects and submits training requests.
4. Train: forward_backward runs the forward and backward passes to accumulate gradients, and optim_step applies the optimizer update.

Prefilling cache mainly appears in the prefill stage: when the prefix of a new sampling request's prompt has already been computed, the shared part is not computed again.

Prefill: processing the prompt

Before the first token can be generated, the model must process the whole prompt once, building KV cache entries for every position. The cost of this phase grows with prompt length and is independent of how many tokens are generated afterward.

In single-turn conversations, prefill is usually a small fraction of the total. In multi-turn agentic trajectories, however, every turn's prompt contains the full history, so the same prefix is prefilled again and again. The longer the trajectory and the more turns it has, the more repeated prefill dominates the cost — when budgeting long-trajectory jobs, counting only the tokens that enter the loss is not enough.

Prefilling cache: reusing prefix KV

The server retains KV for prompt prefixes it has already computed. When a new request's prompt exactly matches a cached prefix, the shared part skips prefill and computation resumes from the first uncached position, reducing time to first token and repeated computation.

A hit requires an exact token-level prefix match: if even one token differs at the beginning, identical content further along cannot hit. The following practices all serve the same goal — letting more requests share one cacheable prefix:

• Build each turn's prompt by appending to the real tokens returned by the sampler. Do not re-tokenize history text and concatenate it, which can encode the same text into a different token sequence.
• Put fixed content such as the system prompt and tool definitions at the very beginning of the prompt, and varying content after it.
• Multiple samples in the same group (for example, same-question candidates in GRPO) share the same prompt prefix, so one prefill covers the whole group.

Prefilling cache only takes effect during sampling, and is managed automatically by the server — it does not need to be enabled in code. You only need to construct prefixes in a cache-friendly way. Two cases bypass the cache and prefill the prompt in full:

• A sampling request with include_prompt_logprobs=True: the server must compute a logprob for every prompt position, so the cached prefix KV cannot be reused;
• compute_logprobs requests: the whole sequence is recomputed and never hits the cache.

Sample: generating token by token

After prefill, decoding begins: the model generates one token per step and appends it to the KV cache until a stop condition or the length limit is reached. The cost of this phase depends on the number of completion tokens and the number of samples — for example, the group size per prompt in GRPO directly multiplies sample cost.

See Sampling for the sampling APIs.

Train: forward, backward, and the optimizer update

The training phase consists of forward_backward and optim_step: the former runs forward and backward passes over the full sequence to accumulate gradients, and the latter applies one optimizer update.

When estimating training cost, distinguish two kinds of tokens:

• Tokens that enter the loss: positions with non-zero weights, typically the assistant-generated part;
• Tokens that are actually computed: the entire sequence. Whether or not a position counts toward the loss, the forward and backward passes pay for it.

Budget training by the latter. For how training requests are scheduled on the shared worker pool and how to fill cycles with a pipeline, see Clock cycles; for the training APIs, see Training.

Estimate cost by phase

When reporting the cost of a training job, list prefill, sample, and train token usage separately instead of counting only the assistant tokens that enter the loss — in long-trajectory jobs, prefill of repeated prefixes and decoding of multiple samples are often the dominant costs.

For the submission timing of async requests, continue with the async guide.