pytrio.SamplingClient

class SamplingClient:
    def __init__(
        self,
        task_id: str,
        base_model: str
    ):

SamplingClient is used for generation and inference. Create it with ServiceClient.create_sampling_client().

import pytrio as trio

client = trio.ServiceClient()
sampling_client = client.create_sampling_client(base_model="Qwen/Qwen3.5-4B")

tokenizer = sampling_client.get_tokenizer()
prompt_ids = tokenizer.encode("Hello, world!")

future = sampling_client.sample(
    prompt=trio.ModelInput.from_ints(prompt_ids),
    num_samples=4,
    sampling_params=trio.SamplingParams(temperature=1.0, max_tokens=128),
)
response = future.result()

Properties

Property	Type	Description
`task_id`	`str`	Current sampling task ID
`model_id`	`str`	Canonical model ID

Methods

sample

def sample(
    self,
    prompt: ModelInput,
    num_samples: int = 1,
    sampling_params: SamplingParams = SamplingParams(),
    include_prompt_logprobs: bool = False,
    topk_prompt_logprobs: int = 0,
    return_text: bool = True,
) -> APIFuture[SampleResponse]

Generate completions from an input prompt.

Parameter	Type	Default	Description
`prompt`	`ModelInput`	-	Input token IDs
`num_samples`	`int`	`1`	Number of generated samples
`sampling_params`	`SamplingParams`	`SamplingParams()`	Sampling parameters. See `SamplingParams`
`include_prompt_logprobs`	`bool`	`False`	Include prompt logprobs in the response
`topk_prompt_logprobs`	`int`	`0`	Number of top-k prompt logprobs to return. `0` disables it
`return_text`	`bool`	`True`	Include generated text in the response

Returns

APIFuture[SampleResponse]. Call .result() to get the generated result. The response contains:

sequences: Generated sequences, including fields such as stop_reason, text, tokens, and logprobs.
prompt_logprobs: Logprobs for each prompt token.
topk_prompt_logprobs: Top-k logprobs for each prompt token.
output_tokens: Number of generated tokens.

future = sampling_client.sample(
    prompt=prompt,
    num_samples=4,
    sampling_params=trio.SamplingParams(temperature=1.0, max_tokens=128),
)
response = future.result()
print(response.sequences)

compute_logprobs

def compute_logprobs(self, prompt: ModelInput) -> APIFuture[list[float | None]]

Compute logprobs for each token in a prompt.

Parameter	Type	Description
`prompt`	`ModelInput`	Input token ID list

Returns APIFuture[list[float | None]]. Call .result() to get the logprob list.

logprobs = sampling_client.compute_logprobs(prompt=prompt).result()

get_tokenizer

def get_tokenizer(self)

Get the tokenizer for the current base model. It uses AutoTokenizer from transformers / modelscope.

tokenizer = sampling_client.get_tokenizer()
prompt = tokenizer.encode("The meaning of life is")

Async Methods

sample_async

async def sample_async(
    self,
    prompt: ModelInput,
    num_samples: int = 1,
    sampling_params: SamplingParams = SamplingParams(),
    include_prompt_logprobs: bool = False,
    topk_prompt_logprobs: int = 0,
    return_text: bool = True,
) -> SampleResponse

Async version of sample. Parameters are the same. Unlike synchronous sample(), sample_async() returns SampleResponse directly after await; do not call .result() on the awaited value.

Returns

SampleResponse. The response contains:

sequences: Generated sequences, including fields such as stop_reason, text, tokens, and logprobs.
prompt_logprobs: Logprobs for each prompt token.
topk_prompt_logprobs: Top-k logprobs for each prompt token.
output_tokens: Number of generated tokens.

response = await sampling_client.sample_async(
    prompt=prompt,
    num_samples=4,
    sampling_params=trio.SamplingParams(temperature=1.0, max_tokens=128),
)
print(response.sequences)

compute_logprobs_async

async def compute_logprobs_async(self, prompt: ModelInput) -> list[float | None]

Async version of compute_logprobs. Parameters are the same, and await returns the logprob list directly.

logprobs = await sampling_client.compute_logprobs_async(prompt=prompt)

pytrio.SamplingClient

On this page