Decision Model Jev

What is Jev?

Jev-style models can be understood as classification and decision models with general world knowledge. Given a state and a set of questions with fixed options, they answer “which option, and with what confidence?”

Many business workflows already have a small, fixed set of choices:

• Intent classification: identify a refund request, an exchange request, or an inquiry from predefined categories;
• Ticket routing: send a message to billing, delivery, or technical support;
• Risk grading: allow low-risk cases, review medium-risk cases, and block high-risk cases.

In the OpenRouter “Top models by task” screenshot preserved in the source repository, Classification accounts for 10.4% of all spend, the largest individual task within the General category (31.2%). These figures refer to that captured snapshot.

The screenshot also shows general-purpose models such as Claude Opus 5, Gemini 3.1 Pro Preview, and GPT-6 Astra serving classification workloads. For frequent decisions among fixed options, outputs can go directly into downstream programs: route by label, rank by score, or set human-review thresholds using calibrated probabilities.

Jev can serve the “fast thinking” part of a workflow, handling frequent, fixed, enumerable routing decisions. A general-purpose model can provide “slow thinking” for reasoning and planning in open-ended tasks. Both can work together within the same business process.

The following example shows how to train this kind of decision model with your own data and evaluate both accuracy and probability calibration.

This implementation

Code source and cost

• The complete implementation comes from KMnO4-zx/agentic-rl-lab/09-pytrio-jev. This page was checked against d2ba99c, and source links below are pinned to that revision.
• The source repository reports 2.72M training tokens and ¥12.36 for the 270-step run. Its cost screenshot also lists two evaluation sessions at ¥0.99 each, bringing those three sessions to ¥14.34. These are reference-run measurements; actual costs depend on input length, training scale, and evaluation frequency.
• See the SwanLab run for training curves. This project spans data compilation, templates, training, inference, and calibration. This page explains the key implementation; obtain all runnable files from the source repository.

This example uses Qwen/Qwen3.5-4B and PyTRIO to train a Jev-style classification and decision model with LoRA. Each input contains a shared state and a set of questions. The model returns candidate probabilities and a decision for each field, supporting three question types:

• choice: select one business option;
• noul: predict yes / no and return p_yes;
• score: return a distribution over levels, the most likely level, and an expected score.

Training uses cross_entropy: an assistant JSON skeleton contains placeholders, and supervision applies only to the answer symbol for each field. Inference uses the same skeleton, reads candidate scores from prompt logprobs, then normalizes them locally and maps them back to business labels.

Configuration

Project layout

Training and inference share the same data protocol. Every link below points directly to a source file on GitHub:

The Python excerpts below come from these modules and require the surrounding source project.

Environment and data

The local machine prepares data, renders templates, controls the workflow, and processes probabilities. PyTRIO runs model forward passes, backward passes, and LoRA updates remotely. Install the dependencies defined by the source repository's pyproject.toml and uv.lock:

git clone https://github.com/KMnO4-zx/agentic-rl-lab.git
cd agentic-rl-lab
git checkout d2ba99ce6964f0525c0c0ab25dcefed50ee50e6b
uv sync --frozen
uv run trio login
uv run swanlab login
cd 09-pytrio-jev

Run all subsequent commands from the example directory. Start with the data preparation script:

uv run python 00-prepare-data.py

The script uses LocalLLaMA/typed-decisions and Team-ACE/ToolACE and creates:

Typed Decisions is split by complete case within each workflow, using 240 / 30 / 30 records for train / dev / calibration. ToolACE records matching evaluation request text are excluded before sampling 1,200 training records. The preparation script does not pin upstream data revisions; preserve the downloaded JSONL files and data source versions for strict reproduction.

Canonical records contain state, questions, and training targets. This example keeps one field from the built-in support-ticket example and adds a training label following the data compilation protocol:

{
  "id": "ticket-1",
  "state": {"customer_message": "I was charged twice. Please refund the extra charge."},
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this ticket?",
      "criteria": {
        "billing": "Payments and refunds",
        "delivery": "Shipping and delivery",
        "technical": "Product technical support"
      }
    }
  },
  "targets": {"team": {"label": "billing"}}
}

Inference can omit targets. A record can contain multiple fields sharing the same state.

Core logic

1. Map business options to single-token symbols

question_options() and compile_record() use A-Z a-z 0-9 as the candidate symbol alphabet, supporting up to 62 options:

All three types predict the mapped symbols. When score uses a dictionary, its business keys are preserved and must be convertible to integers to compute an expected score.

Training labels are used only to build targets. The assistant input always retains a marker skeleton, constructed by compile_record():

skeleton = json.dumps(dict.fromkeys(fields, MARKER), ensure_ascii=False, indent=4)
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_text},
    {"role": "assistant", "content": skeleton},
]

Gold answers are never written into the input skeleton. Training and inference must preserve candidate order. Multiple fields share a sequence, but their order still changes the causal context; rearranging fields need not preserve their probabilities.

2. Locate decision positions with a shared template

Encoder.encode() reads the local chat template, renders system + user + assistant, and retains a fixed empty think block.

The marker decision comes from the model's existing vocabulary, and the Encoder checks that it is a single token. Marker detection scans only the assistant skeleton and skips matching tokens inside field names. For example, a ToolACE skeleton can contain "decision": "decision"; only the value token is a decision position.

The highlighted tokens show marker positions in the input. If a marker has zero-based index m, the shifted training arrays supervise position m - 1. Inference reads position m because prompt logprobs are indexed by the predicted token.

3. Apply cross-entropy only at decision positions

Encoder.build_sft_datum() retains the full input, substitutes the correct symbol into the shifted targets, and assigns nonzero weight only at those positions:

encoded = self.encode(record)
ids = encoded["ids"]
target_tokens = list(ids[1:])
weights = [0.0] * len(target_tokens)
for field, symbol in encoded["targets"].items():
    position = encoded["markers"][field] - 1
    target_tokens[position] = self.symbol_token_id(symbol)
    weights[position] = 1.0
return trio.Datum(
    model_input=trio.ModelInput.from_ints(ids[:-1]),
    loss_fn_inputs={
        "target_tokens": np.asarray(target_tokens, dtype=np.int64),
        "weights": np.asarray(weights, dtype=np.float32),
    },
)

This excerpt is a method body. model_input, target_tokens, and weights have identical lengths. JSON punctuation, field names, and all other context positions have weight 0. A record contributes one supervised token per decision field, while the remote model still processes the full input sequence.

4. Submit asynchronous training and log loss per supervised token

The training loop builds a Datum for each record and submits:

fwdbwd_future = await training_client.forward_backward_async(
    datums, "cross_entropy"
)
optim_future = await training_client.optim_step_async(adam_params)

A background logging task awaits the futures returned by both calls. Pending tasks are collected before saving weights and before training finishes. The logged loss = loss_sum / supervised_tokens uses the number of actual decision positions as its denominator.

Records can have different numbers of fields, so supervised_tokens varies even with a fixed batch size. The current loop drops an incomplete final batch; when using a small dataset, keep at least one batch of records.

5. Read candidate probabilities from prompt logprobs

candidate_logprobs() uses the same Encoder with target compilation disabled:

encoded = encoder.encode(record, include_targets=False)
response = sampling_client.sample(
    prompt=trio.ModelInput.from_ints(encoded["ids"]),
    num_samples=1,
    sampling_params=trio.SamplingParams(max_tokens=1),
    include_prompt_logprobs=True,
    topk_prompt_logprobs=max(len(opts) for opts in encoded["options"].values()),
).result()

One request retrieves prompt logprobs for all fields. The extra generated token is not used for decisions. The candidate extraction function first checks that local and remote input lengths agree, then reads response.topk_prompt_logprobs[m].

softmax() and the business output mapping then normalize over the current field's candidates:

$p(c \mid x) = \frac{\exp(\ell_c / T)}{\sum_{j \in C}\exp(\ell_j / T)}$

Here, T is a local calibration temperature. The request returns top-k over the full vocabulary, so setting K to the number of candidates does not guarantee that every candidate is returned. The implementation approximates missing candidates with min(topk.values()) - 10. Candidate distributions therefore include a top-k truncation approximation, and normalized probabilities still require calibration evaluation.

Run training

Use the training entry point with 64 records to check data processing, marker alignment, and weight saving:

uv run python train.py \
    --max-samples 64 \
    --epochs 1 \
    --batch-size 16 \
    --save-every 0 \
    --swanlab-mode offline

The reference run uses:

uv run python train.py \
    --epochs 2 \
    --batch-size 16 \
    --save-every 60 \
    --swanlab-mode online

With 2,160 records, each epoch contains 135 steps, giving 270 steps in total. --save-every 0 only disables step-based saving; weights are still saved at the end of each epoch.

save_weights() saves and prints a trio://... sampler weights path for inference, calibration, and evaluation. The current script does not save a training state containing optimizer state.

Inference and temperature calibration

Replace the variable below with the weights path printed by your training run. The inference entry point runs the built-in support-ticket example with three decision fields:

JEV_WEIGHTS='trio://YOUR_STEP_270_SAMPLER_WEIGHTS'
uv run python inference.py --weights "$JEV_WEIGHTS"

Omit --weights to use the Base Model. Use --input path/to/records.jsonl --limit 3 to read custom records.

Next, use calibrate.py to fit temperature on the separate calibration split:

uv run python calibrate.py \
    --weights "$JEV_WEIGHTS" \
    --output outputs/calibration.json

The implementation performs golden-section search over log T to minimize candidate-distribution NLL. The reference run reports T ≈ 1.1319, reducing calibration-set NLL from 0.4990 to 0.4959. Fit and use your own temperature when reproducing the run.

The calibration file records temperature, the weights path, and NLL. Before evaluation, check that its weights path matches the checkpoint being evaluated. The current evaluator reads only the temperature and does not automatically validate the weights match.

For a fixed set of candidate scores, positive temperature scaling preserves argmax. Calibration adjusts confidence; evaluate Accuracy, ECE, and Brier together.

Evaluate five benchmark suites

After fixing the checkpoint and temperature, use eval.py to evaluate the Base Model, the uncalibrated checkpoint, and the calibrated checkpoint:

uv run python eval.py --tag base

uv run python eval.py \
    --weights "$JEV_WEIGHTS" \
    --tag step-270

uv run python eval.py \
    --weights "$JEV_WEIGHTS" \
    --calibration outputs/calibration.json \
    --tag step-270-cal

By default, the script evaluates all five suites and writes outputs/eval-<tag>.json. Use --suite typed_decisions to run a single suite. Finally, summarize with analysis.py:

uv run python analysis.py

The following accuracies come from the source repository's reference experiment:

The 2,000 Typed Decisions test decisions come from 400 records. The mean gives each suite equal weight, regardless of its decision count.

The uncalibrated and calibrated results were collected through separate inference requests. Their accuracy differences cannot be attributed directly to temperature scaling. The reference Hard ECE / Brier changed from 0.2311 / 0.6510 to 0.1884 / 0.6331; interpret this comparison with the separate requests in mind as well.

Metrics to monitor

A lower training loss indicates a better fit to the supervised objective. In the reference experiment, Typed Decisions and ToolACE improve while Jevbench-Hard accuracy drops. Retain difficult cases and a separate test set when adapting the project to business data.

Reproduction boundaries

• This example implements a Jev-style fixed-candidate decision workflow using Qwen3.5-4B, LoRA, and PyTRIO. Results come from one reference training run.
• Typed Decisions uses a specialist setup with the same workflows represented in train and test. This differs from other models' zero-shot evaluation conditions.
• A dev split is available, but the current scripts do not automatically select the best checkpoint. Temperature is fitted only on calibration data.
• Candidate symbols must be single tokens, with at most 62 candidates. Recheck mappings, marker positions, and local/remote input lengths when changing the tokenizer or template.
• Fixed-option classification does not cover arbitrary open-ended question answering. Validate your own data distribution, probability calibration, and decision thresholds before deploying it in a business workflow.

See the source repository README for the complete implementation and reference-run details.