Fixing the confidence: temperature scaling

Post 4 found the model's confidence too modest by about 0.1 in the middle. Answers with a confidence around 0.5 were right 60% of the time. Answers around 0.7 were right 80% of the time. Over the whole BANKING77 test set, the expected calibration error (ECE) was 0.041.

This post leaves the model and its answers alone. It fixes only the confidence, which is too low in every bin from 0.4 up. The usual first correction is temperature scaling. It fits one number on the calibration set for each of post 3's two steps (the group step and the intent step). Each number makes its step's probabilities sharper or flatter.

What a temperature does

You may know temperature from sampling, where the model's logits are divided by T before the softmax. Temperature scaling applies the same idea after the answer is generated. It starts from the probabilities post 3 recorded at each step.

In terms of probabilities, a temperature T changes each p into p^(1/T) and then divides by the new total so they add up to 1 again. how_it_works.py shows what that does to the top option of a few distributions:

   raw                  T = 0.5   T = 0.8   T = 1.0   T = 1.5
   0.60 / 0.30 / 0.10     0.783     0.655     0.600     0.517
   0.80 / 0.15 / 0.05     0.962     0.866     0.800     0.673
   0.95 / 0.04 / 0.01     0.998     0.978     0.950     0.855
   0.80 / 0.20            0.941     0.850     0.800     0.716

• T below 1 sharpens the distribution. The exponent 1/T is then above 1, which shrinks small probabilities much more than large ones. After renormalizing, the top option gains (row 1 goes from 0.60 to 0.783 at T = 0.5).
• T above 1 flattens it. The exponent is below 1, which closes the gaps between options. The top option loses as the others catch up (row 1 drops to 0.517 at T = 1.5).
• T = 1 changes nothing.

The order of the options never changes because p^(1/T) grows with p for any T > 0. So no answer changes and neither does the accuracy. Only the confidence moves.

temperature.py does this in the scale function:

EPS = 1e-6  # a letter missing from top_logprobs gets this probability rather than 0

def scale(probs, letters, t):
    """probs over `letters`, rescaled by temperature t and renormalized."""
    logs = {k: math.log(max(probs.get(k, 0.0), EPS)) / t for k in letters}
    top = max(logs.values())  # subtract the largest before exp(), so the total can't underflow to 0
    weights = {k: math.exp(v - top) for k, v in logs.items()}
    total = sum(weights.values())
    return {k: w / total for k, w in weights.items()}

The code works with logs, since p^(1/T) = exp(log(p) / T). That keeps it safe for any T. Subtracting the largest log before exp() gives the top option exp(0) = 1, so the total can't underflow to 0 when T is small.

The log form also links back to sampling. Dividing each log(p) by T and renormalizing is the sampling formula softmax(logits / T) with log(p) in place of the logits. Each log(p) is its logit minus a constant shared by all the options, and the renormalizing cancels that constant. So the result is exactly what dividing the logits by T would give.

Choosing T

Which value of T makes the model's confidence honest? We fit T on the 770 answers of the calibration set (post 4) and keep the test set for measuring the result.

The intuitive choice is to pick the T with the lowest ECE. But ECE makes a poor target for a search. It depends on how the answers are binned, and it jumps whenever an answer's confidence crosses a bin edge. So its lowest point can move when the bins change.

The standard target is the negative log-likelihood (NLL). For each calibration answer, NLL takes the negative log of the probability the model gave the right option. Then it adds these up. Putting 0.95 on the right option costs only 0.05. Putting 0.10 on it costs 2.3.

def nll(rows, t):
    """rows: (probabilities, the allowed letters, the right letter). Lower is better."""
    return -sum(math.log(scale(p, letters, t)[gold]) for p, letters, gold in rows)

Why does NLL find the honest T? Lowering T makes every answer more confident, whether right or wrong. On a right answer, that lowers the loss; on a wrong one, it usually raises it, and confident mistakes cost a lot. Putting 0.999 on a wrong option leaves at most 0.001 for the right one, which costs at least 6.9. If all the answers in a group get the same confidence q, the two pulls balance exactly where q matches the group's accuracy. how_it_works.py shows this for a group that is right 92% of the time:

2. Answers that are right 92% of the time, given confidence q. Average NLL:
   q = 0.80   0.3340
   q = 0.85   0.3013
   q = 0.90   0.2811
   q = 0.92   0.2788   <- lowest
   q = 0.94   0.2820
   q = 0.97   0.3085
   q = 0.99   0.3777

The lowest loss is at q = 0.92, the rate at which the answers are right. That holds for any rate. Average NLL is lowest when the confidence equals the accuracy. So fitting T by NLL pulls the confidence toward the accuracy.

There's only one number to fit, so a simple grid search is enough:

def fit_temperature(rows):
    """The T with the lowest NLL: 60 values from 0.05 to 20 on a log scale, then 25 values
    within 15% of the best one."""
    def log_grid(lo, hi, n):
        return [math.exp(math.log(lo) + (math.log(hi) - math.log(lo)) * i / (n - 1)) for i in range(n)]

    best = min(log_grid(0.05, 20, 60), key=lambda t: nll(rows, t))
    return min(log_grid(best / 1.15, best * 1.15, 25), key=lambda t: nll(rows, t))

how_it_works.py prints the group step's NLL for a few values of T. Unlike the table above, it's the sum over all 770 calibration answers rather than the average:

3. NLL of the right group, over the 770 BANKING77 calibration answers:
   T = 0.4     285.22
   T = 0.6     216.81
   T = 0.7     204.39
   T = 0.8     200.55
   T = 0.9     203.21
   T = 1.0     211.03
   T = 1.2     238.41
   T = 1.5     298.88

Unlike ECE, NLL changes smoothly with T. The curve has a single minimum, near T = 0.8 rather than 1. A minimum below 1 means the raw group probabilities are too flat, so the fit sharpens them.

One temperature per step

The NLL curve above is for the group step. Post 3's confidence multiplies two steps: P(group) from the group step and P(intent | group) from the intent step. There's no reason to expect the two to be off in the same way, so each step gets its own temperature:

• The group step's temperature is fitted on all 770 calibration answers.
• The intent step's temperature is one number shared by all the groups. It's fitted on the 729 calibration answers whose group step picked the right group. If the group is wrong, the right intent isn't among the intent step's options. NLL then has no probability to score.

An answer's confidence is then the product of the two scaled probabilities. Both temperatures apply to every answer, including those whose group was wrong.

The result

Fitted on 770 calibration answers:
  T for the group step:  0.803  (sharpens)
  T for the intent step: 1.138  (flattens)

        answers  accuracy  mean conf. raw  scaled  ECE raw  scaled
calib       770     86.9%           0.824   0.837   0.0545  0.0394
test       3080     86.9%           0.832   0.845   0.0409  0.0301

The two steps need opposite corrections. The group step is sharpened (T = 0.80), as the NLL curve suggested. The intent step is flattened (T = 1.14), because its probabilities are slightly too sharp when the group is right. This is why one temperature per step matters. A single temperature would be forced to compromise between two opposing corrections.

Because the group correction is larger, overall confidence rises. On the test set the mean confidence increases from 0.832 to 0.845, closer to the accuracy of 0.869.

ECE drops from 0.041 to 0.030 on the held-out test set. That's the number to trust, since the temperatures never saw these answers. Accuracy stays at 86.9% because no answer changed.

In the reliability diagram, every scaled point (orange) from 0.4 up still sits above the diagonal. The gaps are roughly halved to about 0.03 between 0.75 and 0.9. The top bin was already well-calibrated and remains so. The gaps stay large between 0.4 and 0.7, where two points are more than 0.1 above the diagonal.

Why did the middle bins improve less than the upper ones? Temperature is a single knob per step. It sharpens or flattens a whole distribution by the same rule, whatever an answer's confidence is. But in our case, the confidence around 0.6 needs a much bigger lift than the confidence around 0.95. One knob cannot give both. That's the main limitation of the method.

Another dataset, a bigger correction

Running temperature.py --dataset clinc150 applies the same fitting procedure to CLINC150, a second dataset that post 7 will introduce. It has 150 intents grouped into 10 domains, along with out-of-scope requests that match none of them. The domains are its groups, with one more group for the out-of-scope requests.

Fitted on 850 calibration answers:
  T for the group step:  0.523  (sharpens)
  T for the intent step: 0.619  (sharpens)

        answers  accuracy  mean conf. raw  scaled  ECE raw  scaled
calib       850     96.2%           0.859   0.959   0.1036  0.0147
test       5500     94.8%           0.860   0.956   0.0885  0.0113

On this dataset, the model is far more modest than before: it achieves 94.8% accuracy with an average confidence of only 0.860. With both steps sharpened, test ECE drops from 0.089 to 0.011. One knob per step can do a lot here. Unlike BANKING77's, the gaps stay large in the upper bins, so sharpening helps almost everywhere.

Comparing the two datasets shows a clear pattern: the fitted temperatures vary widely (from 0.52 to 1.14), and so does the direction of the correction. The intent step requires flattening on BANKING77 but sharpening on CLINC150. A temperature fitted on one dataset doesn't carry over to another. Every dataset requires its own fit on labelled data, as does any change to the prompt, the candidate labels or the model version.

One subtlety: confidence order across answers

Temperature scaling preserves the winning option within an answer, but it can change which of two answers looks more confident. A scaled confidence depends on the entire distribution, not just the top probability.

For example, compare an answer with the distribution 0.62 / 0.38 against one with 0.60 / 0.20 / 0.20. Before scaling, the first answer looks more confident. If we sharpen both with T = 0.5, the first rises to 0.73 while the second jumps to 0.82. One strong rival resists sharpening better than two weak ones. An answer's confidence is also the product of two steps. Those steps are scaled in opposite directions on BANKING77, creating even more chances for answers to swap places.

This matters in post 7, which asks how well the confidence separates right answers from wrong ones. That depends only on the order of the confidences. The reordering turns out to help a little on BANKING77 and hurt a little on CLINC150.

Why not train the calibration in?

TypeSafe trained calibration into Jev itself instead of adjusting the confidence after the fact. Could we do the same here?

In theory, yes. We could fine-tune the model on labelled messages so its raw probabilities come out calibrated. But for our setup, that's a poor trade-off:

• The ternary weights are difficult to fine-tune. Bonsai stores each weight as -1, 0 or +1, with one scale per block of 128 weights. PrismML converted it from the full-precision Qwen3.8-27B. Standard fine-tuning works on full-precision weights, so we would have to fine-tune the original 27B model and redo PrismML's conversion.
• It doesn't remove the need for labelled answers. Trained-in calibration only holds for inputs closely resembling the training set, which is equally true for a fitted temperature. Either way, each new task needs labelled answers to check the confidence. Once you have those labels, fitting a temperature is the cheaper step.
• Every prompt or label change means retraining. Adding an intent or rewording the prompt would require training again. In contrast, temperature scaling only requires re-running the calibration set (about 3 minutes here) and re-fitting T in a few seconds.

Temperature scaling needs only a few hundred labelled answers and no access to the model's weights. That's why it's the usual first step. When one knob per step isn't enough, post 6 fits a curve that can bend where the data demands.

Run it yourself

This post's own scripts don't need the server. They read the answers that post 3's run.py saved. Because post 3 only ran BANKING77, you'll need to run the two commands for CLINC150 first (listed below).

git checkout post-05
# Only if post 3's BANKING77 answers aren't saved yet (about 15 min):
python3 03-where-confidence-comes-from/run.py --split calib
python3 03-where-confidence-comes-from/run.py
# CLINC150's answers, new in this post (about 25 min):
python3 03-where-confidence-comes-from/run.py --dataset clinc150 --split calib
python3 03-where-confidence-comes-from/run.py --dataset clinc150
# Post 5:
python3 05-temperature-scaling/how_it_works.py
python3 05-temperature-scaling/temperature.py
python3 05-temperature-scaling/temperature.py --dataset clinc150
uv run 05-temperature-scaling/plot.py  # needs matplotlib: uv sync at the repo root

Both how_it_works.py and temperature.py use only Python's standard library. Your temperatures and ECE should be close to the results reported here. After a rerun of post 3, most probabilities move by less than 0.01. A few move by as much as 0.04.

---

BANKING77 is by PolyAI, licensed under CC BY 4.0: Casanueva et al., "Efficient Intent Detection with Dual Sentence Encoders", NLP4ConvAI workshop, 2020. CLINC150 is by Clinc, licensed under CC BY 3.0: Larson et al., "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction", EMNLP 2019.