# When one knob isn't enough: isotonic regression

[Post 5](https://liminma.hashnode.dev/temperature-scaling) corrected the model's confidence with a temperature: one number per step that makes the probabilities sharper or flatter. On the BANKING77 test set, it brought the expected calibration error (ECE) from 0.041 down to 0.030. It couldn't go further because a temperature bends the probabilities by the same rule at every confidence level. But confidence around 0.6 needed a much bigger lift than confidence around 0.95, and one knob can't give both.

**Isotonic regression** has no such limit. From the calibration answers, it learns a separate correction for each confidence range. On the same test set, that brings ECE down to 0.011. But the flexibility comes with costs that a single ECE number hides, and those are as much the point of this post as the method itself.

## The idea

Post 4 measured calibration by grouping answers by confidence and checking how often each group was right. We can turn that measurement into a correction: for each of post 4's bins, set the corrected confidence to how often the bin's answers were right on the calibration set.

On its own that mapping would be jumpy and noisy. Isotonic regression adds one constraint: **the corrected confidence never goes down as the raw confidence goes up.** Subject to that rule, it minimizes the sum of squared errors against the calibration outcomes (1 for right and 0 for wrong). Unlike post 4's bins, the ranges aren't set in advance. The fit decides where each one starts and ends.

## Pool adjacent violators

The Pool-Adjacent-Violators Algorithm (PAVA) finds this fit and is short enough to write out:

1. Sort the calibration answers by raw confidence. Each answer starts as its own block with value 1 if right and 0 if wrong.
2. If a block has a higher value than the block after it, merge the two into a single block valued at the share of its answers that are right.
3. Repeat until the block values never go down.

[`pav.py`](https://github.com/liminma/typed-decisions-from-scratch/blob/post-06/06-isotonic-regression/pav.py) shows it on six answers:

```plaintext
Raw confidence: 0.30  0.40  0.50  0.60  0.70  0.80
Right?             0     1     0     1     1     1
Calibrated:     0.00  0.50  0.50  1.00  1.00  1.00
```

The answer at 0.40 was right (1) and the one at 0.50 was wrong (0). That violates the order so they pool into one block with a value of 0.50. Everything else is already in order.

The code keeps a stack of blocks and merges backwards as each answer comes in:

```python
blocks = []  # each block: [sum of y, count, first x, last x]
for xi in sorted(points):
    s, n = points[xi]
    blocks.append([s, n, xi, xi])
    # Merge backwards while the previous block is valued higher than this one
    while len(blocks) > 1 and blocks[-2][0] / blocks[-2][1] > blocks[-1][0] / blocks[-1][1]:
        s, n, _, last = blocks.pop()
        blocks[-1][0] += s
        blocks[-1][1] += n
        blocks[-1][3] = last
```

Here `points` maps each raw confidence to how many of its answers are right and how many there are, since answers with exactly the same confidence can't be put in order.

Each block spans the raw confidence from its first answer to its last. To correct a new answer we check where its raw confidence lands:

- Inside a block: use that block's value.
- Between two blocks: interpolate linearly between them.
- Below the first block or above the last: clip to the nearest block's value.

The merging and the lookup together are the whole method. `pav.py` fits it on the 770 calibration answers alongside scikit-learn's `IsotonicRegression`. Both give the exact same corrected confidence on all 3,080 test answers. In practice you would use scikit-learn:

```python
from sklearn.isotonic import IsotonicRegression

iso = IsotonicRegression(y_min=0.0, y_max=1.0, out_of_bounds="clip")
iso.fit(calibration_conf, calibration_right)
corrected = iso.predict(test_conf)
```

## The staircase

[`isotonic.py`](https://github.com/liminma/typed-decisions-from-scratch/blob/post-06/06-isotonic-regression/isotonic.py) fits the 770 BANKING77 calibration answers and produces a staircase of 11 steps:

```plaintext
     raw confidence  answers  calibrated
  0.102 to 0.115        2       0.000
  0.119 to 0.305       20       0.150
  0.308 to 0.348       11       0.455
  0.361 to 0.481       48       0.458
  0.482 to 0.560       20       0.750
  0.561 to 0.787      123       0.797
  0.789 to 0.797        6       0.833
  0.798 to 0.804        8       0.875
  0.805 to 0.883       69       0.913
  0.885 to 0.965      263       0.954
  0.965 to 0.991      200       1.000
```

![The isotonic fit on BANKING77's calibration answers: a staircase above the diagonal for most of the range, with tick marks for right answers along the top and wrong answers along the bottom](https://cdn.hashnode.com/uploads/covers/6ac6cc370c38c868db37ee0a/9bcb2a38-5b8a-45be-a583-351e66501d03.png align="center")

This is the flexibility a temperature lacked. A raw confidence of 0.60 gets a big lift to 0.797 while one of 0.95 barely moves to 0.954. Each range gets the correction its own calibration answers call for.

## The result and what it costs

Here are the three versions side by side on the 3,080 test answers (with both calibration methods fitted on the 770 calibration answers only):

```plaintext
                  ECE mean conf. distinct values  largest tie   accuracy at 80% coverage
  raw           0.041      0.832            3079         0.1%   94.9%
  temperature   0.030      0.845            3079         0.1%   95.1%
  isotonic      0.011      0.878              66        32.4%   94.6%  (92.7% to 96.3%, by tie order)
```

Isotonic wins easily on ECE: 0.011 against 0.030 for temperature scaling. Its mean confidence of 0.878 is now right next to the test accuracy of 0.869. That low ECE comes with four costs. The rest of the table shows the first cost: ties.

### Ties

The staircase is flat over most of its length, so many different raw confidences come out as the same number. The 3,080 test answers end up with only **66 distinct values**. The largest tie is the 0.954 step, which holds 998 answers (almost a third of the test set).

The answers can't be ranked any more within a tie. Raw confidence ranks an answer at 0.96 above one at 0.89, but both are 0.954 after isotonic regression.

That matters as soon as you use confidence to choose answers. The last column (**accuracy at 80% coverage**) keeps the top 80% of answers by confidence and asks how many of them are right. (Post 7 makes this its main tool.)

The cut keeps 2,464 answers (80% of 3,080). With isotonic regression, the cutoff lands inside the 0.797 step that holds 402 answers. There are 2,270 answers above that step, so the cut needs 194 of the 402. But which 194 do you take? There is no way to say because they are tied. Take the wrong ones first and the kept answers are 92.7% right. Take the right ones first and they are 96.3% right. Averaged over all tie orders, accuracy is 94.6%.

The code for the average is just a few lines. Count the right answers above the tie, then add the tie's accuracy times the number of its answers that make the cut:

```python
for c in sorted(groups, reverse=True):
    n, right = groups[c]
    m = min(n, k - taken)  # how many of this tied group make the cut
    right_exp += m * right / n
    ...
```

Temperature scaling keeps the confidences distinct, so its 95.1% has no such uncertainty. On this test set it is also slightly higher than isotonic's 94.6% average. Isotonic gives the most honest numbers but ranks the answers worst of the three here.

### A flattering ECE

Post 4 showed that you can have a perfect ECE of 0 if you give every answer the same confidence score that equals the overall accuracy. Isotonic regression is partway there. Each step's value is the accuracy of the calibration answers in it. When most test answers land on a few big steps, each big bin's confidence matches its accuracy closely. The reliability diagram shows this:

![Reliability diagram for BANKING77 test: raw, temperature and isotonic, with marker size showing how many answers each bin holds; isotonic's large markers sit near the diagonal while two single-answer bins spike to 1.0](https://cdn.hashnode.com/uploads/covers/6ac6cc370c38c868db37ee0a/aa449976-74b4-4481-8ff5-46f110726ddb.png align="center")

Marker area shows how many answers each bin holds. Isotonic's big green markers sit close to the diagonal. The two green spikes up to 1.0 are bins that hold a single test answer each. The two test answers fell where the staircase jumps from 0.150 to 0.455 and from 0.458 to 0.750 over a tiny range of raw confidence. No other test answer landed in either bin. With one answer each, these bins can only be 0% or 100% right. Both answers were right, so both bins sit at 100% far above the diagonal. ECE weights each bin by 1 of 3,080 answers and barely counts it.

None of that makes isotonic's ECE wrong. ECE only checks that confidence matches accuracy within each bin of test answers. It doesn't check whether right answers get higher confidence than wrong ones. Post 4's constant confidence scores a perfect 0 and can't rank anything. Post 7 measures ranking separately.

### Steps that rest on little data

Each step is an estimate from the calibration answers inside it, and some steps hold very few. The first step at 0.000 rests on 2 answers. The 0.833 and 0.875 steps rest on 6 and 8.

The top step says 1.000 because all 200 calibration answers in it were right. Of the 864 test answers that landed on this step, 6 were wrong. This means 99.3% of them were right. That is excellent but still short of the certainty that 1.000 claims. A program that took 1.000 literally ("never wrong, no need to check") would be surprised six times. When errors are this rare, 200 answers can easily miss them all. At the test set's rate of 6 wrong in 864, the chance that a random group of 200 has no wrong answers is about one in four.

Temperature scaling fits two temperatures, each from hundreds of answers: all 770 for the group step and 729 for the intent step. On BANKING77, isotonic regression fits a value for each of its 11 steps from only the calibration answers on that step. So isotonic regression needs more calibration data to be trustworthy, especially at the ends.

### Only the chosen answer

Temperature scaling corrects the probability of every option. Isotonic regression only corrects the chosen answer's confidence and leaves the other options as they were. Isotonic regression as fitted here can't give an honest probability for the second choice. That matters whenever someone takes the probability at face value. For example, a human agent might use the second intent's probability to decide whether it is worth checking. You can get an honest one by fitting another curve for the second choice at the cost of much more calibration data.

## Which to use

Neither wins on everything:

|  | Temperature scaling | Isotonic regression |
| --- | --- | --- |
| ECE on BANKING77 test | 0.030 | **0.011** |
| Shape of the correction | One temperature each for the group and intent steps | Any non-decreasing curve |
| Calibration data needed | A few hundred answers | More (especially at the ends) |
| Ranking of answers | Changed only slightly (no ties) | Lost within ties (only 66 distinct confidences) |
| Accuracy at 80% coverage | **95.1%** | 94.6% on average (92.7% to 96.3%) |
| Corrects the whole distribution | Yes | Chosen answer only |

A reasonable rule: start with temperature scaling and look at its reliability diagram. If the corrected points still bend away from the diagonal and you have enough calibration data, try isotonic regression on the raw confidence instead. Isotonic regression will likely give a lower ECE than temperature scaling, as it did here. But a low ECE doesn't show what its ties do to the ranking, so you should check that as well. If you use confidence to choose answers, ranking matters more.

The rest of this series uses the temperature-scaled confidence. The reason is how post 8 uses confidence. There the program acts on an answer only when its confidence clears a threshold. The aim is to act as often as possible while keeping at least 98% of those actions right. Since tied answers share one confidence, a threshold has to take all of them or none.

On BANKING77, the top two steps of the staircase are big ties at 1.000 and 0.954. Acting on the 1.000 step alone would cover about a quarter of the answers. Adding the 0.954 step would raise that to about 60%, but then fewer than 98% of the actions would be right. Temperature scaling keeps the confidences distinct, so the threshold can go anywhere. The program then acts on about half the answers and keeps at least 98% of them right.

Could more calibration data fix the staircase by giving it more steps? It would help, but not enough. Smaller steps give the threshold more places to stop, so the program could act on more than a quarter of the answers. But the staircase never reorders answers, so even a finer one can at most match raw confidence. Post 8 shows that raw confidence lets the program act on only about a third of the answers at the same 98% accuracy. Temperature scaling reaches about half because it reorders some answers so that more right ones end up at the top.

## What's next

Both corrections made the confidence more honest. Is it also *useful*? Does higher confidence actually pick out the right answers? This post only touched on that. [Post 7](https://liminma.hashnode.dev/accuracy-coverage-and-auroc) measures it across the full range with accuracy-coverage curves and AUROC. It also checks whether temperature scaling's better order holds on CLINC150.

## Run it yourself

These scripts don't need the server. They read the calibration and test answers saved by [post 3](https://liminma.hashnode.dev/where-confidence-comes-from)'s `run.py`. If those aren't saved yet, first run the two `run.py` lines below. They do need the server. Then install scikit-learn and Matplotlib with `uv sync` at the repo root and run:

```bash
git checkout post-06
# Only if post 3's answers aren't saved yet (about 15 min):
python3 03-where-confidence-comes-from/run.py --split calib
python3 03-where-confidence-comes-from/run.py
# Post 6:
uv run 06-isotonic-regression/pav.py
uv run 06-isotonic-regression/isotonic.py
uv run 06-isotonic-regression/plot.py
```

Your numbers should be close to these but not identical because they come from your own post 3 answers. After a rerun of post 3, most raw probabilities move by less than 0.01 and a few by up to 0.04. Isotonic regression's steps can change in number and position with them.

---

*BANKING77 is by PolyAI, licensed under* [*CC BY 4.0*](https://creativecommons.org/licenses/by/4.0/)*: Casanueva et al., "Efficient Intent Detection with Dual Sentence Encoders", NLP4ConvAI workshop, 2020.*