Why typed decisions?

A bank customer types this into the support chat:

Help me use American Express to add money to the account.

The program behind the chat window has to decide what to do with it. Show the page on topping up by card? Explain which cards the bank takes? Pass it to a human agent? An if statement can make that choice only if something first turns the message into a value it can test.

Language models are good at reading messages like this one. They are less good at handing back something a program can use. This series is about closing that gap: getting a model to return a typed decision (one value from a fixed set) along with a confidence you can trust enough to act on.

Where the idea comes from

TypeSafe AI introduced Jev that is built for this kind of job. TypeSafe describes it as making "fast, structured decisions that software can use directly". It has four properties:

• Typed outputs. The possible answers are defined in advance. The model never returns anything else.
• One pass. Every output comes back from a single query.
• Calibrated confidence. Every answer has a confidence, and higher confidence means higher accuracy.
• Low latency. 70-500 ms end to end.

TypeSafe trained Jev with Reinforcement Learning for Calibrated Decisions (RLCD). This series doesn't try to reproduce Jev. It builds the same four properties from parts you can run yourself easily: an open-weight model, a grammar that limits what the model can write, the model's own token probabilities, and a calibration step fitted afterwards on a few hundred labelled examples. Every part is explained from scratch one at a time. Source code is provided.

Nothing here trains the model. To point it at your own task, you write a one-line description of each label and collect about fifteen labelled messages per label. Five go into the prompt as examples, and ten fit the calibration. The calibration needs a few hundred messages in all, so a task with fewer labels needs more per label. If you change the labels, you just refit in seconds instead of retraining. Because the model runs on your own machine, the messages never leave it.

The trade is accuracy. The model used in the series is right on 86.9% of BANKING77's test messages. In the dataset's own paper, classifiers trained on 10 messages per intent score 83-85% on the same test set, and those trained on full training data about 93%. So the series doesn't aim to be the most accurate. It makes the model say how sure it is so that a program knows which answers to act on, which to ask users for clarification, and which to escalate to a human agent.

The setup

Everything in the series runs on a Dell Pro Max with GB10 (Dell's version of the NVIDIA DGX Spark). That's simply the machine I chose, but the series doesn't depend on it. Another NVIDIA GPU with enough memory for the model should work too.

The model is Ternary Bonsai 2 27B (Apache-2.0). It is a 27B model whose weights are packed into 6.8 GB. It's served by llama-server from the PrismML fork of llama.cpp using a Docker container. The setup folder builds the docker image, downloads the model and checks its checksums, and starts the service in three commands.

The running example is the BANKING77 dataset. It has 13,083 messages that customers sent to a bank's support team. Each is labelled with one of 77 intents, such as card_arrival, top_up_failed and supported_cards_and_currencies. Since many of the 77 intents overlap, even a good model gets some wrong. That makes it a good dataset for studying when to trust an answer.

Let's ask the model about some of these messages three ways.

Attempt 1: just ask

The simplest thing is to ask what the customer wants:

SYSTEM = "You help a bank's customer support team. Say what the customer wants."
r = chat(SYSTEM, "I am still waiting on my card?", max_tokens=120)
reply = r["choices"][0]["message"]["content"]

chat is a few lines that POST the request to the server's OpenAI-style /v1/chat/completions endpoint with temperature set to 0. The full script is free_text.py. Here's what came back for three messages. The reply of the second message is trimmed because it exceeded the 120-token limit.

Customer: I am still waiting on my card?
Model:    It sounds like you are checking the status of your pending card delivery and
          inquiring about why it has not arrived yet.
          (25 tokens, 1045 ms on the server)

Customer: I need assistance with understanding which fiat currencies that you support.
Model:    As an AI assistant, I do not have access to real-time banking data or specific
          institutional policies. Therefore, I cannot provide the exact list of supported
          fiat currencies for your specific bank account. ...
          (120 tokens, 4191 ms on the server)

Customer: How long is the delivery time to the US?
Model:    I'm sorry, but I don't have access to specific shipping details or current
          delivery times for your bank's services. ...
          (72 tokens, 2569 ms on the server)

The first reply is right. A person reading it knows what to do. But a program doesn't because it would need a second step to turn "checking the status of your pending card delivery" into something it can branch on. That second step is the original problem all over again.

The other two replies didn't classify the message at all. They answered the customer apologetically. When we ask a model an open question, it will do whatever seems most helpful. But here that wasn't what we wanted.

Prose is also slow. The model writes one token at a time. A 120-token reply took over 4 seconds in this case. A decision shouldn't take that long.

Attempt 2: give it the list

The obvious fix is to tell the model what the answers can be. label_list.py puts all 77 intent names in the system prompt and asks for exactly one:

system = ("You classify messages sent to a digital bank's customer support.\n"
          "Reply with the single best intent from this list, and nothing else:\n" + "\n".join(intents))

Then it sends 200 BANKING77 messages and checks each reply against the list:

200 BANKING77 messages, the 77 intent names in the prompt, no grammar:
   199 replies were an exact label
     0 replies were a label inside other text
     1 replies were no label
   153 were the right label (76%)
  median time per request: 283 ms

  no label: 'Help me use American Express to add money to the account.' -> 'top_up_by_card'

This is much better. Each request takes under 0.3 seconds since the replies are short. 199 of 200 replies are a label.

The one reply that isn't a label is the trouble. top_up_by_card isn't one of the 77 intents. The model reached for the intent topping_up_by_card but misspelled it. Look at how BANKING77 names its top-up intents:

top_up_by_bank_transfer_charge
top_up_by_card_charge
top_up_by_cash_or_cheque
top_up_failed
top_up_limits
top_up_reverted
topping_up_by_card

Six begin with top_up_, but the one the model reached for begins with topping_up_. The model followed the pattern of the other six and made up a near-copy of a label. On the other hand, it was actually the wrong intent. The message asks for help adding money with American Express. BANKING77 files that under supported_cards_and_currencies because the real question is whether that card is accepted. But a wrong label still lets the program run, while a label that doesn't exist breaks it.

Code that looks the reply up in a table will crash on it or quietly fall through to a default branch. One reply in 200 sounds rare. But it will be 50 if there are 10,000 messages a day, and you can't know in advance which messages will do it.

The made-up label is easy to spot. Two other problems aren't so easy, though:

• The replies have no confidence. card_arrival for "I am still waiting on my card?" looks exactly the same whether the model was sure or nearly tossed a coin. The program can't tell the 153 right answers from the 47 wrong ones, so it has to either trust all of them or check all of them.
• Getting the format right depends on the model behaving. Nothing prevents a chatty reply. The instruction "and nothing else" worked 200 times here. But a stranger message or a different model may not follow it. Post 2 shows a few that don't.

Attempt 3: a typed decision

Now the version this series builds. typed.py sends each message in one request and gets back a label and a confidence:

I am still waiting on my card?
  -> card_arrival  (code A-A, confidence 0.65, 234 ms)

I need assistance with understanding which fiat currencies that you support.
  -> fiat_currency_support  (code G-D, confidence 0.97, 228 ms)

How long is the delivery time to the US?
  -> card_delivery_estimate  (code A-B, confidence 0.65, 228 ms)

Help me use American Express to add money to the account.
  -> topping_up_by_card  (code E-I, confidence 0.35, 228 ms)

Three things changed.

The answer is always a real label. The model doesn't write the label's name. It writes a short code such as A-A. It consists of a letter for the group of intents (A is the group for setting up a card) and a letter for the intent within it. A grammar passed with the request allows only the 77 valid codes. The model can't write anything else, not even a near-copy. The program maps the code back to a label. Post 2 will explain how that works and why the codes are letters.

Each answer comes with a confidence. It comes from the probabilities the model gave each letter as it wrote the code (post 3). No second request is needed and nothing is parsed out of text.

It's fast (under a quarter of a second). The model writes two tokens. The long system prompt, which includes 77 intents with descriptions and examples, is processed once and cached by the server. Each request only pays for the message itself.

The American Express message is still wrong. It's about which cards the bank accepts (supported_cards_and_currencies), but the model chose topping_up_by_card. A grammar can stop a model from inventing a label, but it can't make it pick the right one. This time the mistake comes with a confidence of 0.35, and that's something a program can act on.

A word on accuracy. Attempt 2 got 76% of its 200 messages right. Attempt 3 gets 88.5% (177 of 200) on the same 200. But that isn't a fair contest. The typed prompt also gives the model a one-line description and five example messages for each intent (post 2 shows it). That probably accounts for much of the gap. The point of attempt 3 isn't higher accuracy. It's that the answer is always valid and comes with a confidence.

From a confidence to an action

Here's what the end of the series (post 8) does with three of these answers. First, the confidences are corrected to match how often the model is actually right. That gives 0.65, 0.98 and 0.37 (posts 4-6 explain why they need correcting and how). Then two thresholds, both fitted on labelled messages, split each answer into one of three routes:

I am still waiting on my card?                     0.65  -> ASK
  Ask: is this about card_arrival or card_delivery_estimate?

I need assistance with understanding which ...     0.98  -> ACT
  Handle it as fiat_currency_support.

Help me use American Express to add money ...      0.37  -> HAND OVER
  Pass it to a person.

The sure answer (the 2nd message above) is acted on. The middling one (the 1st message) becomes a quick question to the customer to clarify between the model's top two guesses, card_arrival or card_delivery_estimate. The wrong one (the 3rd message) is escalated to a human agent.

Here's what that buys on 3,080 BANKING77 messages that played no part in fitting the thresholds. The model's answer was right for 86.9% of them, so acting on every answer would mean about 400 wrong actions. That isn't good enough, but it doesn't need to be. The confidence says which answers are safe to act on. The rules act only on the half the model is most confident about (1,566 messages), and 98.7% of those were right. That's 21 wrong actions instead of about 400. The other half are asked about or passed to a human agent.

The three examples above come from the labelled messages used to fit the thresholds, so they only illustrate the routes. The 98.7% comes from the 3,080 test messages, which the thresholds never saw. Post 8 explains how the thresholds are chosen and how much that 98.7% could vary on other messages.

The four properties, measured

Here's where the series' version stands against Jev's four properties on this machine and this dataset:

Property	How the series gets it	Measured on BANKING77 (3,080 test messages)
Typed outputs	A grammar allows only the valid codes (post 2)	3,080 of 3,080 answers valid, none retried
One pass	The model writes two tokens in one request	One request per message
Calibrated confidence	Token probabilities (post 3) are checked (post 4) and corrected afterwards on 770 labelled messages (posts 5-6)	Calibration error: 0.041 raw, 0.030 after temperature scaling, 0.011 after isotonic regression
Low latency	Short answers and a cached prompt	About 0.23 s per request

Some limitations:

• Calibration is fitted, not trained in. It needs its own labelled messages (770 here, ten per intent), and it has to be fitted again when the labels change. Post 5 explains why training it into the model isn't practical here.
• Fast for one user, not for many. The 0.23 s latency holds when requests come one at a time. Sending 8 at once barely raises the throughput (from about 4.3 to 4.7 requests per second), and each request then takes about 1.7 s. Batching usually helps by sharing the cost of generating tokens. But each answer here is only two tokens: most of a request's time goes into processing the message, and batching doesn't speed that part up (measured with throughput.py).
• "One pass" is one decision per request. Jev returns many decisions from one query. Post 8 discusses doing the same here.
• BANKING77 is cleaner than real traffic. BANKING77 messages are short and each has a single intent. Measure on your own data before you trust these numbers.
• Numbers vary a little between runs. If you rerun the scripts, probabilities can differ in the second or third decimal place. A few close calls can flip. The conclusions don't change, though.

What's next

Each post adds one piece with code:

1. Why typed decisions? (this post)
2. Forcing a valid answer: grammars; why the codes are single letters
3. Where confidence comes from: reading the model's token probabilities
4. Is the confidence honest? Reliability plots and calibration error
5. Fixing it: temperature scaling
6. When one knob isn't enough: isotonic regression
7. Does confidence separate right from wrong? Accuracy-coverage curves and AUROC
8. From confidence to action: thresholds and routes (act / ask / hand over)

Run it yourself

The code is at github.com/liminma/typed-decisions-from-scratch. You need Docker with the NVIDIA Container Toolkit, and uv that the download-model.sh script uses. The setup scripts default to GB10; on another NVIDIA GPU, change the two build settings described in the setup folder.

git clone https://github.com/liminma/typed-decisions-from-scratch.git
cd typed-decisions-from-scratch
git checkout post-01
./00-setup/build-server.sh
./00-setup/download-model.sh
./00-setup/start-server.sh

python3 01-why-typed-decisions/free_text.py
python3 01-why-typed-decisions/label_list.py
python3 01-why-typed-decisions/typed.py
python3 01-why-typed-decisions/throughput.py

The scripts use only Python's standard library. They download BANKING77 the first time they run. They talk to the llama-server that start-server.sh starts at http://127.0.0.1:8080 by default. If you start it on another port, set LLAMA_URL to match.

---

BANKING77 is by PolyAI, licensed under CC BY 4.0: Casanueva et al., "Efficient Intent Detection with Dual Sentence Encoders", NLP4ConvAI workshop, 2020. The intent groups and descriptions in the prompt aren't part of BANKING77; they're in the repo's datasets/ folder.