Why typed decisions?
Getting an LLM to return a value your code can branch on with a confidence you can act on
Series
Typed decisions from scratch
A deep dive into getting typed and trustworthy decisions from an LLM from scratch.
This series covers grammars, logprobs, calibration, AUROC, and thresholding to produce outputs your code can safely rely on.
Code: https://github.com/liminma/typed-decisions-from-scratch
Posts in this series
Forcing a valid answer
Grammars, one-letter choices, and an LLM that can only answer with a label from your list
Where confidence comes from
Reading how sure an LLM was of its answer, from the probabilities it already computed
Is the confidence honest?
Building reliability diagrams and computing ECE from scratch to check an LLM's confidence
