Decision models#
So far, we have focused on chat models: we send text to them and get free-form text back. Often, though, we are expecting some kind of more structured response, as we have seen before, as well.
There are some special cases of structured responses that may be better served by a model class called Decision or System One models. Screening a pile of abstracts for a literature review, for example, flagging a document as relevant or not, or sorting survey responses into categories: these are classification tasks that strictly speaking would require just a single label from the model for each prompt.
A Decision model answers exactly that kind of typed judgment directly. Instead of a sentence, it returns a label chosen from a fixed set of options, a probability for each option, and a confidence score, without lengthy reasoning and without the risk of breaking the output schema. langchain_dartmouth makes a variety of Decision models available through DartmouthDecisions, both local and cloud-based.
In this recipe, we will use a Decision model to screen abstracts for a literature review: First we classify each abstract’s research method, then we check whether it is relevant to a research question.
Setup#
We start the same way as in earlier recipes, loading our environment variables:
from langchain_dartmouth.decisions import DartmouthDecisions, Choice, Noul, Score
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())
True
Just like with chat models, we can list the Decision models available to us before picking one:
for model in DartmouthDecisions.list():
print(model)
id='cloudflare.clef' name=None description=None is_embedding=None capabilities=None is_local=None cost=None
id='typesafe.jev' name=None description=None is_embedding=None capabilities=None is_local=None cost=None
We will use cloudflare.clef, the default model for DartmouthDecisions, so we don’t need to pass model_name:
decisions = DartmouthDecisions()
Input data#
Imagine we are screening abstracts for a literature review. Here are a few invented abstracts covering a mix of research methods:
abstracts = {
"abstract_1": (
"We surveyed 412 undergraduate students across six institutions to measure "
"the relationship between weekly screen time and self-reported sleep quality. "
"Using multiple regression, we found a significant negative association "
"between screen time and sleep quality scores (p < 0.01)."
),
"abstract_2": (
"Through semi-structured interviews with 18 first-generation college "
"students, this study explores how participants describe their sense of "
"belonging on campus. Thematic analysis identified three recurring themes: "
"mentorship, financial stress, and peer community."
),
"abstract_3": (
"We propose a formal framework for reasoning about fairness in resource "
"allocation, building on axioms from cooperative game theory. We prove that "
"no allocation rule can simultaneously satisfy envy-freeness and "
"proportionality under our stated conditions."
),
"abstract_4": (
"This mixed-methods study combines a 200-respondent survey with follow-up "
"focus groups to examine faculty attitudes toward generative AI tools in "
"the classroom. Quantitative results showed moderate adoption, while "
"qualitative data revealed persistent concerns about academic integrity."
),
}
for key, abstract in abstracts.items():
print(f"{key}: {abstract}\n")
abstract_1: We surveyed 412 undergraduate students across six institutions to measure the relationship between weekly screen time and self-reported sleep quality. Using multiple regression, we found a significant negative association between screen time and sleep quality scores (p < 0.01).
abstract_2: Through semi-structured interviews with 18 first-generation college students, this study explores how participants describe their sense of belonging on campus. Thematic analysis identified three recurring themes: mentorship, financial stress, and peer community.
abstract_3: We propose a formal framework for reasoning about fairness in resource allocation, building on axioms from cooperative game theory. We prove that no allocation rule can simultaneously satisfy envy-freeness and proportionality under our stated conditions.
abstract_4: This mixed-methods study combines a 200-respondent survey with follow-up focus groups to examine faculty attitudes toward generative AI tools in the classroom. Quantitative results showed moderate adoption, while qualitative data revealed persistent concerns about academic integrity.
Classifying by research method#
A Decision model answers typed questions, called Noul (binary), Choice (categorical), or Score (ordinal), about a piece of state: the text or structured data being judged. We’ll start with a single Choice question asking the model to classify each abstract’s research method:
method_question = Choice(
instructions="What research method does this abstract primarily describe?",
criteria={
"quantitative": "Numerical data analyzed with statistics, such as a survey or an experiment.",
"qualitative": "Non-numerical data such as interviews, observations, or text, analyzed thematically.",
"mixed-methods": "Combines both quantitative and qualitative data within the same study.",
"theoretical": "Formal argument, proof, or framework, with no empirical data collected.",
},
)
response = decisions.invoke(
{
"state": abstracts["abstract_1"],
"questions": {"method": method_question},
}
)
answer = response.choices["method"]
print("Choice: ", answer.choice)
print("Probabilities:", answer.probabilities)
print("Confidence: ", answer.confidence)
Choice: quantitative
Probabilities: {'quantitative': 0.9758332371711731, 'qualitative': 0.00642280001193285, 'mixed-methods': 0.013597069308161736, 'theoretical': 0.004146871622651815}
Confidence: 0.9758332371711731
Unlike a chat model’s free-form response, the choice field is always exactly one of the labels we declared in criteria. The probabilities dictionary shows the full distribution across every candidate, and confidence summarizes how concentrated that distribution is. A low confidence score means that a human reviewer should double-check the classification.
Note
Decision models and LLM-as-a-judge. It is common paradigm to use a chat model as a judge: prompting it to classify or grade some text, then parsing the label back out of its response. While this can work well, it means an extra round trip through free-form text and requires careful prompting and output parsing. A Decision model skips both problems: it answers from a declared set of options directly, so the response is faster and cheaper to produce and it cannot hallucinate a label outside criteria, because the label is the answer type.
Let’s classify the rest of our abstracts the same way:
for key, abstract in abstracts.items():
response = decisions.invoke(
{"state": abstract, "questions": {"method": method_question}}
)
answer = response.choices["method"]
print(f"{key}: {answer.choice} (confidence: {answer.confidence:.2f})")
abstract_1: quantitative (confidence: 0.98)
abstract_2: qualitative (confidence: 0.98)
abstract_3: theoretical (confidence: 0.98)
abstract_4: mixed-methods (confidence: 0.98)
Composing multiple questions#
A single request can ask more than one typed question about the same state at once. Suppose our literature review is specifically about the effects of technology use on student wellbeing. Alongside the method classification, we can add a Noul question asking whether an abstract is relevant to that research question:
research_question = "the effects of technology use on student wellbeing"
relevance_question = Noul(
instructions=f"Is this abstract relevant to a literature review on {research_question}?"
)
for key, abstract in abstracts.items():
response = decisions.invoke(
{
"state": abstract,
"questions": {"method": method_question, "relevant": relevance_question},
}
)
method = response.choices["method"]
relevance = response.nouls["relevant"]
print(f"{key}: method={method.choice}, " f"relevant={relevance.noul:.2f}")
abstract_1: method=quantitative, relevant=0.97
abstract_2: method=qualitative, relevant=0.06
abstract_3: method=theoretical, relevant=0.02
abstract_4: method=mixed-methods, relevant=0.12
A Noul answer is a probability that the answer is “yes”: values near 1 support inclusion, values near 0 support exclusion, and values near 0.5 flag ambiguity. Combined with the method classification, we can now write ordinary Python logic to triage the abstracts, for example, keeping anything above a relevance threshold for manual review.
Rating against a rubric#
Choice and Noul cover picking a label and answering yes/no, but some judgments fall on a spectrum. Score asks the model to rate the state against an ordered rubric, such as how rigorous an abstract’s methodology appears. Criteria are numbered from zero in the order we supply them, and the returned score is an expected value, which means it may fall between two rubric levels:
rigor_question = Score(
instructions="How rigorous is the methodology described in this abstract?",
criteria=[
"Minimal methodological detail; rigor cannot be assessed.", # Level 0
"Some methodological detail, but with notable gaps or ambiguity.", # Level 1
"Clear, well-justified methodology appropriate to the claims made.", # Level 2
],
)
for key, abstract in abstracts.items():
response = decisions.invoke(
{"state": abstract, "questions": {"rigor": rigor_question}}
)
answer = response.scores["rigor"]
print(f"{key}: score={answer.score:.2f}")
abstract_1: score=0.92
abstract_2: score=0.98
abstract_3: score=1.17
abstract_4: score=1.00
legend gives us the original rubric descriptions keyed by their zero-based level, and probabilities gives the full distribution over those levels, the same way Choice exposes a distribution over labels:
response = decisions.invoke(
{"state": abstracts["abstract_1"], "questions": {"rigor": rigor_question}}
)
answer = response.scores["rigor"]
print("Score: ", answer.score)
print("Legend: ", answer.legend)
print("Probabilities:", answer.probabilities)
print("Confidence: ", answer.confidence)
Score: 0.916561171412468
Legend: {0: 'Minimal methodological detail; rigor cannot be assessed.', 1: 'Some methodological detail, but with notable gaps or ambiguity.', 2: 'Clear, well-justified methodology appropriate to the claims made.'}
Probabilities: {0: 0.17124414443969727, 1: 0.7409506440162659, 2: 0.08780526369810104}
Confidence: 0.7409506440162659
Summary#
In this recipe, we have learned how to get typed judgments from a decision model:
DartmouthDecisionswraps Dartmouth-provided Decision models, which answerNoul(binary),Choice(categorical), andScore(ordinal) questions about a piece of state, returning probabilities and confidence.A single request can combine multiple typed questions about the same state
Because the answer is drawn from a declared set of outcomes rather than generated text, Decision models avoid the parsing and hallucination risks of using a chat model as a judge, while also being faster and cheaper to run.
confidencegives us a principled way to decide when to trust an automated judgment and when to route it to a human reviewer.