The Decision System

A dashboard is an artifact. A decision system is that artifact in motion — a canonical definition for every measure, surfaces built backward from the specific recurring calls people make, and a distribution that outlives the individuals who built it. The flag is not the system. The system is everything that has to be true for the flag to reach someone authorized to act on it, measuring something real, in time to matter.

We’ll follow that flag (the at-risk / early-warning indicator) through the whole framework. It’s worth noticing early that the pattern travels. The same decision-system shape shows up as a behavioral-health team’s relapse-risk score, a hospital’s deterioration alert, a funder’s off-track grantee. These aren’t structurally identical. The stakes, the data behind them, the right to intervene, and the cost of being wrong all differ. But the shape recurs, and that recurrence is what makes a single framework worth writing.

Pipes and the faucet

Let’s start with what’s already solved. Moving data, storing it, transforming it at scale (the plumbing) is, for a growing number of institutions, increasingly tractable with mature tools. The pipes got upgraded. Warehouses are cheap, pipelines are decent, the bytes arrive.

The water didn’t get better. What comes out of the faucet (the resolution of the signal, the authority to act on it, the meaning it carries) is still mostly ungoverned. Data governance manages the pipe: who can access a table, how a field is typed. Decision governance manages the faucet: who is allowed to act on what comes out, at what resolution, on what cadence. The first is, for many institutions, increasingly an engineering problem with engineering answers. The second is an architecture problem, and it is mostly still in front of us.

Three things fail at the faucet, and the rest of this essay is about them: meaning, authority, and validity. They are not a checklist; they hold one another up. Meaning makes the institution internally consistent. Authority decides who acts on that consistent signal, and when. Validity asks whether the consistent thing is true enough to be worth acting on at all. Detach any one from the other two and the system doesn’t degrade gracefully — it produces confident nonsense at speed. Get all three right and the flag becomes a decision. Miss any one and you have a very expensive way of producing a red dot.

Meaning: define it once

Our flag depends on words. Attendance. On-track. Proficient. Every one of them is a definition, and every definition is a place where the institution can quietly disagree with itself.

Numbers don’t disagree because the math is wrong. Numbers disagree because the words do. Ask two schools what “chronic absenteeism” means and you’ll get two answers — does present but two hours late count? Remote? And the sharpest case of all: a student is suspended, and somewhere a staff member codes that day as present rather than absent. No engineer touched the pipeline. The definition drifted at the source, at the point of capture, in a single keystroke. Now the flag that depends on it means something different in one building than in the next.

The fix is structural, not a directive. You build a semantic keystone: a single layer, governed in code, where every metric is defined once. On-track is computed in one place, and every surface (the teacher’s console, the school’s program report, the district’s executive view, and every AI feature downstream) reads from that one computation. There’s no sanctioned second definition for the number to drift toward. The slow, unglamorous work that makes this real is definitional reconciliation: getting the registrar and the dean, or the program officer and the grantee, to commit to the same canonical meaning before anyone builds a chart on top of it. It’s tedious. It’s also the foundation.

For our ninth grader, this is the difference between at risk meaning the same thing in her building as in the one across town. But notice the limit of what reconciliation buys you: it makes the number consistent, not correct. A definition everyone agrees on can still point at the wrong thing. Hold that thought — it’s the third layer.

Authority: who acts, and when

Now the flag means one thing. Who moves?

One of the most common failures in any early-warning system is not a bad model — it’s a blown assignment. The signal fires and no one is named to receive it. This is a seam problem: the failure lives in the handoff between specialists, not inside any one of them. The cleanest sprinter on the team loses the race if the baton hits the ground in the exchange zone.

The instrument that fixes it is a seam contract: an explicit, written understanding (operational, not legal) of who acts on a signal, with what authority, on what cadence, and at what resolution. It turns integrated bytes into decision-ready intelligence by naming an owner for the recurring call. And because the call is recurring, the surfaces serve it directly: the teacher gets a progress-monitoring console tuned to this week’s action, the principal gets a program report, the superintendent gets the portfolio view. Three surfaces, one keystone underneath.

For any of this to stick, the function needs traction, and traction is a property of the seat. It belongs with the seat that holds legitimate cross-functional authority (often the COO, sometimes a Chief Impact Officer or equivalent integrator, in a smaller shop the CEO), but never inside a single technical or financial function by default, which will bend the data toward its own incentives: optimized for uptime, or for cost, but not for the cross-functional meaning a dean or a clinical director needs. For our ninth grader, the seam contract is the difference between a flag that lands on a named counselor by Tuesday and a flag that everyone could see and no one owned.

This is where the support systems live and the framework stops being linear. Once the flag fires and an owner acts, the response is an intervention: in a school, often a tier of support. But the support a student does or doesn’t receive becomes an input to the next risk signal. The factors are connected. The same variables (attendance, engagement, prior support, history) that predict academic risk are the evidence spine a funder uses to judge a program and the risk factors a behavioral-health team watches. A signal in one decision is a predictor in the next.

That recursion isn’t a footnote; it’s the reason the three layers can’t be run independently. A decision system isn’t a pipeline that ends at an action. The action becomes part of the data that trains the next signal. Which means a definition that drifts in the meaning layer doesn’t just produce one bad number — it teaches the next model to be wrong on purpose, indefinitely. What that does to validity is the next section.

Validity: canonical is not the same as true

Suppose you’ve done everything right. The flag means one thing. An owner acts on it on a known cadence. Every surface agrees.

The flag can still be measuring the wrong thing.

This is the hardest layer, and the one institutions skip. Construct validity asks a question the dashboard can never answer: is the system measuring the trait it claims to (academic risk, wellbeing, quality), or merely a surface proxy that correlates with it? Train an “at-risk” model on enough history and it will learn to predict the proxy: the zip code, the demographic pattern, the prior-discipline record. It will be accurate. It will also be measuring the wrong student.

The cost isn’t only that this is unfair, though it is. It’s that the institution now misallocates at scale: pouring intervention dollars at a proxy while the actual construct goes unseen: the student who is slipping but doesn’t match the historical pattern. You can defund a program that works and miss a cohort that needs you, both at once, with a perfectly accurate model.

And recursion makes this worse, not better. Once an intervention becomes a predictor, the model can no longer cleanly separate risk from service received from institutional attention. The student who finally got help looks “high-risk” in next year’s data because the system finally started watching her; the student no one ever served quietly disappears from what the model learns. Left alone, the system learns its own past behavior and calls it prediction.

The failure has a shape, and the shape travels. A behavioral-health risk model trained on prior service utilization doesn’t predict who is at risk; it predicts who the system has already served. Different sector, identical mistake — accuracy against a proxy, mistaken for measurement of the construct. When the same error shows up in a school and a clinic, you’re not looking at a domain quirk. You’re looking at a recurring failure mode, and the framework is what lets you name it before it ships.

Two disciplines guard this layer. The first is refusing the rolled-up average. A single number (the graduation rate, the program’s headline outcome) can hide opposing trends underneath it: a falling overall rate that masks a rising rate in one subgroup, the average quietly erasing the very contrast the decision rests on. Burden and disparity are different signals, and only disaggregation makes the resource choice explicit and honest: where the next dollar goes. The second discipline is asking what the measure is for: decision utility weighs the expected cost and benefit of the specific action, not just the statistical accuracy of the score. A validated flag that triggers no useful act is a validated waste.

For our ninth grader, validity is the question no dashboard asked: was the flag measuring her academic risk, or was it measuring her zip code with her name on it?

The stress test: agentic AI

Until recently, most decision systems still kept a human pause somewhere in the chain — someone read the report, interpreted the flag, judged it before anything happened. That pause was rarely designed. It was just there, a free safeguard no one had to budget for.

Agentic AI weakens that default. The unit of work shifts from answers (which a human reads) to actions (which an agent takes). In an agentic workflow the flag may no longer wait to be read; it enrolls the student, escalates the case, moves the resource. And every weakness we just walked through (a drifting definition, a missing owner, a proxy mistaken for a construct, a recursion no one is watching) now executes at machine speed, without the pause that used to catch it.

The fix isn’t a new procurement rubric. An autonomous agent is a specialist that moves faster than you can read, and a specialist needs a seam contract. The agent’s contract just has to make explicit what a human’s could leave implicit: an autonomy range (how much it may do unsupervised, from return only verified responses to act review-by-exception, set by the stakes of the decision, not the cleverness of the model); a reversibility envelope (how and when its action can be undone, and who is told when a record someone already acted on gets corrected); a named human owner who answers for what it does; and a consumption contract, so metered spend lands on the department that generated it. And like every other reader, the agent reads through the same semantic keystone — because an AI ungrounded in canonical definitions will cheerfully invent metric names and answer questions no one can reconcile.

The stress test is the proof. AI doesn’t introduce new requirements so much as it removes the slack that let institutions get away with skipping the old ones. Meaning, authority, and validity were always central. Agentic systems are the first thing heavy enough to make a hollow architecture fall down.

Design the rules for the failure mode

The best institutional rules aren’t written to describe normal play; they’re written to prevent predictable failure. Goodhart’s law (a measure that becomes a target stops being a good measure) isn’t a slogan for a poster. It’s a design constraint. If a measure will be gamed, the system has to protect the construct before someone hollows it out by chasing the proxy. That’s a whole essay of its own — see Why the rules look weird. For the decision-system, the corollary is simple: write the weird rule first.

There’s a payoff worth naming, and a temptation worth refusing. The marginal cost of storing and processing data has collapsed; the cost of making it mean something has not. That collapse tempts institutions toward a fantasy of total visibility: seeing each person whole. They can’t, and shouldn’t pretend to. The honest version is smaller and harder: stop mistaking the fragment you measure for the person in front of you. For years a student arrived in our systems as a postage stamp: a score, a category, a proxy for an enrollment number. The aim of the architecture isn’t to swap that thumbnail for a perfect portrait. It’s to keep the institution honest about how little of her it sees, and to make that partial view legible to a decision instead of merely stored.

What the system is for

A decision system is meaning, authority, and validity, holding each other up: a single canonical definition, a named owner for every recurring call, and a measure that tracks the real construct — stress-tested by AI and governed by rules built for the failure mode rather than the demo.

The aim was never a better dashboard. It’s an architecture that turns fragmented bytes into institutional action and keeps doing it after the people who built it have moved on.

Go back to the ninth grader. The flag didn’t save her; flags never do. The system did — if the definition held, so at risk meant the same thing in every building; if a named counselor owned the signal by Tuesday instead of admiring it on a screen; if the measure was tested, so it found her and not her zip code; and if the support she received fed back into the data as help given, not as fresh evidence that she was the problem. That last clause is the one most systems miss, and it’s why the layers have to hold together rather than take turns.

The flag was always the easy part. The system is the work.

This essay was written in July 2026 for the Analytic Bytes Library. It is the umbrella framework the rest of the library sits under: meaning, authority, and validity as the three faucet-layer failures, stress-tested by agentic AI. The ninth-grade early-warning flag is a composite drawn from the author’s practice across K–8 charter networks, a youth-mental-health foundation, a regional behavioral-health agency, a DC public charter school context, and Andhra Pradesh state systems. Organizational details are abstracted where appropriate.