● Library

Notes from the decision system.

Essays, artifacts, and field notes — on decision-system architecture, measurement, and the operating teams that ship them.

Decision-system architecture is the umbrella frame for this library — composed of four disciplines: Measurement, Integration governance, AI systems, and Data foundations. Each piece sits in one of the four; the arc pill on each card names which.

The decision chain is what the diagnostic pieces trace — construct, representation, inference, decision, action, consequence, and the return leg back. Each piece names where it stops working; the line under the subtitle says which link.

Also live: Public data as a decision system — a working demo of the framework applied to federal releases, with the honest question attached to every panel: did the number actually move?

When measurement arrives after the build.

Teams building health and education platforms usually have someone who knows the domain, someone who owns the product and someone who builds it, and rarely anyone whose work is to decide what the product should record. In testing, measurement tends to come first; in these platforms it tends to come last. Evidence for a claim can still be built from the next pilot on; evidence about what already happened cannot, if nothing kept the record at the time. What to write down and record before the first pilot, and a rule for which validation work cannot wait.

Once accumulation is cheap.

Personal AI infrastructure at practitioner scale becomes engineerable when measurement design and a decision layer sit under it, so its consequential judgments can be inspected rather than merely accumulated. Six weeks of counting an operating layer: 230 artifacts, 100 still carrying weight, one lifecycle taxonomy, and a four-move architecture that holds it up.

Partnership is a handoff.

Vendors, implementation partners, customer success managers and funders all call the relationship a partnership. What the word usually names is labor the vendor cannot supply and the buyer’s team absorbs — the org logic, and the working definition of enrolled or engaged. The fix is a line item for that labor, and a seam defined before signing.

Integration governance

The delta nobody budgeted.

A state reissues attendance codes. A counselor leaves. A vendor's release lands two quarters late. Each triggers the same jobs — reconfigure, re-teach, re-map — and each lands on whoever is present. The recurring cost in human-serving organizations isn't running systems but keeping them matched to a world that won't hold still. AI makes that translation cheap. It also becomes one of the change events it's being bought to absorb.

Data foundations

For the record.

In schools, clinics, campuses and nonprofits, systems exist to produce reports for someone else, so the people doing the work keep their own spreadsheets. AI makes it cheap to translate that local knowledge into explicit rules. It does not supply the person who decides whether the rules are right, or the discipline that reconciles the two records.

Three trainings, three questions.

Forty-two of sixty-five senior data postings name what the person must be and never what the role may decide. Three trainings answer them — engineering, business and measurement — and with no decision named, the only thing separating them is which proxies each accumulated. They were built to answer different questions: will it run, will it return, does the number mean what the decision assumes.

The gate runs one way.

We Must Pace the Frontier proposes the architecture of an acceptance test for risk: a capability trigger, an evidence requirement, and an independent evaluator with a desk, a badge, and the right to publish. OpenAI said within days that it would take the evaluators too. The benefits the essay opens with, cured diseases, faster growth, a floor that rises, have evaluators and methods but no equivalent default gate, and pacing the frontier cannot supply one, because it cannot make a two-year outcome arrive in six months.

Every metric is a claim.

The evaluation report arrives after the cohort has passed. The dashboard arrives after the event, and no layer carries its warrant. Between the two there is nothing an organization can act on and defend. A validated link between an early signal and a later outcome should be treated as infrastructure: the semantic layer carries the claim, the evidence and date behind it, and when it must be rechecked.

Four jobs, one loop.

A measurement function has four jobs: design what to collect, analyze so operations can act this month, analyze so the funder gets a claim they can check, and revise the measurement for the next round. Split the four across research, operations and development, and nobody holds the question of what the data licenses the organization to say.

Required for what?

Sixty-six senior data and AI leadership postings, read against the three anchors of a decision system. Ninety-eight percent name what the person must bring. Thirty-five percent name what the person may decide. Eleven percent name how anyone will know the person's calls were good.

The spend trap.

A CFO can have a clean chart of accounts and a program team can have a real evidence spine, and neither one produces a defensible cost per outcome. The missing piece is the shared, governed keys that let spend and outcomes divide same-scope. The finance-side companion to The Reach Trap and The Valid Dollar — the wire between them.

Seven causes, one bit.

Educational measurement requires an instrument to argue for itself before it decides something about a person. A résumé screen carries no such argument. Three failures — the construct is unspecified, the outcome does not identify its cause, and the screen's errors are invisible to its author — and three remedies, one for each.

The floor is the frontier.

Amodei's five areas mostly raise the ceiling; economic development is the one that raises the floor, and it belongs in the causal model from the start. But the floor rose is a claim, not a result. Beneficial is a claim about an outcome — defined before the build, and re-measured as the loop accelerates.

Validity is the layer beneath responsible AI.

Responsible-AI frameworks in mental health check privacy, accountability, equity, and monitoring. What they don't ask is whether the system measures the mental-health construct it claims to. Four validity questions sit underneath the responsible-AI floor and decide whether the rest earns its place.

The valid dollar.

A response to Dollars in Pockets, the common-unit impact metric from NextLadder Ventures and GitLab Foundation. Applies Kane and Messick's validity discipline to composite ROI figures, and proposes scoring them on the weakest inference link rather than the average. Separates the verbs a number earns — supports, contributed to, produced — by the strength of the counterfactual behind it. Closes by asking that funds publish per-pathway warrant beside the composite.

Extending the gates.

A method note from the PDDS project. The first six panels used clean administrative counts and point-in-time policy levers. Extending into survey data (weighted estimates, confidence intervals, suppression) and continuous levers (Title V, ESSA) forced a revision of the gates. This note documents what changed, and why the revisions tightened rather than loosened the discipline.

Auditing an AI-native practice.

Analytic Bytes (AB) spent ninety days operating as an AI-native practice. This is the public report from the audit that followed, and the case for practitioner AI governance as a discipline the field has not yet named clearly. Introduces HITL/AITL as a signal-informed proxy for the load-carrier axis, and demonstrates methodology documentation done inline.

My relationship with AI.

A personal field note on twenty months of working with GPT, Claude, and Gemini across a career transition, a brand build, and a re-entry into public-facing work. Five stages, from vending-machine outputs to multi-tool selection, name what the relationship actually looked like. The working pattern and the voice discipline transferred across tools; whether the rest was maturity or the tool is still open.

Before it was called AI evaluation.

The classical automated-scoring literature already worked through most of what modern AI evaluation is now rebuilding — fairness across subgroups, adversarial testing, a priori thresholds, risk-tiered deployment, population-matched corpora, prompt curation, cross-domain generalization, and responsible framing. Eight bridges, with the receipts.

The absorbed data role.

Data work in mission-driven organizations decomposes into three buckets — entry, interpretation, and curation/governance — and the third one, the specialized semantic-layer role, rarely has a funded home. It gets absorbed by whoever is capable with numbers: a math teacher, a clinician, a director of instructional technology. This piece names the absorption pattern across four sectors and argues for the three-layer architecture that would replace it.

Data foundations

When the stakes are the mission.

When enterprise AI gets a signal wrong, someone loses money; when mission-driven AI gets it wrong, someone loses the intervention meant to reach them. This piece names the evaluation gap opening as mission-driven organizations adopt frontier AI, proposes a five-part essential minimum, and points toward shared infrastructure that could make it sustainable at sector scale.

We used to settle for thumbnails.

For most of human history, every institution carried you as a thumbnail: a name, a category, a score. That compression wasn’t malice, it was budget, and the price of the full picture has now collapsed. The choice institutions face isn’t whether to keep sorting faster or to hold people at higher resolution; it is whether they can answer the validity question underneath it.

Who writes the contract.

The contracts that govern integrated data have to be written by someone, named, inside the institution. But the hiring rubric fuses two roles — engineering and governance — into one job description and screens only for the first. The person who can actually do the work almost never scores well on the rubric written to find them.

Data foundations

Two bets, one institution.

Every institution standing up AI in 2026 is making a model-layer bet, vendor-anchored or federated, whether or not it has named it as one. For institutions running both administrative and research-flavored work, the honest answer is a deliberate hybrid calibrated by domain. What makes the hybrid coherent is a third set of contracts nobody writes by default: the seam contracts between the two bets.

Integration governance

Why the rules look weird.

Most rules in a system describe normal behavior. The handful that look weird — that interrupt, that constrain, that pre-resolve a specific exploit — are the rules doing the structural work. Five sports rules, five structural lessons for institutional design.

Integration governance

The reach trap.

A program officer, a portfolio manager, and a CSR lead can all report reach, but none can say whether it worked or where the next dollar should go. The usual diagnosis is a reporting problem, but the cause is a category error: a system of record for activity asked to behave like a system of record for outcomes.

The decision system.

Most “data problems” are decision-system problems. Three things fail at the faucet — meaning, authority, and validity — and detach any one from the other two and the system produces confident output that cannot close a decision. This is the umbrella framework that unifies the Analytic Bytes library.

Integration governance

Functions don’t run plays.

The mismatch between horizontal work and vertical org charts is structural. Multiple honest forms can carry it: councils that deliberate and hand playbooks back to functions, flash teams that merge for the deliverable, standing squads drawn permanently from across functions. All rely on the same seam contract underneath. Without that contract, none of them holds.

Integration governance

It’s not a communication issue. It’s a blown assignment.

Many cross-functional breakdowns get diagnosed as a communication issue and answered with another meeting. They’re a football play instead: alignment, assignment, execution. Each phase fails differently, and a communication-issue diagnosis collapses all three.

Integration governance

Plumbing got upgraded. The water didn’t.

The data integration layer is mostly solved — the pipes have been industrialized. The work that remains is closer to running a water authority: sourcing, testing, pressure, authority, who’s allowed to drink. The unglamorous part isn’t the pipes anymore. It’s the water.

Integration governance

The contracts between systems.

Integration is two questions stacked on top of each other: do the bytes move, and when they arrive, can anyone act on them. Institutions have answered the first across three eras and skipped the second, and the agentic era is about to make that gap consequential.

Integration governance

The numbers don’t agree because the words don’t.

Two people read different student-persistence numbers from the same data. The governance council is functioning. The framework looks complete. What's broken is definitional, and the work to fix it is the work most councils skip.

Integration governance

The take-home test.

More than a dozen interview take-home tasks, done cold across a decade, read as one experiment. The same few failures showed up in almost every one — and none of them was a skills gap.

Data foundations

What is this system actually measuring?

Universities have built the scaffolding to govern AI and skipped the evaluation layer. The measurement-science question every adopted system should face — what is this actually measuring, and is that what we meant?

Actions, not answers.

Agentic AI produces actions, not answers — and the human checkpoint that came free with every answer is gone unless you design it back in. Why agentic adoption is a decision-system question, not a technology one.

Burden, disparity, and the next dollar.

Burden and disparity are two different signals in the same CDC mortality data. The priority list you build from one is not the list you build from the other — and a framework that shows both changes where the next prevention dollar goes.

Where should data sit?

Who owns data infrastructure is one of the org chart’s most muddled questions. The fix is not a better title; it is a principle — report to the integration seat, never to a single function.

Data foundations

Stitching the layers.

Why national education data, classroom assessments, and local instruments are most useful stitched together — and what the integration architecture looks like.

Data foundations

Grounding the AI layer.

Where AI belongs in the modern data stack, and the single contract that keeps every AI feature honest.

When GenAI redesigned my dashboard.

A GenAI redesign of my own dashboard came back uglier — and clearer. What that taught me about data teams and AI tools.

Three surfaces, one keystone.

Why BI tool selection is the last decision, not the first — and the three reporting surfaces most analytics products owe their audiences.

Data foundations

Artifacts

Diagrams, tables, and frames from the analytical work. 25 and counting.

The decision chain

The decision chain

Decision-System Architecture — the four disciplines

Decision-System Architecture — the four disciplines

The Contract at the Seam

The Contract at the Seam

The Decision System — reference architecture

The Decision System — reference architecture

One Architecture, Three Stacks

One Architecture, Three Stacks

The Agent System

The Agent System

See all 25 artifacts →

More entries coming. Have a topic you want me to write about? Send a note.