Every metric is a claim.

At a national mental health nonprofit the role I held was written as data strategy and impact analytics, reporting to the chief design and impact officer. After it was eliminated, the same responsibilities were advertised again under research and evaluation. By my count, eighteen of the twenty-three responsibility statements in the posting I was hired against are still being advertised across the three that replaced it. What changed was the verb. Lead became inform; design and maintain became leverage.

The organization kept its evaluation function and its reporting function. What it could not do, in the time I was there, was act this quarter on a number it could also defend. I cannot speak for what happened after I left, and the only evidence I have is the postings, which divide the work further than it was divided before. That is the gap this essay is about. It is not the ownership gap that Four jobs, one loop takes up. It is a gap in time. The evidence that would answer the question exists at two moments, and neither moment is the one the decision is made in.

Both arrive after the fact

Start with the impact evaluation, which by design reports after the cohort has finished. That is what the method is for. A randomized trial or a matched comparison answers whether the program changed the outcome for the people who went through it, and the outcome has to have happened for the answer to exist. For a workforce program that means a year of employment data. For a housing program it means twelve months without a shelter re-entry. The report is credible about people the organization has already finished serving.

A dashboard arrives after the event. Enrollment yesterday, attendance this week, a no-show pattern a clinician can read this month. It is current, and in most systems the warrant for reading it is nowhere in the layer that carries the number. A dashboard can hold a validated risk score; what the usual architecture does not hold is where that validation came from, for whom it holds, and what it permits. Nothing in the layer says what the number is allowed to be read as. Whether attendance in month two tells you anything about employment in month fourteen is a claim, and the dashboard does not make it. So the organization gets one number it can act on and cannot defend, and one number it can defend and cannot act on.

The evaluation field knows this. J-PAL’s playbook for AI-enabled programs says that AI systems generate rich engagement, monitoring and technical-performance data, “but these signals alone cannot show whether an intervention improves outcomes in the real world” (executive summary, last updated 17 February 2026). That sentence states the problem from one side. The field split monitoring from impact evaluation on purpose, and the split is reasonable. What happens next is organizational rather than intellectual. In practice the evaluation gets a design, a counterfactual, a pre-registered outcome and a warrant for its claim. Monitoring gets governed as a data-quality problem (accuracy, completeness, timeliness). Those are properties of the data. Validity is a property of the inference — the failure a decision system names third, the one that survives good data and clear ownership — and in the organizations I have worked in it became explicit only on the evaluation side, never on the half of the system that runs every day.

The funder side has noticed the gap in time from its own end. Lucy Brainard, writing in Stanford Social Innovation Review about how foundations evaluate the capacity building they pay for, proposes three horizons: satisfaction at the end of the engagement, behavior change at six to twelve months, and performance at a year or more. The middle horizon, she says, is the one funders routinely skip, and it is the one where the learning is. That middle horizon is the number between evaluations, seen from the grant side.

One argument, two halves

Kane’s argument-based approach (1992) gives the general principle. Validity attaches to an interpretation and a use rather than to an instrument, and it is established by a chain of inferences: from the observation to a score, from the score to the wider domain, from the domain to the construct, from the construct to the decision. Each link needs a warrant. The valid dollar read that chain at its weakest link for one fund’s evaluation spend, and The construct keeps moving followed what happens to the chain when the construct itself shifts under a measurement architecture that does not.

Translate the chain onto a data stack and the pipeline steps can be read through the same logic. The collection protocol and the validation rule have the structure of a scoring claim. The dashboard rollup by site or cohort has the structure of a generalization claim; it asserts that this aggregate speaks for those people. The semantic-layer definition, where an aggregate of observed fields is read as “housing stability,” has the structure of an extrapolation, from what was counted to the construct the organization cares about. The funder decision the dashboard drives is the use. Every one of those steps changes what the number means, and in most stacks I have seen none of them is treated as a claim. I am borrowing Kane’s scaffolding. His inferences do not describe a data pipeline one for one, and they do not by themselves say how an early signal is tied to a later outcome. That empirical problem has its own traditions, and they split along the same line as the two claims below. Prognostic modeling and leading-indicator work address whether an early signal predicts a later outcome well enough to act on. Surrogate-endpoint validation addresses the much stronger question of whether movement in an intermediate measure can stand in for movement in the final one. Kane says a claim needs a warrant, and those traditions say how one is built. None of them puts the warrant in the layer the organization’s dashboards read from.

The link the dashboard cannot warrant on its own is that one: from an early signal the organization can observe every week to the later outcome it exists to produce. Two different claims can sit on it, and they need different evidence. The first is operational: month-two attendance predicts month-fourteen employment well enough, in this program and this population, to flag risk and direct attention. The second is an impact claim: movement in attendance can be read as movement in employment, or as the program producing it. A prognostic signal licenses the first without licensing the second. Attendance may predict employment because it travels with transport, health, family support or prior work history, and raising attendance may move none of those. A dashboard can be worth a great deal with only the first claim, so long as nobody reads it as the second.

Whether evidence establishes either claim depends on what the study measured, and the study does not have to come first. A signal can be named at the outset and collected from day one. More often it is found later. Someone mines the history the system has accumulated, notices that month-two attendance predicts month-fourteen employment, and the predictor is added to the pipeline and captured every week. That is how most leading indicators are born, and the essay has no quarrel with it. What it insists on is the step after. A predictor found in existing data is a hypothesis. The data it was found in cannot warrant it, because a search over enough fields will always find something that predicts, and the strength it shows there is inflated by the search; that is overfitting, and it is as familiar in a dashboard as in a model. The warrant arrives when the relationship holds out of sample, on a cohort it had no part in choosing, with the claim written down before anyone looks. So the statement that survives is narrow: a signal may enter the stack by design or by discovery, and either way it carries no license until it has held up on data it was not found in. Discovery is cheap, and the infrastructure makes it cheaper. Confirmation is the study’s job, and nobody’s charter says so.

That is the division of labor the two halves were never given. The study establishes what a continuous signal may be read as, and which of the two claims it earns. The infrastructure carries the signal, between studies, at the grain a program director or a state lead acts on. The number then carries a dated warrant, and the date matters, because the conditions that supported the relationship change. Populations shift, eligibility rules change, a trusted staff member leaves, the program itself is redesigned in response to what the last study showed. Each of those can move the link. The evidence does not become false on a date; what lapses is the permission to rely on it without checking. Somebody has to decide when that check is due. That decision is governance work, and it sits in neither the evaluation team nor the data team as either is chartered today.

There is a known objection here, and it strengthens the case. Acting on a signal changes the process that produced the relationship; that is Goodhart’s and Campbell’s warning, and it applies to a month-two number a program manager is now steering by. The relationship may survive the intervention or break under it. Either way the original warrant can no longer be assumed, and the signal becoming a target is itself one of the conditions that puts a recheck due.

The requisition got there first

Organizations are already hiring for the join, even where no charter describes it. Eight postings since May, seven of them captured this week, describe the seat. Four are worth reading closely.

A pan-African education and workforce group is hiring a head of impact to lead “the evolution of our Impact function from a reporting and monitoring function into a world-class evidence, learning, validation, and impact intelligence capability.” The posting wants randomized and quasi-experimental methods, tracer studies and longitudinal research. It wants the same person to strengthen “data pipelines and verification systems” across surveys, platform data, the CRM and employer feedback. It asks for an impact team that is “proactive, responsive, and embedded across the organization rather than functioning as a retrospective reporting unit.” And it writes the dated-warrant idea straight into a dashboard requirement: data products that can “distinguish between actuals, estimates, projections, and externally validated outcomes”. That is four grades of warrant on one screen, written by a hiring manager.

A New York social-services nonprofit with a staff of nine hundred is hiring a vice president of program evaluation and quality assurance. The seat sets “the vision of business intelligence and performance analytics,” shares responsibility for “the organization’s data ecosystem and analytics architecture” with the chief information officer, and runs quality assurance, program evaluation and incident management. The seat reports to the vice president of IT. The placement, under IT, is the question Where should data sit? was written about. And the incident half is fully built out in the text (reporting tools, workflows, dashboards, regulatory standards, trend analysis) while the outcome half is “evaluation findings” to be translated. The risk claim has continuous machinery; the benefit claim has a report. That is the asymmetry The gate runs one way describes at the AI frontier, sitting inside one department of a homeless-services agency.

A funder this library has already written about is hiring for the calibration itself. The co-author of the impact metric The valid dollar responded to wants an impact modeling analyst to build the ROI models behind its target of more than $100 in lifetime earnings per dollar spent, with “appropriate counterfactual assumptions,” and in the same seat to manage “the Foundation’s core datasets of grantee impact data”. One duty appears in no other posting in the set: “Compare modeled expectations with realized investment outcomes and use findings to strengthen future models, assumptions, and investment decisions.” That is the recheck, written as a job. What the posting does not say is when. Assumptions are refined “based on investment outcomes, new data, research findings, and organizational learning,” a list of triggers with no schedule and no threshold.

The fourth comes closest to the method. A senior program officer for data and AI infrastructure at a large foundation is to identify and broker the dependencies between two portfolios, among them “the need for evaluation approaches to be designed in concert with the data systems that will collect outcomes”. That clause is the join, named. If the study is to establish the link the dashboard will carry, the study has to be designed with the dashboard’s signal in it, whether that signal was chosen at the outset or found in the data later. The posting names that as a coordination problem between two portfolios, to be brokered. No charter carries it as a method.

The other four postings put evaluation, dashboards, governance and data infrastructure into one seat and disagree about where it reports. A New York anti-poverty organization places it under the chief program officer. A large family foundation places it under the head of finance and operations. An Indian education nonprofit places it under the founder and chief executive, and asks the same seat for real-time dashboards and periodic evaluation cycles. And a second foundation’s principal officer for measurement, inside the president’s office, is to maintain a “ladder of outcomes” with leading indicators on it, without saying what licenses an indicator to stand on the ladder. Add the social-services nonprofit above, under IT, and that is four reporting lines for one seat. Where should data sit? argued that where a function reports decides what it is allowed to become, and this one has not been decided.

Eight postings I went looking for cannot show a trend. What they show is the same shape appearing across eight organizations and three reporting lines, before its operating method has a name. I have not found that method in evaluation or data practice. Two regulated fields run a version of it. Bank supervisors’ model-risk guidance asks that every model start from a clear statement of purpose, that its results be checked against actual outcomes, and that it be revalidated as often as its risk warrants (Federal Reserve, OCC and FDIC, SR 26-2, 2026). Drug regulation lets a surrogate that is only “reasonably likely” to predict benefit support approval, on condition of a confirmatory trial; if the trial fails, the surrogate “would no longer be accepted for this use” (FDA explainer, 2018). Neither reaches the dashboard number a program director reads as an outcome. The evaluation literature has the warrant and no infrastructure practice. Semantic-layer practice is good at recording what a metric means and rarely records what claim the metric is licensed to support. Whoever takes these seats will be writing the join themselves. The organization that eliminated the seat I held moved the role under research, where the evaluation half is comfortable and the infrastructure half is somebody else’s. It is the pattern Who writes the contract? describes: an impact-anchored placement that holds while its champion is there and dissolves back into the silos when the champion leaves. The twenty-three statements did not go to one seat. The evaluation half went to two seats under research, and the infrastructure half went to technology. The seat I held had sat on the infrastructure and driven the semantics on it, so that the program team could stop mining its own numbers out of instruments and spreadsheets, and research could read the same numbers for impact reporting and the learning loop. Now technology runs the stack, research keeps its own definitions for the impact reporting, and nobody holds the layer that says what a number is licensed to mean. The postings above move the other way. Both directions exist in the same year, and neither has the method.

Why the constrained organization needs it most

The objection is that this is a rich organization’s problem. A foundation with a data science team can afford a ladder of outcomes and a recheck schedule. A homeless-services agency with nine hundred staff and a $140 million budget cannot. It should keep its evaluation function, keep its reporting function, and get on with the work.

I think the opposite, and the reason is time. Every participant passes through once, whoever funds the program. What money buys is slack: a next cohort, a budget to redesign for it, room to absorb the wait for a fourteen-month outcome. Where slack is scarce, the cost of waiting is harder to absorb, and the organization acts on the month-two number anyway, with or without a warrant. The person in the shelter bed, the young adult six months out of foster care, is served now or not served, and by the time the outcome study reports, that person is gone. A dashboard that says attendance is falling in month two, carrying an operational warrant from a study that tested it, is the difference between an intervention in month three and a finding in month fourteen. The organizations with the least room are the ones for whom a warranted early signal is worth the most. They are also the ones least likely to have anyone chartered to build it, which is why they end up where The reach trap left them: back-breaking collection, reach to show for it, and no claim about what moved.

That is the practice I have built across higher education, K-12, mental health and MEL. In everything I do I am infrastructure-practice native, building from construct to consequence. I want the modern stack and the infrastructure as part of my practice alongside the measurement, not either/or. And I care about the consequence for the humans at both ends: the frontline worker providing the service and the person receiving it. I sit in the seams between research, technology and program, and what I work well inside is a box of constraints. Give me the constraints of a nonprofit and I get creative with the tools, the stack and the validity discipline, whether the job is an integration, a dashboard or a signal somebody can act on. Messy data is easier to work through when construct-to-consequence tells you which of the mess matters. The evaluation-only version of this discipline cannot by itself serve the decision in month two. The infrastructure-only version cannot defend what the month-two number means. The person in month two needs both.

What AI changes, and what it does not

Data infrastructure and AI make the everyday work cheap: collecting the data, building the dashboard, drafting the report. Four jobs, one loop already argues that cheaper production does not supply an owner for the reasoning around it. The narrower question here is how often the link can be re-established.

A recheck is expensive today because it is a study. Someone has to design it, collect the outcome, wait for it, and analyze it. That cost is why organizations run one every few years and treat the result as permanent. AI does not remove the wait. Month-fourteen employment still arrives in month fourteen, and no model compresses it into month three. What has changed is what happens once cohorts mature continuously and the outcome arrives through administrative rails: wage records, enrollment feeds, employer platforms, the CRM the organization already runs. The pan-African posting above lists “financial or KYC rails, LinkedIn/digital trace data” as sources it expects its impact function to use. With rolling cohorts and those feeds, the relationship between month-two attendance and month-fourteen employment can be re-estimated each quarter, where volume permits, on the cohorts whose fourteenth month has now arrived. Automation reruns that analysis against the semantic layer at the grain of a site. The re-estimation stops waiting for a bespoke study cycle. It does not stop waiting for the outcome.

What no tool decides is whether the recheck is due, and what to do when the answer changes. Those are the questions The construct keeps moving raised about the measurement architecture, and they land here on the warrant. When the re-estimate shows the link has weakened, somebody has to decide whether the population shifted, the program changed, or the indicator was never as good as the first study made it look. Then they decide whether the dashboard number keeps its license, loses it, or drops from an impact claim to an operational one. That decision is validity reasoning. A model can draft the memo. It cannot own the claim. The one piece of empirical work I owe on this is the simulation logged in my idea log as Twenty cases and a rhythm: how fast a warranted signal decays without a recheck rhythm, and what the rhythm catches. Until that exists, the cadence argument is a design claim and not a measured one.

What to write on the metric

Semantic layers already carry what a metric means. The contracts between systems argued that the definition, and who may change it, belong there. This essay asks the layer to carry one thing more. For a MEL lead or a data lead who owns a semantic layer, the adoption surface is three fields on every metric that is being read as if it said something about an outcome.

1. 01The claim it is licensed to support. Stated as a claim, in the organization’s own words, and graded. “Month-two attendance is validated as an early-risk indicator for month-fourteen employment” is one claim. “Rising attendance is evidence that employment is rising” is a different and stronger one. If nobody can fill this field, the metric is a count, and the dashboard should say so.
2. 02The evidence behind it, and the date. The study or analysis that established the relationship, its design, the population it was established in, the strength of the relationship, and when. Two states of this field look alike on a dashboard and are not: found in historical data, and held up out of sample on a later cohort. Say which. A citation, an internal analysis with a date, or the word none. The pan-African posting’s four grades (actuals, estimates, projections, externally validated) are a coarser version of this field, and a good one.
3. 03When it must be rechecked, and who decides. The condition or date that puts a recheck due: a program redesign, a change in eligibility, a population shift above some threshold, the signal becoming a target, or a fixed interval. And the name of the person who owns the decision when it fires. Operationally this is an expiry, and what expires is the permission to rely on the evidence unchecked. Without the owner, the trigger fires into the same silence Four jobs, one loop describes; with it, the field forces the decision that neither the evaluation team nor the data team is chartered to make today.

Three fields go on the metric, in the layer the dashboards read from. An evaluation report in a shared drive cannot carry them, and a data dictionary describes the column, not the claim. A funder can ask the same three things of any dashboard number a grantee shows them. If a grantee reports month-two attendance at 82 percent and cannot fill field 2, the funder knows what that number is: a count of attendance, accurate and current if the data quality holds, and nothing yet about employment. The valid dollar has the table for how strong a claim in field 1 the evidence in field 2 can support; this essay is about where the three fields live, and field 3 is what keeps them current.

Two floors away

At the national nonprofit, the reporting prototype I built ran a model against the semantic layer and mapped its output to the theory-of-change domains, so that an impact narrative could be drafted from live evidence rather than written after the fact and back-justified with the metrics that happened to move. It worked. What it could not do was tell the reader which of those live numbers had ever been shown to say anything about the outcome in the domain heading above it, or which claim each one was licensed to support. The prototype had the carrying half of the argument and not the warranting half. The evaluation team, two floors away, had the warrant and no way to put it into the layer the prototype read from. The program team between them got no number at all. Nobody could decide whether to act on the month-two signal or wait fourteen months for the outcome, so the program did its own data management, pulling together what it could to give its clients some insight and to shape delivery for the months ahead. No seat was chartered to carry the warrant across. The requisition that replaced the role moved it under research, where the warrant is comfortable, and the semantic layer is technology’s.

The evidence establishes the warrant. The infrastructure carries it. Somebody decides when it must be checked again. Those three sentences are the whole method. Split them across two seats, as the organization I left did, and the warrant falls between them. Put them in one seat, as the postings do, and that seat still needs the third sentence written into its charter: who decides when the evidence must be checked again. None of the eight says. Until one does, the number on the dashboard is still a claim, and nobody has signed it.

Written September 2026 for the Analytic Bytes Library. The eight job postings were captured between May and 14 September 2026 and are held as snapshots in the piece’s source pack. None of the eight organizations is named. The organization in the opening and the reporting prototype are from the author’s own record and are not named either. The J-PAL quotation was checked against the playbook’s full text, and the Brainard piece against the published article; the simulation named in the AI section is logged in the AB idea log and has not been built.