Four jobs, one loop.

Measuring more has a bad reputation right now, and it has earned it. Organizations adopt another survey, another dashboard, another data platform, and the claims they can make at the end do not get stronger. Often they get weaker, because there is more to defend and nobody designed any of it against what would need defending. What got dropped was the reasoning part: how to design a measurement, and how to analyze it so that it serves the people running the program, the funder reading the report, and the measurement itself next round.

In one organization, research and evaluation designed the theory of change and the instruments, and program teams collected the data. My team, a senior data analyst and a senior data scientist, did the reporting. While I was there, program teams waited months for data they needed the next week to review with clients, and the learning loop ran on anecdotes and meeting notes. How much data was missing came to light only when someone ran an impact analysis. My role was written to hold the seams: near-term program reporting and long-term impact reporting on one infrastructure, and learning loops that would carry collection problems back to research on a regular cadence instead of once, at analysis time. The role was eliminated before that loop could close. Every metric is a claim follows what happened to the function after that.

In a school network, the role sat a level down under instructional technology and became system access. Job 2 survived as dashboards; job 1 went, because what got collected was what the systems already captured, and job 4 went with it. In a behavioral-health agency, the role sat under compliance, with grant reporting as the main deliverable. Job 3 survived in its narrowest form, the report the funder required, and the other three went, because the funder had already chosen the measures. In a fourth organization, the role was eliminated and the work folded into a platform build, which is the absorption pattern with engineering as the absorber. The dashboards got built. Nobody designed against a claim, nobody owed a partner a claim they could check, and nobody revised anything.

Across four placements, the loop never had an owner long enough to close. I argued in Where should data sit? that where a data function reports decides what it is allowed to become, and that it should report to the integration seat rather than to any single function. The principle carries over to measurement with more at stake, because a measurement function owes a claim as well as a number. The four seats above are four single functions. Each kept the job it understood and let the others go, because placement decides which of the four jobs a function can see. A seat inside one function cannot see the loop it belongs to; a seat above the functions can.

Four jobs across the measurement cycle

The four jobs form a loop. Job 1 feeds two uses side by side, and job 4 feeds job 1.

1. 01Design the measurement. Decide what to collect before collecting it, against the decisions and claims the evidence will have to support, limit or rule out. This is where the construct gets named and the instrument gets chosen or built. The design exists so the organization can know, at the end, what it may act on and what it may say.
2. 02Analyze so it serves operations. The people running the program can act on the result this month. A no-show pattern a clinician can read; a cohort a state team can reach this week.
3. 03Analyze so it serves the external party. A funder, a partner, a board gets a claim they can check, with the evidence stated and the strength matched to it.
4. 04Revise the measurement. The next round measures better because of what this round showed. Sometimes that is an item that stopped discriminating. More often it is a definition that drifted across sites, a collection protocol, a threshold, or a metric dropped. Sometimes the instrument is fixed by a funder or a statute and cannot be changed. The revision then moves to what surrounds it: who gets measured, what the number is compared against, how it is collected, and what claim it is allowed to carry.

I use measurement broadly here: the instruments and the metrics, and the evidence design around them (sample, comparison, counterfactual) that decides what the observations can support. Job 4 returns to job 1. That is the loop, and it is what separates a measurement function from a reporting function. A reporting function runs jobs 2 and 3 on whatever arrives. A measurement function decides what arrives, and changes it when it fails.

Florence Randari makes the same cut one job earlier: producing evidence and preparing it for use “are not the same thing.” Her five moves for preparing it are jobs 2 and 3 in working form (Randari, LinkedIn, August 2026). Nobody in her list decides what gets collected or revises the measurement. Her premise is evidence that already exists, so that is fair on her terms; on the terms of this note it is the whole subject.

Two debts pile up around a broken loop. Data debt builds upstream, where collection outruns design: fields nobody can define, instrument versions nobody can reconcile. Dashboard debt builds downstream, where display outruns interpretation and ownership: the number moves and nobody can say whether to act.

A different cause than the one on the table

Rashi Trivedi’s The Impact Decision Gap (LinkedIn, August 2026) is a recent statement of the problem from the funder side. The symptom is the same as here, and the cause is different. Her case is an evaluation that is technically strong, a report that is credible, and a decision that was almost predetermined before either arrived. Funding committed, targets announced, years of relationships to protect. Sometimes, she writes, “we don’t have an evidence problem at all. We have an institutional incentive problem.”

That is one room. In it, good evidence enters and gets overridden. The room this note is about is the other one. The evidence that enters was never built to answer the question being asked, because nobody held job 1 when the collecting started and nobody will hold job 4 when it ends. The report is credible about something adjacent to the decision. No incentive has to be corrupt for that to happen. Max Yurkofsky went looking for the gaming and distortion that Goodhart and Campbell predict, in school improvement teams in two districts, and did not find them. What he found were teachers gravitating to the more measurable parts of their work, with no pressure from leadership behind it (Yurkofsky, Substack, August 2026). Two districts is an observation. It is consistent with what this note would predict where nobody designed what to collect against the claim it had to support, and it does not demonstrate that. My reading is that it is a capability problem first, and behind it an org-design problem. The four jobs were split from each other, and the question of what the data licenses the organization to say fell between the pieces. That is a different cut from keeping the measurement function independent of the programs it measures, and I come back to the difference below.

The two causes need different repairs. Trivedi’s needs a decision named before the evaluation is commissioned, and she says so. This one needs an owner for the loop: one charter that holds all four jobs, whoever does the work inside each.

Construct to consequence, and back

The four jobs are validity reasoning turned into operating work. They are also the decision chain run as a job description: job 1 defines the construct, jobs 2 and 3 draw inferences from what stands for it, and job 4 is the return leg, the only one of the four that carries what happened back into how the next round measures.

Messick (1995) moved validity away from the instrument alone and toward the interpretations and uses made of a score, consequences among them. Kane (1992) made the chain explicit: a claim depends on the warrants that connect what was observed to what is concluded. Reading that chain at its weakest link is the operator’s rule I applied in The valid dollar, and The construct keeps moving runs on the same two references. Job 1 is the construct and the observation design. Assessment designers call this evidence-centered design: it starts from the claim and works back to the evidence (Mislevy, Steinberg and Almond, 2003). Jobs 2 and 3 are the interpretation and the use; they pull in different directions. Job 2 needs the signal this month, at the grain of a site or a caseload, and can act on something provisional. Job 3 needs stability, comparability and restraint about inference, because the claim will be checked. The same data can license “follow up with these twenty-two sites this week” and not license “the program improved outcomes.” Quality-improvement researchers drew this line decades ago, separating measurement for improvement from measurement for accountability (Solberg and colleagues, 1997). Job 4 closes the loop: what the interpretation, the use and their consequences showed about the measurement, written into the next design.

Read that way, splitting the jobs breaks the continuity the chain depends on. Research holds job 1, and its output is a validated scale that operations cannot act on this month. Operations holds job 2 and builds the dashboard on fields research did not design and cannot vouch for. Development holds job 3 and writes the funder report at whatever verb the deadline allows. Job 4 belongs to nobody, so the measurement does not change, and next year the same fields carry the same debt. Yurkofsky’s third dynamic is what the missing return looks like from the ground: practical measurement systems built to drive improvement, which the teachers found impractical and rarely used. A system nobody uses sends nothing back to whoever built it, and job 4 never starts. Each team is doing its job well (I have sat in all three). The chain is still broken, because the chain was never any one team’s job.

The valid dollar shows the cost at one fund. A pathway is capped at the contribution link because there is no counterfactual behind it. The fund buys a better instrument, which improves a link that was already strong, and the claim stays where it was. That is what a broken loop spends money on: the link that is easy to improve rather than the one holding the claim back.

Which split breaks it

An objection arrives here from anyone who has built an independent M&E function, and it is a fair one. Measurement is supposed to be split from the programs it measures. Put the people who count outcomes inside the team whose outcomes they count, and the numbers drift toward what the team needs them to be. So is this note arguing against independence?

No, because there are two different axes you can cut on. One cut splits the four jobs from each other: design into research, dashboards into operations, the funder report into development, revision into nobody. The other cut splits the function from the programs it measures: all four jobs stay in one charter, and that charter reports outside the teams whose numbers it audits. The first cut breaks the loop, and the second protects it.

One charter does not have to mean one team. Research can own the instrument and operations the dashboard. What cannot be distributed is the question of what the data licenses the organization to say. One accountable owner has to hold it and be able to send the answer back to job 1. Job 4 goes missing because nobody in the room is asking for it. Jobs 2 and 3 have someone who will, the program team this month and the funder at the deadline. Job 4 is owed to next year’s measurement, which sends no one to the meeting. A council can govern the work, the way the council in The contracts between systems governs who may change a definition; it cannot stand in for the person accountable for the answer, and a committee is often how that accountability disappears. In none of the four seats above did anyone hold it long enough for the answer to come back.

The strongest version of the objection is a two-way split: jobs 1 and 4 in one seat, jobs 2 and 3 in another, both under one charter. It is a reasonable proposal, because jobs 2 and 3 already travel together, and that pairing is what a reporting function is. I would expect it to hold on the way out and fail on the way back. Job 4 runs on what jobs 2 and 3 learned in use. That means the field that misled, the caveat the funder queried, the definition that drifted between sites, the dashboard the site team stopped opening. Little of that gets written down, so little of it survives a handoff between seats. A charter that makes the split anyway has to turn the return into an artifact those seats owe to job 4, with someone accountable for its arriving. The closest I have come is the first seat above: the jobs sat with different teams and the role was written to own the return, and it was eliminated before the return had run. So this is a reading of the mechanism, not a finding.

One posting makes the distinction cleanly, and it is worth quoting because whoever wrote it was trying to fill a seat. A head-of-data-and-M&E role at an Indian education nonprofit, as captured in May 2026, lists four things under one job. Design the data architecture “from field input to impact indicators — definitions, collection protocols, validation logic, and storage.” Build dashboards and report packs for the CEO, state teams and donors. Produce impact reports and evidence briefs. Ensure “field learning feeds into policy redesign, process improvement and product updates.” That is jobs 1, 2 and 3 under one head, and the learning that job 4 runs on. What the posting does not spell out is the last step, revising the measurement itself; its fourth item feeds learning into the program, not back into the indicators. The role reports to the founder and chief executive, which is the integration seat. The posting says three times that the function must be independent of the program teams it measures, and lists “sit under state programs whose performance it must assess” under what the role does not do. Its six-month success criteria include “at least two key program design changes in direct response to M&E evidence,” which is the learning half of the loop written as a hiring metric.

That is one document, and it stops one step short of the full loop. I have not found a second that separates the axes this cleanly. The distinction holds on its own logic. What the posting shows is that at least one organization wrote most of it down.

Job 2 already has a name

The for-profit world industrialised the middle of this loop and called it business intelligence: operational data turned into something a manager can act on this week, at the level they manage. Job 2 is that discipline in a mission-driven organization. Businesses fight over constructs too (churn, engagement and quality are all contested). The difference is the distance between what an impact organization can observe and the social outcome it has to make claims about. Add the evidence regimes, funder and statutory, that those claims have to survive. BI industrialised much of the operational middle. It did not solve the measurement-and-claim loop around it. In The contracts between systems I described the leader looking for institutional intelligence (a 360 view for the CEO, a number a district officer can act on this week). The integration ran, and the institution still could not make the decision. The gap there was governance: who reads the number, who can change its definition, who has to be told. This note takes up the other half: whether the number was ever built to be acted on.

Vehicles carry the middle

Data infrastructure and AI tools make production inside the loop cheap, most of all through its middle. They do not make ownership of the loop cheap.

A warehouse, a semantic layer and a dashboard tool make job 2’s output cheap. A reporting pipeline with a model drafting the narrative does the same for job 3. All of that makes it easier to skip matching the strength of a claim to its evidence. Bridgespan’s practitioner guide to nonprofit MEL, written from work in Africa, Asia and the United States, grants that technology can cut the burden of collection. The same sentence calls collecting more and more data “a temptation generally to be avoided” (Searle et al., The Bridgespan Group, 2025). Its test for a proposed metric puts a decision question beside the burden question: what decision can’t we make because we are missing this information today? That is job 1 as a screen, written before the drafting models made the middle cheaper still.

Now the organization adopts more measurement because collecting and displaying it costs almost nothing, and the claims do not get stronger, because nothing in the stack owns the loop. A model can propose what to collect, flag an item that stopped discriminating, and draft a replacement. What no tool holds is the decision that follows. Does the revised measure still represent the construct? Did the change break the trend line? Does the evidence now license the claim the funder is about to read? Those decisions sit at jobs 1, 3 and 4, and none of them belongs to the stack. They need an owner, and the cheaper production gets, the easier it is to run an organization that never names one.

The vehicles do their job. They carry jobs 2 and 3 faster than any team could. They also make a broken loop look like a working one, because the dashboards refresh nightly and the reports go out on time. The absorbed data role is about who ends up holding the plumbing when nobody is hired for it. The four jobs sit one step further along. Even a funded plumbing role, the curation and governance seat that note argues for, does not hold job 1 or job 4. Curation and governance keep the data sound; they do not decide what to collect or change it next round.

What to do with this

Two readers can use this directly.

If you write the requisition for a measurement, M&E or data role, list the four jobs and check which ones the posting holds. A posting that holds jobs 2 and 3 is hiring a reporting function, whatever the title says. If jobs 1 and 4 are held elsewhere, name where, and name who carries the question between them. If nobody does, the posting is the place to fix that, before the hire.

If you fund or govern an organization, ask four questions of its measurement, in this order. Who decided what to collect, and against what question? Who can act on the result this month? What claim can we check, and on what evidence? What changed in the measurement since last round, and who changed it? An organization that answers the first from research, the second from operations, the third from development, and the fourth with silence has split the loop. Its dashboards may still be very good.

The independence question stays separate. Ask it second. A function can hold all four jobs and still report outside the teams it measures; that is the arrangement the posting above describes, and it is the one I would write.

I have watched the charter that holds all four get written, and then split four different ways. Each split was reasonable on its own terms. The loop was the thing nobody wrote down.

Written September 2026 for the Analytic Bytes Library. The four placements are from the author’s own record; sectors are named and employers are not. The job posting is quoted as captured in May 2026 and the organization is not named. The Randari, Trivedi, Yurkofsky and Bridgespan pieces were checked against the published pages, with snapshots held in the piece’s source pack. The Messick and Kane citations are carried over from the AB measurement arc, where they were verified.