When measurement arrives after the build.

I talk with a fair number of vendors and early teams building products and platforms for health and education. The teams vary, but the shape repeats. There is someone who knows the domain, often a clinician or an educator. There is someone who owns the product and someone who builds it.

The questions I ask are the ones a testing program asks before it lets a score be used: what decision an output feeds, for whom it was checked, and what happens when it is wrong. Often the founders raise these questions before I do, and say they have been coming across them. That is the pain point they name, and it is a real one. They learn the questions when they sit down for buyer conversations: a pilot partner’s review, a procurement form, a clinical lead asking how the product was checked. By then the platform is built. The buyer’s side of the same conversation is in Partnership is a handoff, where the buyer’s team ends up absorbing what the vendor could not supply.

I come to that conversation from the measurement side, and from a change in my own work. At ETS, a validation study started from a clean dataset: a defined sample, scored responses, documented variables. In the health and education settings I moved into afterward, nobody handed me that dataset. Before any validation study could begin, I had to learn the platform and the systems behind it, and work out how to assemble data accurate and complete enough for the question being asked. That reconstruction was often slow and hard, and it came before the study, not in place of it. In my design work since, what a buyer asks for has usually traced back to a place in the build where something was not captured, logged or built in.

These teams need an engineer first, and they say so. That is reasonable, since nothing can be tested before it exists. What they rarely have is someone whose work is to decide, during the build, what the platform should record. By measurement work I mean deciding what each output claims about a person, what record would show that the claim holds, and checking that it does. Product analytics, which counts what users did, is a different job. This note is about products whose output is a claim about a person: a score, a flag, a recommendation. Where the product makes no such claim, most of it does not apply. J-PAL draws the same line for AI programs: engagement, monitoring and technical-performance data are useful, but “these signals alone cannot show whether an intervention improves outcomes in the real world” (J-PAL, AI Evidence Playbook, 2026).

Who drives the build

My own work has moved the way these products are moving, from tests that measure a person at one point in time to systems that bring in more about a person over time. In a standardized assessment, where a score decides a placement or a pass, measurement tends to drive the build. The construct, the items, the scoring rules and the evidence for reliability and validity come first, and much of that evidence is published. Engineering shapes the result too, especially in adaptive and automated testing, but the claims come first. Assessment design has a name for this order, evidence-centered design: decide what the scores must support and what evidence would show it, then design the tasks that produce that evidence (Mislevy et al., ETS, 2003).

When I worked on automated essay scoring at ETS, one scoring model drew on several kinds of expertise before it touched anyone’s score. People who defined the construct wrote the tasks and rubrics. Natural language processing scientists built the scoring features. The NLP scientists tested their features on modest development sets, with the measures their field uses, such as precision and recall. Psychometricians then checked how the scores behaved on large operational samples and across subgroups, and validity researchers tested what the scores meant and for whom. The criteria the model had to meet were set before it was evaluated: agreement with human scores, differences in score distributions, differences across subgroups, and association with external measures. I co-wrote the paper that set those criteria out in print, and its conclusion was about “the a priori establishment of performance expectations” before a system is used operationally (Ramineni & Williamson, Assessing Writing, 2013). The teams I described at the start cover that ground with three roles: the domain, the product, and the engineering.

Many newer platforms tend to run the other way. Some mental-health platforms now go beyond a PHQ-9 score and its cut points, which screen for depression and indicate how severe the symptoms are. They combine chatbot messages, daily wellness check-ins, measures of social needs, comorbidities, primary care records, and the PHQ-9 and GAD-7, gathered in different formats at different times. The aim is to follow a patient’s state as it changes and flag risk in near real time, rather than waiting for a questionnaire at the next scheduled visit. Learning platforms do the same with class work, quizzes, assessments and behavior on the platform, to decide that a student needs more help, and sometimes to inform calls that carry the consequences a standardized test once carried, such as placement or progression. Student-success platforms at universities often do the same at the institution’s scale, combining course-site activity, grades, attendance, registration and financial-aid status into a flag that a student may not return next term, and routing that student to an advisor. The reason is real. In EAB’s 2026 survey of student-success leaders, early alerts ranked among the strategies seen as most effective for keeping students enrolled, and the report’s principal author described students “who have a 3.5 GPA and never missed class—and then they vanish and we don’t know where they went” (Inside Higher Ed, September 2026). In these systems engineering usually drives the build, and the measurement work a high-stakes standardized test would have had at the start comes later, if it comes.

The difference is not only one of order. A PHQ-9 uses technology only to deliver it, and the same nine questions work on paper. The risk a platform’s flag estimates exists apart from the platform, but the flag itself does not. What the platform logs, how it handles missing inputs, how it combines them and when it computes the flag are part of what the flag means. In these systems engineering decisions are measurement decisions, whether or not anyone treats them that way. Automated scoring was the same case in large-scale, high-stakes testing, which is why the scoring engine itself had to meet the criteria set out above, from agreement with human scores to differences across subgroups, before anyone’s score depended on it.

The combination is a new decision system. It may claim a construct, such as depression severity, a prediction, such as risk of crisis within a month, or a rule for what to do next. Whichever it is, it needs evidence of its own. Evidence about each input helps, but it does not establish how the combination performs. A PHQ-9 validated for depression screening carries no evidence about a risk flag it has been folded into. Assessment has worked on this problem for decades. Evidence-centered design treats combining evidence from many tasks into one inference about a person as a modeling problem in its own right. It asks for an explicit account of how each piece of evidence bears on the claim (Mislevy et al., 2003). In health, a combined risk flag is a clinical prediction model, and there are reporting standards for developing and validating such models (Collins et al., BMJ, 2024). Either way the combination needs its own answers:

1. 01What at risk means, and by when.
2. 02How inputs on different scales and schedules are put together.
3. 03Whose data goes missing.
4. 04Whether the flag holds for each group.

In the teams I talk with, this work is usually not near the top of the build roadmap.

The case for combining is real. Much of its promise is earlier help: a flag can reach a patient before a crisis, or a student before a failing grade, while the consequence can still be avoided. Models built from health records and routinely administered depression questionnaires predicted suicide attempts and deaths better than existing risk tools did (Simon et al., American Journal of Psychiatry, 2018). The question is whether the combination gets its own development and testing. That promise is also why the evidence matters, because an early flag decides who gets help first.

That help is also meant to reach many people at once. These platforms are built for reach, as automated scoring was built for large-scale testing, and scale raises the bar rather than lowering it. A standardized test controls its conditions: the same tasks, the same timing and the same administration for everyone. A platform that follows a person through daily check-ins, messages and records gives up most of that control, and every input it adds is another source of variation and another way for data to go missing. Missing data is rarely random. The people who stop answering check-ins may be the people a flag most needs to see, as the pilot described below shows. So seeing more of a person can help, and it can also mislead, and which one happens depends on whether the combination was tested for it. A system that sees more and controls less needs more measurement work, not less.

What the story is asked to cover

When a buyer, a funder or a pilot partner asks what the product does for its users, an early team answers with what it has: a demonstration, some usage figures and an account of the clinical or teaching design. The team learns which framing a hospital responds to and which one a school district responds to, and it reshapes the story around the platform it already built.

Some of that is ordinary market learning, and useful. The difficulty is what the story is being asked to carry. A story cannot cover the gap left by data the product does not have. If the platform did not record something, the team cannot produce evidence for a claim that depends on it, however the story around the claim is told. The people asking are the ones deciding whether to put the product in front of patients or students. A user has to make a decision before acting on what the product shows. If that decision cannot be made because the evidence is missing, the gap is in what was built, and in what is being sold.

So the first cost of late measurement is evidence. Time is the second cost, and it follows from the first: months go into adapting the story to the market, and later into reconstructing data that can be recovered only slowly and in part.

Where the story stops working

In my experience the story holds until one of three points.

1. 01Entry into the market. A hospital, a school system, a university or a government service will often ask for evidence before it signs, through a procurement questionnaire, a clinical review or a data-protection assessment. Guo and colleagues describe this step for digital health companies: early sales tend to follow close relationships with an early adopter, but “wider implementation requires robust evidence of benefit” (Guo et al., npj Digital Medicine, 2020). Education buyers are moving the same way. Among the 53 percent of states that have published guidance on buying AI-enabled tools, 79 percent now include the evidence-based quality indicator, up from 45 percent in 2025. Yet 68 percent of state leaders say it is difficult to verify vendors’ claims about AI products independently, and 52 percent say their state collects no data on whether ed-tech tools are used or effective (SETDA, 2026 State EdTech Trends Report, a survey of state education agency leaders in 45 states and one territory). University technology leaders have moved the same way: their top issue for 2027 is “determining where AI adds real value” (EDUCAUSE, 2027 Top 10).
2. 02Share in a crowded market. A buyer comparing similar claims looks for the product that can show its claim.
3. 03Extension to a new use or population, such as adolescents, a school, a community college after a selective university, or a second language. The story travels easily; the evidence was collected for the first population and does not come with it.

At each point the team has to revise what it built, and how much depends on the roadmap. A short roadmap aimed at quick commercialization gets small fixes: a new field, a new log, a study for the buyer in front of it. That can be the right call. But the evidence it produces fits one buyer and one population, so it tends to keep the product in its first lane, and it goes stale once the product or the market moves past it. A longer roadmap needs foundational fixes, because the platform has to produce evidence continuously, for more than one population, and feed what it learns back into the product as the market changes. That keeps the product able to move into new uses without starting its evidence over.

What can be rebuilt and what cannot

The roadmap decides how much of the platform gets rebuilt. How much evidence can still be produced depends on what was recorded, and on what kind of evidence is needed.

Evidence for the product’s claim can usually be built late. Definitions can be written after launch, thresholds re-examined, and the next pilot designed as a study, with the records the first one lacked. It costs time, and in my experience much of that time goes into reconstructing the data before the study can start.

Evidence about what already happened is different. Which safety checks failed open during the first pilot, letting a message through when a check could not run? What happened to the people who were escalated to a human, and did anyone follow up? Who stopped using the product, and what was going on for them? If the platform did not keep these at the time, and no other system did, no later study recovers them. A new study can only describe the pilots that come after it.

In a mental-health app pilot I worked on, the platform logged logins and assessment submissions, so we could see engagement fall with each new group of registrants and see that many patients stopped partway through the month. It did not record why they stopped. The follow-up calls to bring people back lived in a spreadsheet outside the platform, so the platform’s record ended where the follow-up began. For one patient, the app’s account record showed a last login date, while the usage logs held no activity for that patient at all, so the two records disagreed.

The strongest objection comes from health research. Real-world evidence is assembled after the fact from records kept for other purposes, such as health records and insurance claims (Corrigan-Curay et al., JAMA, 2018). That shows evidence can be built late when a second system recorded the event. Some product events do leave traces elsewhere, in a partner’s records, a message system or an audit log, and a careful reconstruction looks there first. Which of the product’s own safety checks failed open is rarely among them.

One study, or validation the platform runs

Once a team decides to fill the gap, it can commission a single validation study. Or it can build validation into the platform, so that the platform keeps what each claim needs and rechecks the claim as the product and its users change.

The single study is familiar, and for some claims it is right. Its limit is that the product keeps changing after the study ends, and the study does not move with it. Researchers in behavioral intervention technology noted more than a decade ago that a technology’s shelf life may be shorter than the time it takes to evaluate it. They proposed evaluating such technologies continuously as they change (Mohr et al., American Journal of Preventive Medicine, 2013). Guo and colleagues add that because a study typically takes two to three years, “evidence published today may reflect a product that has been updated and refined multiple times” (Guo et al., 2020).

Validation built into the platform costs more during the build, because someone has to decide what each output claims, what record would support it, and what should trigger a recheck. That cost is repaid at each of the three points above, because the evidence already exists when the buyer asks.

Stakes decide how much of this has to come before deployment. A low-stakes feature with an easy way back can carry its validation through monitoring after launch. Crisis detection, clinical decision support, anything involving children, and anything that makes a consequential call about a person without a human reviewing it cannot. That is the calibration argument in Validity is the layer beneath responsible AI, applied to a team deciding what to build first.

Doesn’t the team already have this covered?

The strongest objection is that the team already has the people: a domain expert who knows what good looks like, user research on what people want and can use, and analytics on what they did. Plenty of successful products were built without anyone trained in measurement. The lean startup method puts measurement inside the loop: build something small, measure how people respond, learn, and change it (Ries, The Lean Startup, 2011). For claims about adoption, usability and willingness to pay, that loop produces real evidence.

Each is needed, and each answers a different question from the one a buyer in health or education eventually asks. Analytics records only the behavior someone chose to log, and as J-PAL notes, those signals do not show that anyone is better off. A district buying a product does not show that it improves learning. None of these asks whether the product’s output supports the decision it is used for, for whom, and with how much error.

That work can sit with someone already on the team. What matters is that it happens during the build. It is the first of the jobs in Four jobs, one loop, designing what to collect against the claims it will have to support, applied to a product before its first pilot. One caution: the person who defines what the product should detect is not well placed to be the only person checking whether it does.

Some teams answer with governance instead: a named clinician or educator who reviews the product and can block a release or override an output. That settles who is accountable for a decision. It does not show that the output measures what it claims, for whom, and with how much error, and the oversight has effects that need measuring too. Blocking too much hides what the system would have done; blocking too little lets errors through; blocking that falls more on one group than another is a fairness question of its own. Each override belongs in the records. Validity is the layer beneath responsible AI draws the line between oversight and validity for responsible-AI rubrics in mental health. A team needs both, and they are different work.

The questions I ask did not start with me. Testing programs have worked through rater disagreement, cut scores, population shift and scoring drift for decades. A project I worked on with a university writing program, presented at the National Council on Measurement in Education annual meeting in 2014, calibrated automated scores for writing placement against how students later performed in their writing course. The usual target is human ratings of the placement essays themselves. Short, timed essays on familiar topics, which students can prepare for, draw on a narrower part of writing than course assignments do. That can be enough for a placement test meant to sort fluency and control, and less so for one expected to say how well a student will argue. Which target fits depends on what the scores are used to decide. Product teams meet these problems one at a time. Having the work on the team means reaching the next question before a pilot partner or a procurement form raises it.

What to record before the first pilot

This part is for founders, and for whoever on the team ends up owning the work.

Deciding what to record is a small piece of work compared with a validation study, though building the records into the platform takes real engineering. Both have to be done before the first pilot, because the first pilot happens once. European law asks high-risk AI systems to “technically allow for the automatic recording of events (logs) over the lifetime of the system” (Regulation (EU) 2024/1689, Article 12). Logs of that kind trace the system. The records below keep that trace and add the people the system acts on.

Before a pilot starts, I would want seven things in place, adjusted to the claims the product makes. Three are decisions written down before the first user, starting, as evidence-centered design does, from what each output claims.

1. 01The claim and the decision. For each output, the claim it makes, the decision it feeds, and who makes that decision.
2. 02The outcome. The criterion each claim will be checked against, and the baseline or comparison it needs.
3. 03What counts as completing. A definition of completion and expected use, so that stopping can be told apart from finishing.

Four are records the platform keeps from the first day of the pilot.

1. 01The output and its version. What the system produced for each person, and which model, thresholds and content were live at the time.
2. 02Failures and fallbacks. Every check that could not run and what the system did instead (including each time it failed open).
3. 03What people did with it. Every escalation or handoff to a person, whether anyone acted on or overrode the system, and what happened next where that can be known.
4. 04Who was reached and who stopped. Who the pilot was meant for and who used it, described the same way, and who left before completing, and when: a patient who disengages, a student who stops out. A denominator and group-level characteristics are often enough.

Keep the minimum each claim needs, at the level it needs, on the legal basis the setting requires, whether that is HIPAA, FERPA or GDPR. None of these needs a validation study. The written decisions can be revisited later; the records cannot be reconstructed if no system kept them. Synthetic or simulated cases can check that the platform keeps these records correctly before real users arrive, but only once the claims are written down, because the claims say what to test for. They cannot stand in for the records, or for evidence from real users.

Putting the study off while demand is still uncertain can be a sound strategy. Putting off the decision about what to record is the part that cannot be made up later.

The timing follows from stakes first. Where the product informs a consequential decision about a person, such as crisis detection, a clinical recommendation, a course placement in a school, or an early-warning flag that puts a university student on an academic intervention, the validation work starts before deployment, and the pilot can be designed as the study. Where stakes are low, buyer demand and the time an outcome takes to show set the pace. If the first buyer will ask for evidence before signing, as hospitals, school systems and government services often do, the study comes before the pilot. If the outcome takes months to observe, the study has to start before a buyer asks, or the answer arrives a year late. Otherwise it can wait until a buyer asks. In every case the records are designed into the build and switched on for the first pilot, since any later study depends on them.

The questions founders keep meeting in buyer conversations can be answered during the build, before any buyer asks them. The first is what the platform will keep about the people it acts on, from the first day of the first pilot, so that the team can later show each decision the product informed was supported by evidence. Once that pilot has run, part of the answer is no longer there to collect.

Written October 2026 for the Analytic Bytes Library. No company is named. Published sources were checked against the sources themselves in October 2026. The mental-health app pilot is described from the author’s own internal analysis, which is not public and is not cited.