Three trainings, three questions.

What the absence does to sorting

The requirements section is a claim of necessity. Ten years of data engineering is required in order to do what. Once the second half of that sentence is missing, the items in the list still have to do something, and what they do is sort.

Sorting on attributes is not a neutral fallback, and the corpus shows why. Roughly three quarters of those eight hundred and thirteen items are abilities and dispositions, which is the largest category and the least falsifiable. Nothing checks comfortable with ambiguity. What can be checked is the rest: tools, domain knowledge, years, degrees, a thin tail of certifications. So the sort concentrates in the part of the list that has a truth value, and that part is mostly about duration and credential. Nobody decided to rank people by years. Years are what is left once the unfalsifiable items are set aside.

One item in that corpus states the position cleanly: experience establishing enterprise governance structures with formal decision rights. Read it closely and the rights belong to the structures, not to the candidate. What is required is experience building the thing that carries them. This posting grants none of its own.

Kyle Brink argued in MIT Sloan Management Review that organizations declaring their hiring meritocratic have mostly never validated the instruments they use to measure merit. What passes for merit, on his account, is a set of unvalidated proxies: pedigree, tenure, an interview impression (Brink, MIT Sloan Management Review, August 2026). A degree is a proxy of that kind, asked to carry a construct nobody wrote down.

That is the gap Seven causes, one bit is written to close. Its repair asks every requirement to carry three things: what capability, what would count as evidence of it, and when it has to be true. Supply those and comfortable with ambiguity becomes checkable and sorts on its merits. Leave them out and it cannot sort at all, so the credential items sort by default.

Who writes the contract asks who inside the organization owns the definition in the first place. The question here sits between them: given that the decision went unnamed, what are the three being compared on, and what would it take to compare them on something else.

It does not follow that the three are equal, or that credentials carry no information. A credential carries a great deal of information about what somebody was trained to do. The failure is in using it to answer a question that was never asked.

Will it run, will it return, is it supportable

Three backgrounds turn up in the pool for a senior data, analytics or AI leadership seat: engineering, business, and measurement. That list is from working alongside all three rather than from a survey, and nothing here depends on how common each one is.

Here are the three, each at its strongest. Each training makes one of these questions automatic. None of them stops a person asking the other two.

Engineering training answers whether the system will work. Not whether it works once, on clean inputs, in a demo. Whether it holds under load, under change, under the person who inherits it in two years. An engineer who has run a migration knows things about failure that no amount of strategy reading supplies: which dependencies break silently, what a rollback actually costs on a Friday, why the thing that worked in staging did not survive contact with the production schema. The evidence standard is that it runs, and keeps running.

Business training answers whether this is worth continuing. That is one question, not several, and it is narrower than it sounds: given what has been spent and what has come back, does the next increment get funded. This one gets caricatured and the caricature is wrong. An MBA program is training in allocation under uncertainty; the case method is repetition on deciding with incomplete information and a clock. Sequencing and coalition belong to it because an initiative nobody adopts returns nothing, so adoption enters as a condition of the return rather than as a second standard. A general manager who has taken a program through two budget cycles knows how to kill an initiative that is not working, which is a specific skill and a rare one. The evidence standard is that the next increment gets funded on the strength of the last one.

Measurement training answers whether a claim is supportable. Given this evidence, what can you assert, and what can you not? That question sounds academic until you notice that it is the only one of the three questions that asks what a number is a claim about. The evidence standard is that the argument survives a reader who is trying to break it.

I mean a specialization inside research training, not the whole of it. A molecular biologist and a historian are both research-trained, and neither is necessarily trained to ask whether a metric supports a particular consequential use. The tradition that asks it by default is measurement and evaluation: psychometrics, program evaluation, assessment. Some people do it without ever training for it. Others hold the degree and never built the habit.

One thing about that training is worth separating from the credential, because it is the part that transfers. You are graded on stating the strongest case against your own claim, and you do it knowing a reader is coming who is paid to find what you missed. Neither half works alone. Audit and code review put that reader in place but never ask the writer to make the case against themselves. Many workplaces supply neither half, and a stakeholder review least of all, because nobody in that room gains anything by finding the flaw.

None of that is unique to a doctorate. An engineer who carried a migration through eighteen months has argued against her own plan in public. A general manager who held a turnaround together has done it in front of a board.

None of the three trainings is a general-purpose problem-solving ability. Each is a trained response to a recurring kind of problem, and the response transfers to problems of that kind.

So the sorting question is which of the three the seat has to answer first. That is a property of the work, and the sponsor knows it before anyone applies. It goes unwritten because the requisition form does not have a field for it.

What each training costs

Every one of the three trainings buys its strength by not training something else. I will take mine first, because it is the one I am least neutral about.

The measurement-trained hire can over-warrant before anyone has decided anything. The reflex is to make the construct, the inference and the evidence explicit before the number gets used. Where a measurement carries consequences for a person, that burden is the right one and the training is right to install it. In an operating seat it can turn a decision that needs a bounded provisional answer into one that appears to require publication-grade warrant. A quarter goes into defining something the organization would have accepted in a week, and the definition arrives so careful that nobody uses it. The worse version is treating the deadline as the variable. A warrant can always be strengthened, so there is always a reason not to close, and in a seat where the alternative to a partial answer is no answer that habit does real damage.

The engineering failure is that the specification sits outside the test. Testing checks the system against the specification. It does not check the specification against the decision. Software engineering has long separated verification from validation (Boehm, 1984); in practice, the tests that gate release are verification tests. A pipeline can pass every test it has and still deliver a number that will not carry the weight the executive puts on it. Good engineers interrogate semantics and downstream use, and many do it well. The discipline’s primary tests can all pass without that question having been answered.

The business failure is that the metric arrives as a given. The metric is what the board reads, so it becomes the object. Strong operators do challenge a KPI. What the training does not supply is a standing obligation to ask whether the measure still supports the reading being taken from it. I have watched programs get run well against a measure that had already stopped meaning what everyone assumed it meant. Nobody in the room was doing anything wrong. The question of whether the measure still held was not anyone’s question.

Read those three together and a pattern shows up. The failure of the first is slowness. The failure of the other two is a claim that nobody checked. Each training can be right by its own standard and still be wrong for the seat, which is why the seat’s standard has to be named before the comparison starts.

When the output becomes evidence

Many operational transformations deliver a capability, and you can watch whether the capability works.

A supply chain transformation delivers goods arriving on time. A finance transformation delivers a close that runs in three days instead of eleven. If the thing does not work, the organization finds out, because the failure shows up in the world as a shipment that is late or a close that overruns. The feedback is slow and noisy, but it exists.

A data, AI or reporting transformation often delivers a capability too. A pipeline that moves data, a platform that stays up, a retrieval layer that answers in time — those are capabilities, and you can watch them work.

The difference arrives at a specific point. When the output becomes evidence for a decision, the transformation stops delivering only a capability and starts delivering a claim. That threshold is where validity enters, and a great deal of data and AI work crosses it on the day it goes live. Validity is the third of the three things a decision system has to hold, after meaning and authority.

You cannot watch whether a claim is true by watching the system run. That is the difficulty once the output becomes evidentiary. The dashboard refreshes on schedule. The pipeline passes its tests. The model returns a score with four decimal places. Every operational signal says the transformation succeeded, and the number can still mean something other than what the decision assumed it meant.

I have written that failure up three times from three directions. The numbers don’t agree works the case where one metric is put to eight different uses and stops supporting most of them. What is this system measuring? works the gap between the construct named and the construct actually captured. Validity is the layer beneath responsible AI works what happens when the measuring instrument changes underneath a published number and nobody is required to say so.

Those are validity failures. The measure works and the inference does not.

Validity of interpretation and use is not the central object of either of the other two disciplines. Engineering has validation traditions of its own, and testing is one of them: a discipline for checking a system against its specification, with the specification arriving as an input. Business has audit, experiment design and the business case, disciplines for checking a return against a forecast, with the metric arriving as an input. In both, the validity of the underlying construct can enter as an assumption rather than as the thing the discipline is built to interrogate.

Educational measurement is the field that made the question explicit, and it did so across decades rather than in one paper. Messick established that what gets validated is the interpretation and use of a score, consequences included, rather than the instrument itself (Messick, 1995). Kane treats validation as an evaluation of that proposed interpretation and use: make the inferential chain and its assumptions explicit, then ask whether the evidence supports them (Kane, 2013, Journal of Educational Measurement 50(1), 1–73). That is the training. What it teaches you to catch is exactly what goes wrong when a number gets used as evidence.

Neither of them wrote about a requisition. The chain is theirs; reading a job specification as the instrument, and asking which standard of correctness that document leaves unnamed, is the extension I am making here.

Measurement training does not make somebody better at transformation. What it makes automatic is one question: does this number support the decision somebody is about to take.

And once the output is evidence, the question reaches the transformation itself, because the proof that it worked is produced by the very system under review.

Which means the three are not a sequence. The second question depends on the third. A return gets claimed on the strength of a number, so whether the spend returned is only ever as good as whether that number supports the claim.

The federal graduation rate is the example I keep coming back to, because nothing about it is broken. It counts students who started full time, first time, at one institution, and finished at that same institution. Part-time entrants are outside the cohort by definition. So are transfer-ins. A student who leaves and finishes her degree somewhere else does not become a graduate of the college she started at, in that rate (NCES, IPEDS Graduation Rates component). Nationally the cohort takes in about forty-seven percent of new undergraduates each fall (IHEP). That is a count of who gets measured, not of who finished. The rate is computed on those students and is silent about the rest. IPEDS added an Outcome Measures component in 2015–16 to count part-time and transfer students, but the headline rate is still the one read in the room. At an open-access college serving working students the share is smaller still, and the published rate gets read in the room as how well the college serves the students it has.

The arithmetic is right. The definition is public and anyone can look it up, which is the part people find hardest to believe once the gap surfaces. What fails is the step from the rate to the sentence said about it, and there is no test that goes red when that step is wrong.

The door and the seat are not the same test

A requisition is two documents, and nobody notices they are separate.

The requirements section is the door. It selects on attributes already accumulated: years, tools, degrees, headcount managed, titles held somewhere else. The seat is the work. It turns on one of the three questions, and which one it turns on is a property of what is already built and what the organization has already committed to.

Those two do not have to agree. In every requirements section in that corpus, attributes are named and a decision is not, so the part that sorts carries no statement of the problem those attributes are for. Forty-two documents say it nowhere at all. The other twenty-three do state it, in another section, written as a description of the work rather than as something a candidate is scored against. So it is present in the document and absent from the part that sorts.

That last count is not a filing complaint. A requirements section is an inventory of attributes, which is what it is for, and where the decision sits in the document is the employer’s business. It matters because the inventory is the part that sorts.

An organization may well establish the connection somewhere else: in an interview, a case exercise, a hiring manager’s head. None of it is in the document, which is all the candidate has before applying.

The consequence is not that the wrong people apply. The screen works, in the narrow sense that it reliably finds somebody who clears the door. Whether that person can hold the seat is a separate matter, because the two tests were never connected to each other.

What makes this expensive is the delay. Someone who clears an attribute screen looks correct for as long as the job is still being learned, and nothing in that stretch distinguishes learning the seat from failing to hold it. By the time the difference is legible, a good deal of a transformation has been spent, and the sponsor reads the result as a hiring mistake.

It was a specification mistake, and it was made before anyone applied.

A validation standard is not what is missing. Personnel selection has had one for decades. The SIOP Principles for the Validation and Use of Personnel Selection Procedures reached a third edition in 1987 and a fifth in 2018, the current one approved by the APA Council of Representatives and aligned to the 2014 Standards for Educational and Psychological Testing. Fred Oswald, who edits Psychological Methods, makes the point that those Principles govern a screening instrument whether or not the instrument happens to be a language model (Oswald, Working Things Out, August 2026). SIOP’s 2023 guidance on AI-based assessments already applies the Principles to these tools. A method for validating a selection procedure has been written down and published by the relevant professional society, through five editions. The documents in this corpus do not reach it.

A search can fail while selecting a strong candidate. The candidate clears a screen that looks valid, for a construct nobody specified, and the strength is what can mask the mismatch: a capable person may perform around it long enough that the failure gets attributed to execution rather than to the specification.

Which question does this seat answer first

Required for what? asks the requisition writer to add one line naming what the role settles without another signature. This is the second line, and it goes in the same place.

Of the three (will the system run, will the spend return, does the number support the decision), which one does this seat have to answer first? Not which ones matter. Which one is first, in the first year, given what is already in place.

If that keeps returning all of them, ask the harder version. Suppose the person you hire is excellent at two of the three and weak at one. Which weakness sinks the transformation anyway? That question has one answer, and the answer is the binding constraint you are hiring against.

That is a choice selection measurement already has a name for. A compensatory rule lets strength on two dimensions offset weakness on a third, and a conjunctive rule does not: some dimensions are hurdles you clear or fail regardless of the rest. Much hiring runs compensatory by default, because a list of attributes invites a tally. The question above asks the sponsor to say which of the three is not compensatory. I made the same move on a different object in The valid dollar, scoring a composite on its weakest inference link rather than on the average; the rule is borrowed from that literature and applying it to a seat’s three competencies is mine.

If the system is not built, the answer is the first question and you should hire an engineer. If the system runs and nobody will fund the next phase, the answer is the second and you should hire the general manager. If the system runs, the money is committed, and executives are already making decisions off numbers whose meaning nobody has argued for, the answer is the third.

One caution about order. Getting the first two wrong announces itself: the system stops, or the money stops. Getting the third wrong does not.

It also does not arrive all at once. Whether a number supports a decision can only be tested once someone uses it. But what gets collected, at what grain, defined how, is settled while the system is being built, and settled by default if nobody is assigned to it. So an engineer-first answer is right about the build and still leaves open who holds those definitions while the building happens. Four jobs, one loop is about what happens when nobody does.

The second answer has its own version. A general manager needs a number showing the last phase worked, and the one that gets picked is the one that will fund the next. That is what the job rewards rather than a failure of it. But a figure that has funded a phase is hard to retire afterwards, because it now sits in the record as the thing that worked. The valid dollar works that problem at fund scale.

What to select for

Where the third question comes first, the seat needs demonstrated capability to validate an interpretation and its use. Somebody with measurement training may have direct preparation for it. The selection procedure should still test the capability rather than read it off the credential — an essay that spends three thousand words on unvalidated proxies does not get to finish by recommending one.

If the honest answer is that all three come first, the requisition is describing more than one role. That is a real finding and it is better to have it before the search than eleven months into it.

Attributes tell you whether a candidate meets a requirement. Naming the question tells you whether the requirement was the right one to write.

Two things this does not do.

It does not tell you who to hire. A sponsor can name the problem correctly and still choose badly. Naming it correctly makes that error checkable afterwards, which is an improvement and not a solution.

And it does not close the seam it sits on. The candidate still cannot see the definition being applied to her. Publishing the sentence would change that, and almost nobody will, because a stated construct is a thing you can later be measured against.

What it does is smaller and worth having. It converts a comparison between three credentials into a comparison against one stated problem. Three people trained differently can be read against a problem. Against a title, only the accumulated proxies are legible, and none of them was validated for the job it is being asked to do.

Written September 2026 for the Analytic Bytes Library. No employer, client or vendor is named, per standing practice. The counts in the opening are from the sixty-five-posting corpus built for Required for what?, and the coding rules and adjudication lines are held with that corpus rather than repeated here. The SIOP Principles, Messick, Kane and the Oswald piece were checked against their published sources. The three trainings are read from the author’s own record of working alongside all three, not from a survey of practitioners, and the essay makes no claim about how common each background is in these seats.