Before any AI runs. Before any ATS filter fires. Before the first recruiter opens the first resume — most hiring teams skip a step that determines whether their entire screening process will work. They don't define what they're actually looking for.
They write a job description. They post it. They let the applications come in. And then they screen against a mental model that exists only in the head of whoever is doing the reviewing that day — inconsistent, undocumented, and impossible to calibrate.
This is the most expensive hiring mistake most teams make. It's also the most fixable.
The gap between requirements and criteria
A job description is marketing. It describes the role to attract candidates. A screening criteria framework is a rubric. It defines how you evaluate the candidates who respond.
These are not the same thing, and treating them as the same thing is where most screening processes break. The JD says "5+ years of experience in a data-driven environment." That's a requirement. The criteria question is: how do you evaluate whether a candidate has meaningfully used data in their work? What counts as "data-driven"? What's the minimum bar? What would a "clearly meets this" candidate look like versus a "borderline" candidate?
Without answers to those questions, you don't have screening criteria. You have a prompt that different reviewers will interpret differently — and produce different shortlists from the same candidate pool.
The JD is what you ask for. Criteria is what you evaluate against. Confusing the two is where most shortlists go wrong.
Why criteria definition matters more when AI is involved
AI screening amplifies your criteria. Good criteria, applied consistently to 500 resumes, produces a shortlist that reflects actual fit. Vague criteria, applied consistently to 500 resumes, produces noise at scale — with high precision and high confidence.
This is the garbage-in, garbage-out problem applied to hiring. The AI cannot intuit what you actually want from what you wrote in a JD. It evaluates against what you told it to evaluate against. If you told it "5+ years experience" without defining what experience quality looks like, it will count years. If you told it "strong SQL skills" without specifying must-have vs. nice-to-have, it will weight SQL the same way for a role where SQL is critical as it does for a role where it's marginal.
The investment in criteria definition before you run a screening pass pays back every time you run one. Teams that skip this step spend the time on the back end — reviewing a shortlist that doesn't quite match what they wanted and trying to figure out why.
A criteria framework that works
Questions to answer before you configure any screener
These four questions surface the criteria that actually matter for a role — as opposed to the requirements that ended up in the JD because someone copied them from a similar posting:
-
1What did the last successful person in this role actually do in their first 90 days? Not what the JD says. What they actually did. The skills and experience that enabled that work are your real criteria.
-
2What would "hired but wrong" look like? Define the failure mode. A candidate who passes all keyword filters but fails in the role usually fails on something specific — a missing skill, the wrong experience level, the wrong domain. Make that explicit in your criteria.
-
3Which of the JD requirements would you genuinely waive for the right candidate? Whatever you'd waive is a nice-to-have, not a must-have. Move it to the nice-to-have list. Must-haves only belong there if absence would make you decline even an otherwise strong candidate.
-
4Is this role primarily skills-driven or experience-driven? A specialist role (data engineer, security architect) is primarily skills-driven — what you can do matters more than where you did it. A leadership role is primarily experience-driven — scope, ownership, and trajectory of past work matters more than the specific skills listed.
Common criteria mistakes and how to fix them
Too many must-haves. Every must-have reduces the candidate pool. A well-designed role has 3–5 genuine screen-out criteria. More than that usually means some of them are actually nice-to-haves that ended up in the wrong column. The fix: for each must-have, ask "would I decline an otherwise perfect candidate for not having this?" If the answer is no, move it.
Years-of-experience as the only criterion. "5+ years" is a proxy. It's a proxy for relevance, for level, for capability — none of which it reliably measures. What you usually mean is "has done the kind of work this role requires, at the level this role operates." Translate that into skills and experience quality criteria instead.
Not updating criteria between roles. The criteria you set for the last data analyst hire don't automatically apply to the current one. If the team's needs have evolved, the criteria need to evolve. Treat criteria configuration as a per-role exercise, not a template you carry forward.
Skipping the weight conversation. Skills and experience are both important — but rarely equally important. For this specific role, which matters more? Setting explicit weights (Skills 40%, Experience Quality 30%, Eligibility 30%) forces that conversation before screening starts, rather than implicitly resolving it through reviewer gut feel after the fact.
An AI screener doesn't fail to understand your role. It succeeds at evaluating exactly what you told it to — which is the problem, if what you told it was vague.
Good criteria are the product you're building
Every hiring process produces two things: a hire, and a criteria framework. The hire is what you need now. The criteria framework is what makes the next hire better, faster, and more consistent.
Teams that invest 30 minutes in deliberate criteria definition before each screening pass don't just get better shortlists — they build organizational knowledge about what good looks like for each role type. That knowledge compounds. The fifth time you hire a data analyst, you have four calibrated criteria sets to learn from. The first time, you're starting from scratch.
AI screening makes criteria operational at scale. But the quality of the criteria is yours to define. No AI can tell you what matters for a role you're building. That's a judgment call — and it's the most important one in the hiring process.