--- title: "we don't know how to hire. we're just very confident." description: "After 50+ hires, I doubt more references mean better predictions. A case for humility, real work samples, and hiring good humans." tldr: "References can help, but more of them don't make hiring predictive. References share narratives, experience ages fast in AI, halo effects mislead, and nobody's sample is statistically meaningful. Reduce the gap between what you think someone can do and what you can observe. And hire good humans." date: 2026-10-11 tags: [HIRING, LEADERSHIP, CAREER, OPINION] draft: false author: "Nikola Balić" topics: [Hiring, References, Recruitment, Leadership, Talent assessment] entities: [Brett Berson] answers_questions: - Do more references make hiring decisions better? - Why does years of experience predict less in AI roles? - What should hiring optimize for instead of prediction? - What matters most after 50+ hires? ---
hiring advice is autobiography dressed up as methodology.
i recently read [a thoughtful argument from brett berson](https://x.com/brettberson/status/2107861788557762941) that references might be the single most valuable tool in hiring. not the usual five-minute calls where someone asks whether john was a good employee and gets the shocking revelation that john was, in fact, wonderful. the author proposes something much more rigorous. deeper questions, different perspectives, conversations throughout the hiring process, and as many as 12–15 references for executive positions. but something bothers me. not because references are useless. they're not. but because we seem to believe that collecting more information about someone's past makes us significantly better at predicting their future. and i'm not sure that's true. i've hired more than 50 people throughout my career, probably more, across different roles and companies. i've tried different approaches, made some great hires, some questionable ones, and a few that probably deserve their own documentary. i've learned quite a bit. mostly how much i still don't know. ## twelve references. one story. let's say you're hiring an executive and you speak with 12 people who've worked with them. sounds thorough. but what if eight of those people worked at the same company, under the same leadership, within the same organizational culture? what if they share similar incentives, internal politics, or simply the same version of events? you might hear eight consistent stories. that doesn't mean you've discovered the truth. it might mean you've discovered the company narrative. more opinions don't automatically mean more independent information. and references aren't neutral observers either. they're participants in the story. a manager might think someone was difficult because they constantly challenged decisions. the employee might think they were doing their job by questioning decisions that made no sense. both could be right. both could be wrong. the company might have been a complete mess. it happens. people leave companies for all sorts of reasons, or simply because life is short and there are other things worth doing. now add confidentiality agreements, professional courtesy, personal relationships, bruised egos, and the reasonable desire not to discuss someone else's career with a stranger. good luck getting the complete picture. and the problem becomes even more interesting with younger candidates or people working in new fields. if someone has worked at one company for six months, or two companies over two years, where exactly are we supposed to find 12 meaningful, independent references? we can find 12 people. that's not the same thing. references can provide useful insights. recurring behavior, how someone handles conflict, whether people trust them, and the circumstances in which they've done their best work. but that information needs interpretation. a reference isn't a camera recording what happened. it's another human remembering what happened, through their own lens. and humans are not objective creatures. especially when other humans are involved. ## yesterday's experience, tomorrow's problems. the shelf life of some professional experience is getting short. we're hiring people to work with technologies, workflows, and capabilities that barely existed a couple of years ago. someone who spent the past six months intensely experimenting with new tools, building things, breaking things, and figuring out what actually works might be more useful for a particular role than someone with five years of adjacent experience. not always. experience matters. judgment matters. knowing how to build systems, navigate complexity, and work with other people matters. some of those abilities become more valuable as technology changes. but years of experience, relevant experience, current capability, and the ability to learn are four different things. we keep treating them as one. and then we ask someone who managed the candidate two years ago to evaluate how suitable they are for a job that looks nothing like the one they had. the challenge isn't just finding people who know how to do the job. it's finding people who can figure out how to do a job when the job itself keeps changing. and that requires something references are not particularly good at measuring: the ability to learn, unlearn, adapt, and create something that wasn't there before. someone's past is useful information. it's just not a particularly reliable map of a future that hasn't been invented yet. ## everyone is an expert. especially at things they're not experts in. the halo effect. we assume that because someone is excellent at one thing, they're probably excellent at other things too. a great engineer becomes a team lead. a great cto becomes a ceo. a founder who's brilliant at fundraising is assumed to be brilliant at running a company. sometimes everyone discovers, rather expensively, that these are different jobs. being a great ceo of a ten-person company doesn't automatically make you a great ceo of a hundred-person company. different scale, different problems, different skills. the same applies to hiring. just because someone has hired a lot of people doesn't mean they're good at hiring. including me. we all have our preferences, experiences, blind spots, and stories we use to explain why something worked or didn't. and we're particularly good at explaining things after they happen. one suggestion in the reference-checking argument was to evaluate the credibility of the references themselves. look at their careers, understand their standards, even ask other people how well calibrated their judgment is. which sounds reasonable. until you realize we're now checking references on references. who checks the people checking the references? and who checks those people? at some point, perhaps somewhere around the fourth generation of reference checks, we should probably just hire someone. the deeper issue is that we're adding layers of subjective judgment and assuming each layer brings us closer to objectivity. sometimes it does. sometimes we're just becoming more sophisticated at convincing ourselves we're right. ## the evidence problem. it's largely anecdotal. including mine. most companies simply don't hire enough people into sufficiently similar roles, under comparable conditions, to make statistically meaningful claims about which hiring methods work best. a startup that hires five executives over three years doesn't have a meaningful dataset for proving whether 12 reference calls outperform three. even 50 hires across different companies, roles, managers, and stages don't tell you much about the predictive accuracy of one particular method. the environment keeps changing. the expectations keep changing. the people doing the hiring keep changing. and in ai, sometimes the entire job description changes before we've finished debating it. we only observe the performance of people we actually hire. we don't know how the rejected candidates would have performed. perhaps the person with mediocre references would have been your best employee. perhaps the brilliant interviewer would have been a disaster. perhaps someone you rejected because they lacked three years of experience would have figured everything out in three weeks. we'll never know. and even when we do hire someone, what exactly are we measuring? retention? revenue? productivity? innovation? team satisfaction? someone might leave after six months having created value. someone else might stay five years and contribute very little. someone might be brilliant but fail because their manager was terrible. or because the company changed strategy four times and somehow expected everyone else to remain perfectly consistent. how do you separate a good hire from a good environment? how do you establish causality? you generally can't, at least not from a handful of outcomes. none of this means there isn't serious scientific research on hiring. there is. structured interviews, job-relevant assessments, and other selection methods have been studied across organizations. that evidence deserves attention. but research across many organizations is different from an individual's belief that their personal hiring process is optimal. experience gives us useful heuristics. it doesn't automatically give us proof. and there's a difference between saying "this has worked reasonably well for me" and "this is the best way to hire people." one is an observation. the other is a much bigger claim. hiring advice is autobiography dressed up as methodology. usually with a framework attached. ## maybe we're evaluating the wrong half of the relationship. another thing that bothers me is how much of the hiring process focuses on evaluating the candidate. their skills. their experience. their personality. their references. their perceived shortcomings. fair enough. but what about the company? what about the manager? we like to talk about culture fit. but culture is not an objective standard of quality. sometimes not fitting into a culture is a sign of good judgment. sometimes the person asking all those uncomfortable questions is the person the company needs. sometimes they're just difficult. knowing the difference is the hard part. people perform within systems. change the system and you might change the outcome. we've all met someone who struggled in one environment and became great in another. the reverse happens too. yet our hiring processes often treat professional performance as an inherent property of the individual, like height or shoe size. something you can measure, record, and carry from company to company. it isn't that simple. ## less prediction. more learning. first, i wouldn't turn any of this into another universal hiring framework. we have enough of those. the right process depends on the company, the role, the risks, and how expensive or reversible a hiring mistake might be. hiring an intern is different from hiring a cfo. some roles require extensive verification. others benefit much more from seeing someone work. but i've increasingly come to appreciate one principle. reduce the time between what you think someone can do and what you can actually observe them doing. give people realistic problems. see how they approach uncertainty. see how they communicate, how they use tools, how they respond when something doesn't work. when appropriate, use short, fairly compensated projects or practical assessments rather than endlessly discussing hypothetical situations. and once someone joins, be clear about expectations. give them feedback. create the conditions for success. find out whether the collaboration actually works. there's an old piece of advice about hiring fast and firing fast. i understand why it exists. the longer you avoid confronting reality, the more expensive things become, for everyone involved. but speed without judgment or empathy is just another way to make bad decisions faster. people aren't software subscriptions. the goal shouldn't be to maximize turnover. it should be to learn quickly, communicate honestly, make timely decisions, and treat people fairly when things don't work out. references can be part of that. they just shouldn't become a substitute for our own judgment or our responsibility to create an environment where people can succeed. and ideally, candidates should be investigating us just as seriously. because this isn't an audition for the privilege of working at our wonderful company. it's a relationship. both sides are making a bet. ## hire good humans. after hiring more than 50 people, i've noticed something about the ones i remember most fondly. the people i enjoyed working with, and who often made the biggest difference, weren't necessarily the ones with the most impressive cvs. or the most experience. or the longest list of people willing to say wonderful things about them. they were good humans. and by that i don't mean people who always agreed with me, avoided conflict, or made everyone feel comfortable. quite the opposite, sometimes. i mean people with integrity. people who were curious, honest, accountable, and willing to learn. people who could disagree without making things personal. people who admitted when they were wrong. people who helped others become better instead of making them feel smaller. people you could trust when things got complicated. which, in startups, is usually around lunchtime. of course, being a good human doesn't automatically make you good at your job. and we should be careful here too. "good human" can easily become another subjective label for people who look, think, and behave like us. that's not what i mean. i mean observable behavior. keeping commitments. taking responsibility. treating people with respect. being honest when it's inconvenient. doing the right thing when nobody is watching. skills still matter. competence still matters. but those human qualities have been some of the strongest foundations for doing great work together. i can't prove it with a statistically significant sample. and after everything i've just written, it would be rather embarrassing to pretend otherwise. it's simply what i've observed. so yes, do your interviews. check your references. test the work. ask difficult questions. make the best decision you can with the information available. just remember that no number of references can eliminate uncertainty, and no hiring framework can replace the responsibility of actually getting to know someone. we don't need another hiring religion. we need a little more humility about what we know, a little more curiosity about what we don't, and a little more humanity in how we make decisions about other humans. hire good humans.