Do you know just how bad the traditional interview is as a tool for choosing the right person for the job? In one research study, of students selected for medical school by an ‘unstructured’ interview (the type where someone sits across a table from you and asks you things like “tell me about your greatest achievement at work”), it was found that the process is no better than chance at choosing the right people. Students rejected by one assesment process performed no differently than those selected by the same process.
You may as well halve the pile of CVs on your desk, drop the top half in the bin and save some time. If interviewees were employees, they would have been put on probation and fired for terrible performance decades ago.
Yet, still we continue with the time honoured tradition of putting an hour in the diary and subjecting a nervous stranger to a battery of novel but inherently unstructured questions to test whether we should allow them the privilege of committing several years of their life to us.
Of course some companies (and I hope you work for one of the enlightened ones), take a better and more considered approach to interviews. If you, however, still know a company that makes the same old mistake of boring, intimidating, unnatural and ineffectual interviews I’d like to introduce you to a process that has worked for me, and helped me hire hundreds of engineers, product owners, QAs and scrum masters. I wanted to share it with you, not so that you can follow it blindly, but so you can understand the reasons why I created this structure, and to give you the tools to improve on my process.
Why don’t you just do an interview like a normal person?
Have you ever wondered why the world needs millions of articles written about ‘how to ace your interview’? The answer is staring us in the face; interviews have so little to do with being good at your job that they need their own self-help category. Wouldn’t it be better to have a million authors writing articles about ‘how to be good at your job’, instead?
I’ve written before about the 20th century curiosity that is the Curriculum Vitae (or resumé for our North American cousins), and the unstructured interview process that developed from similarly humble beginnings with the birth of the office age. Not only is the standard, hour long interview process horribly broken, but there is a notably, scientifically better model to assess fitness for a role: the assessment centre.
Tomas Chamorro-Premuzic, an organisational psychologist specialising in talent, writes:
“Different exercises and simulations are designed to enable a direct observation of a candidate’s ability to perform on the job. In some instances, assessment centres focus on candidates’ ability to perform job-relevant tasks, but they may also be used to infer candidates’ work ethic and likeability — particularly in exercises requiring interpersonal interaction with other candidates. If time and money aren’t issues, assessment centres are hard to beat as a talent identification method. It is as simple as saying that if you want to work out whether someone will be good at a given job, then you can just make him do that job and see. This common-sense idea has been supported by several meta-analytic studies, which estimated the validity of assessment centres at .54 (higher than any other assessment method).”
The Talent Delusion, Tomas Chamorro-Premuzic
Chamorro-Premuzic continues, “It is also noteworthy that job applicants tend to regard assessment centres as fairer than other talent identification methods”. These assessments are not only demonstrably more valid activities for selecting the best staff, but are also perceived to be more fair by the candidates themselves.
So, what’s not to like? Well, assessment centres are expensive to run and take a significant amount of time and discipline to set up. Their cost and complexity makes them best suited to roles where there is a high degree of consistency in the roles being tested, and a high volume of roles. Toyota, for example, use a day long assessment centre of actual factory tasks to hire their line workers.
However, given the difference between a great team member and an average one this investment in time is almost certainly worth it. It’s sometimes quoted that a great developer offers ten times more productivity than a poor one, and, while it’s hard to find evidence to back that statistic up, it intuitively feels that a destructive staff member has at least that amount of productivity impact on a team.
Without labouring the point that interviews are broken, how do I suggest you fix it?
The trial shift
If we want to test how much value a candidate can bring to our organisation, the simple and best way is just to employ them to do the job and see how well they do. Unfortunately, life (and, in some countries, stifling government regulations) get in the way of doing this for most roles. So, my intention was to develop a process which got as close to this as possible, whilst being respectful of the candidates and efficient for the business to run.
One early inspiration came from a wildly different industry to my own; the restaurant kitchen. The seed was planted with Anthony Bourdain’s eye-opening Kitchen Confidential, which introduced me to the concept of the ‘trial shift’ as a process for interviewing for roles in commercial kitchens. Here’s what British chef Marcus Wareing has to say about the process:
“I’ll invite you into the kitchen for a trial. Sometimes a chef will come in on a day off, but usually you’ll come for three or four days. I can’t remember the last time I did a sit-down interview. I usually sit down with you at the end of the day, but I’m more interested in the way you cook. A short trial tends to be unpaid so you may only be allowed to watch service. However, my brigade is not allowed to use chefs on trial as dogsbodies. All potential recruits are looked after.”
Marcus Wareing, Chef & Restaurateur
Simple, right? You come in, you do the job and if you’re any good you get signed up. Notably, Wareing calls out one of my key concerns; how can we make sure that interviewees don’t feel like they’re being used as free labour? Checking with friends in other industries, I found that the trial shift (or “trade test”) exists in other service industries like hair styling, where the candidate joins the salon for an on-the-job evaluation.
So, how do we create a trial shift for digital product development?
The random product workshop
This then, is my process for bringing in a new potential team member, exposing them to the reality of working with an existing team and getting them to open up as if they were already a member of the team — the random product workshop.
In the product workshop we’re going to invite a candidate to join existing team members to discuss, argue and define the features of a potential product. We’re going to strictly timebox the process and scope to a 2 hour workshop, and a 30 day product roadmap. We’ll give this new cross-functional team an overview and some grand goals, a notional budget and ask them to develop a plan for 4 weeks of work. We’ll ask the team to present back their work after 90 minutes of prep.
Set the stage
It was important to me that this process avoided two pitfalls. Firstly, we want to ensure that candidates have no concerns that they are seen as free labour, or being taken advantage of for their creativity or problem solving skills.
Secondly, we want to avoid the imbalance that the team’s existing domain knowledge would present if we worked on our own product. If you happen to run a job site (as I did), asking someone with no job site knowledge to build job site features would leave them gasping for air.
To solve this, we randomise the product we’re working on so that it’s not only novel to all of the team, but also to ensure that it isn’t generating intellectual property that we would benefit from unfairly.
Before the interview:
- Come up with two lists: one is a list of types of product (eg social network/community, news, classifieds, shopping, productivity tool) and types of customer (first time mums, teachers, cyclists, truck drivers, cowboys, astronauts). Write them down on two types of coloured card, one colour for each type
- Fold the cards and put them into a bowl
- Define your criteria (or judging rubric) for what makes a good hire, and create a quick survey to share with your team. Collecting feedback to standardised criteria is a step towards reducing bias (more on this later)
Select the team
The aim is to provide a team to work with the candidate that lacks the target role — for instance, if your teams typically comprise of a designer, developer, QA and product owner, you will fill the developer role with a developer candidate. It’s ok, and indeed preferred, to leave out duplicates; there’s no need to stand up a full scrum team of eight people. It’s great (but not essential) if you can let the actual team members who have the vacancy be present for this process and make the selection themselves.
Run the process
- Bring your team together, ideally in a team/meeting room where they won’t be disturbed
- Ask the interviewee to pick one card of each colour (A news site for astronauts! A social network for… hamster owners?)
- A senior team member does a brief role play as ‘CEO’. Given the now selected product, the CEO is put on the spot to describe the product that they want created. Typically the speech goes like this,
“I made a lot of money with a previous startup, selling cryptocurrency to real miners. I’ve got a new idea. When I was working with miners, I realised that there’s a huge market for second hand mining tools, and I’ve got a new idea… eBay for coal miners. I want to build an app that lets people buy and sell old mining equipment. You’ve got 30 days to build me an MVP, and over the next 90 minutes I want you to work together to suggest what you can build, and how you’re going to build it. You’ve got a budget of 10 bitcoin*.”
*insert your own currency here
4. The team now have an short opportunity of no more than 10 minutes to ask the CEO questions about the product. Mobile or web? Does the MVP include payments? How many users will there be. This is a critical stage to watch the interviewee take part with the team and demonstrate curiosity.
5. The CEO leaves the team alone for 90 minutes. In this time, they are encouraged to build up a project plan and architecture of the what and how of the product. What features will be built? What tech will be used? What won’t be done? I generally leave the team well armed with post-its and Sharpie markers, because these are required by Regulation 20.1b in high tech companies.
6. The team should work together collaboratively to spec the product. Where possible, they should encourage the interviewee to demonstrate the qualities mentioned in the judging criteria. If it’s a technical role, do they appear knowledgeable? Are they guiding the team in their area of expertise, or taking a back seat?
7. A significant element of this process, even for highly technical roles, is to identify candidates who are intelligent, creative and will challenge their team appropriately. We also look for strongly held opinions that will be easily withdrawn in the light of better data.
6. After 90 minutes has elapsed, the CEO returns to the team who will pitch back their progress. The expectation is that the team, with the candidate taking part appropriately, will be able to explain the work that will be completed in 30 days. For technical roles, it’s worth having someone who can challenge the technical decisions and the arguments used to defend them.
Why do I do it like this?
- Interviewing is incredibly flawed, and typical 1:1 (or 2:1) unstructured interviews do a terrible job of understanding if the interviewee is any good at the job. You’re only testing if they’re good at interviews (which ideally will be a very small part of their actual role)
- Teamwork is an incredibly important part of modern product engineering. Being able to present an opinion and argue your corner is a critical skill regardless of the job. We’re looking for people with strong opinions, but weakly held
- You can train skills but not attitude. This is a deeper topic, but largely we are looking for people who work intelligently and collaboratively with others in their team as this is an equally critical requirement of product team members as their particular product or engineering specialisms.
- We try to avoid individual, manager bias but including a team. Ideally, you should have the whole team record their answers to the judging rubric privately, immediately after the interview. These scores should then be shared and reviewed with the team openly.
- Randomising the activity is more fair to the interviewee. Not only does it put everyone in the team at the same disadvantage, neutralising some domain knowledge, but it also eradicates any suspicion that the company might just be using the interviews as free labour
- It’s transparent — half of interviewing is a sales pitch, especially in the fiercely competitive tech landscape. This lets the interviewee get a much better sense of the team dynamics that they can really expect, not just showing off the free canteen and sleep pods.
- The feedback from candidates — both successful and unsuccessful has been positive. It invariably feels more open, transparent, engaging and less stressful than a typical interview
The Downsides
- This is still not perfect — you should read more deeply into the subject to understand how to improve this process for your team, preferably with more data driven insights into the candidate.
- It’s hard to run this type of workshop assessment with remote teams (but I’m working on it). I’d love to hear if you have any experience of this with fully remote candidates. Remember, the aim is to replicate real tasks as closely as possible.
- For deeply technical roles, it’s not really demonstrating hard skills. Hopefully, developers will have been through a technical screen earlier in the process (more later)
- It’s expensive — it’s not cheap to take a team of people for a couple of hours to fill a role (but it sure does pay dividends when you get the right people)
Gating your investment in interviews
One common criticism of interviews by candidates is the use of technical screening tests for developers, or seemingly trivial phone screens that they don’t progress past.
Unfortunately, the truth is that every job opening is usually oversubscribed with candidates, and there’s still no good way to judge the quality at scale. From my experience with job boards, it’s normal to have over fifty applications per job, and not uncommon to have over a hundred. We clearly can’t run this process for 200 possible applicants.
In order to make the best use of our resources, it’s important to screen down the number of appropriate candidates. Where possible, this initial screening should be done with personal information removed (CVs or résumés with no names, and if possible no education history).
In technical roles, a technical pre-screen can be used to screen out candidates who can’t complete a simple test with the aid of the internet. It’s still unfathomable how many people can fail to Google the result of a trivial coding challenge, and this is a helpful way to reduce applications. Again, be cautious not to screen by any qualities other than their capability, and don’t try to set impenetrably hard, unpaired technical tests. The point is just to quickly screen out those with no clear technical competence.
Finally, choose your own weapon for initial interviews before this product workshop. Typically I will only use this process with 2–3 candidates per role, and it’s important that you decide how to do this initial pass fairly and impartially. I particularly like technical teams, like engineering, to run a pairing code excercise with an existing team member before the product workshop.
Reducing interview bias and building better interviews
Interviewers tend to choose people like themselves, which is the obvious enemy of diverse teams and good hires. Try to have a varied team involved in the selection, and make sure that the team can score the interview privately and as soon as possible after the interview.
I strongly recommend picking up Tomas Chamorro-Premuzik’s book, The Talent Delusion to discover more about identifying talent scientifically, and for a quicker read, this article on bias reduction by Iris Bohnet.
Over to you
This is just one exercise that I have used to help screen for product and engineering roles, and it can definitely be improved on. I’d love to hear what has worked for you, and particularly how you have helped reduce bias in your interviews and increase diversity in your teams.