Interviews useless?

Peter Martin has an interesting post on a turnaround by the UQ Medical School, who’ve decided to scrap interviews as part of the selection process for their graduate medical degree. He cites an American psychologist, Daniel Kahneman, who is also a Nobel prize winner for economics:

Kahneman says we would get better results picking candidates for jobs by using almost any objective measure – length of service, academic qualifications, it doesn’t really matter – rather than by subjecting them to interviews.

He says we appear programmed to believe that we are good at picking talent – “delusional optimism” he calls it – and impervious to evidence that suggests we are not.

I’m not sure if I’d agree with the conclusion that “human judgement” is always inferior to data because – rather in the manner of the economist – that seems to ignore that data is itself a human construct (exam results, for instance, aren’t objective measures of ability or intelligence and indeed often they measure the ability to do well on exam papers more than anything else). But all the evidence I’ve ever seen in the HR literature says the same – job interviews suck as a predictor of job performance. I imagine they’re still used because employers like to “eyeball” candidates, but having sat on heaps of selection panels in my time, I can only recommend that they be weighted very lowly in the overall selection decision. Martin also writes:

Other medical schools haven’t caught on, including the new one at the ANU. It is setting itself up for the sort of embarrassment faced by Adelaide University which some years back was found to have rejected students with Tertiary Entrance Scores of 99.9 and family backgrounds in medicine, apparently because its interviewers didn’t like they way they presented.

Adelaide has been charged with using interviews to socially engineer its student mix, ensuring that it is not overwhelmed by the children of doctors or the products of private schools or brainy Chinese and Vietnamese.

Here I want to quibble a bit. I was a student rep – a long time ago – on an Academic Board committee at UQ which considered the switch to graduate entry. The whole point was a recognition that academic performance at high school alone wasn’t sufficient to predict who would be a good doctor – so the committee went for a graduate model that would admit candidates with more life experience and with academic backrounds in other disciplines (it’s possible to do a medical degree after doing a BA or a BMus or whatever if you pass the GAMSAT test). The idea was that forming a person as well as getting high marks was an important part of medical training. If this counts as “social engineering”, so be it, but perhaps where these medical schools have gone wrong is by hiding behind subjective judgements in an interview rather than being explicit about wanting to widen the net for the medical profession. That aim deserves discussion that goes beyond “my daughter got a 99 TER and five generations of our family are surgeons and didn’t get into Adelaide” which has been the way that it’s normally been framed.

Advertisements
Posted in education, health, sociology
42 comments on “Interviews useless?
  1. Debbie(aussie) says:

    Thanks for this Mark. I have passed this on to my daughter, who is doing a bachelor of nursing and hopes to go on to medicine. Interviews terrify her, but she stresses with exams too. I thought the two would sort of balance out. Maybe things will have changed, yet again, by the time she gets there.

  2. mbahnisch says:

    No probs, Debbie. Exams can be better predictors – at least of being able to pass further exams!

  3. Debbie(aussie) says:

    So far so good. She is making the GPA needed for GAMSAT, so long as they don’t up it too far.

  4. zebbidies spring says:

    I’ve been disappointed by the focus on 99.9 TERs as being the best criterion for choosing a doctor. I have found that a super-bright doctor who won’t listen to their patient is not a good doctor, no matter how well they pass their exams.

    TO be quite honest, I think that sending our very best and brightest to be doctors (or lawyers) is a tragic waste of their talents. Being reasonably bright and very hard-working and persistent seems to be the best mix for doctors.

    It’s rather sad that the genuinely intellectual giants among our youth should be steered away from sciences, arts or engineering where they have a real prospect of extending out knowledge of how the world works. It’s very sad to see middle-aged doctors who have made the wrong choice of career and are unable to leave because of the money, the status and the expectations of those close to them.

    I supported the use of interviews, though it does appear they haven’t been very effective. I do hope that the taking of only graduates continues.

  5. bahnischba says:

    But all the evidence I’ve ever seen in the HR literature says the same – job interviews suck as a predictor of job performance.

    This was well known back in the 1980s when I recall reading articles and hearing lectures about it. Over several decades up to 1991 I conducted, or was part of the panel for something over 600 interviews. Panel leaders all knew of the difficulties but each one thought they had cracked the secret of doing successful interviews. They were wrong.

    Nevertheless, I do think they interviews have some value. Extreme personality unsuitability, for example, will sometimes be easy to identify (and sometimes not). I found the best way was to talk to someone who knew the applicant’s work AND understood the new role they were aspiring to and was not nominated by the applicant as a referee. I’m sure head hunters do it, but the Goss reforms of the public service made it illegal

  6. As someone who got high marks at school, and horrified many caring adults by “wasting” those high marks and not going into medicine, I completely concur with zebbidies spring that 99.9 TERs are not a good way to pick good doctors. Maybe you need a non interview (less subjective) way to test for communcation skills potential, but you do need some way of testing for it.

    Every HR guideline about interviewing I’ve ever seen says that “situational” interviewing is the best – ask the candidate to tell you about a specific way in which they have approached a problem in the past (e.g. getting along in a team with someone who wasn’t pulling their weight). So I’d be interested if all the studies suggest that that is just as hopeless, because that’s certainly what the HR professionals in my organisation recommend!

  7. Sir Henry Casingbroke says:

    Well, here is an opportunity to come back with some empirical evidence as to who makes a good doctor. Newcastle Univeristy some years ago, when it set up a school of medicine, used as a model McMaster University (Can)to admit medical degree candidates NOT with the highest exam results at the end of high school, but to invite people with life experience and a sense of vocation in preference to bright kids – ambos, nurses, insurance salesmen, and so on. The final criteria were drawn out as a result of an interview process.

    I’m not entirely sure about this, but Newcastle may have pioneered the interview notion as part of entry requirments into a medical faculty, at least in Australia.

    Okay. I wonder if anyone out there knows, was Prof. David Maddison’s experiment a success as judged by evidence?

  8. Fozzy says:

    As a former Newcastle Uni student (though not in Medicine rather Computer Engineering) I’m always proud of the fact that Newcastle Uni was the one that started the idea of interviewing Medical Students. It was a bit of local mythology that studies found the students from Newcastle to be “better” medical students (I admit “better” is a somewhat loaded word and I don’t know what metric was used).

    To give some additional background to this blog entry, there was a discussion on this topic on Life Matters last Wednesday between a Prof of U of Q and a Prof of U of Newcastle. You can catch the podcast for another day or so but the transcript should last longer than that. They can both be found via:

    http://www.abc.net.au/rn/lifematters/stories/2007/2109709.htm

  9. mbahnisch says:

    So I’d be interested if all the studies suggest that that is just as hopeless, because that’s certainly what the HR professionals in my organisation recommend!

    Well, I think it’s got a marginally better track record than other types, but it’s also pretty useless. I think it’s a fallback position because HR folks are unable to talk people out of wanting to hold interviews.

  10. philiptravers says:

    Is it intelligent to accept medicine is a developed intellectual occupation..some of them are like Utah Police with Tasers!?.Has Doctor shopping increased or decreased with the much more intelligent individuals from overseas!?.Are they going to demand that the patient who feels he or she whatever is being overserviced that isnt the case?So when it comes to interviews rather than data,this couldnt be an exception to the ongoing data collection and interviews.

  11. mister z says:

    I used to work for an organisation that took a relatively rounded approach to recruitment — balancing performance at interview against references/referee reports (taken up for all interviewees), career history etc.

    I now work for an org where every other factor is simply a funnel to who gets interviewed, and the final decision made on interview performance on the day alone. Some fairly awful self selection and patronage can be observed as a result…

  12. Tony Healy says:

    mister z, that sounds interesting. Any examples?

  13. The Feral Abacus says:

    I caught part of the LifeMatters interviews, and thought the Prof Wilkinson of UQ was being a little slippery with his data.

    The gist of the interview was whether the interview process was successful in selecting suitable candidates for medical training, and whether it helped to introduce students from a wider range of social strata into medicine. But – from memory – Prof Wilkinson addressed these issues by discussing the relationship between interview assessment and exam marks.

    I’m not convinced that a medical student with a high final exam marks will necessarily make a first rate GP. The use of exam marks as the metric of good medical training outcomes is also somewhat confounded by the fact that of these students, some will become surgeons, others specialists, and a few medical scientists. Exam marks are not likely to provide an equally good indicator of later success in each of these career paths.

  14. mbahnisch says:

    Which was the point of introducing interviews into the selection process in the first place. It seems to me that rather than reverting to just exam marks, they ought to find a more rigorous selection mechanism to achieve the initial (and I think worthwhile) aim.

  15. Kwakpwns says:

    I was actually interviewed by the UQ SOM (school of medicine) this year. I had always felt that interviews are far too subjective to be able to judge whether applicants have the appropriate characteristics for medicine and i feel that was confirmed this my interview. I wanted to enter medicine because i believe i am bright enough (my GAMSAT score was 68, which is around the 93rd percentile and my GPA is somewhere well above six), i am quite sociable and i want to be a doctor for genuinely altruistic reasons, ie i hope one day to do some work for medecin sans frontiers (doctors without borders). Nevertheless, I was only accepted in on a bonded medical place, which is a lower form of entry that requires one to be bound for (in my case) four years to a rural/regional area. My interview experience confirmed my belief in their subjective nature – at one point we have to pick a topic from a list (mine included things like safe injecting rooms, capital punishment [the interview was close to the bali anniversary], the aboriginal intervention, etc – and argue a point of view. The fact they rely on a 15-20 min viewing is also a hindrance because if u dont do as well as u could (because of nervousness etc) then you’ve blown your chances.

    I do however think that it is fair enough that unis meet students before admitting them. I’ve noticed that some unis have adopted a pass/fail system for the interview portion of the admissions process, and i think that this is probably an appropriate mechanism. The rest of their process is guided by their GAMSAT and GPA scores, which i guess is fair enough.

  16. mbahnisch says:

    That’s interesting, Kwakpwns. I can see how the ‘argue the topic’ segment might add fuel to the fire of the accusations of bias.

    Again, I think some sort of psychometric testing might be a good alternative to the interview – it could easily be scored or analysed anonymously and should enable med schools to seek out those with the dispositions towards practice they’re looking for in addition to rank scoring people on gpa and entrance exam.

    Part of the problem here is probably one that plagues academia generally – people in charge of making these decisions are specialists in medicine not in evaluation/testing! They really ought to be seeking expert advice before changing their selection practices.

  17. The Feral Abacus says:

    Hmmm – Kwakpwns, that sounds appropriate for a student entering law, but medicine? So really the interview process is selecting for those who are self-confident and articulate. That could be seen as a way of maintaining the social status quo.

    Mark, psychometric testing would be useful if personality profiles appropriate to practising medicine were well understood. I guess it would at least be useful in identifying those who were manifestly unsuitable.

  18. mbahnisch says:

    Yes, I’m not sure how well those profiles are understood, TFA, which is why I think they might need to go back to the drawing board in trying to work out how to best select.

  19. The Feral Abacus says:

    Mark – as far as I can see UQ have not demonstrated that interviews do not assist the medical school in selecting students who go on to become good doctors. They’ve only found that interview scores are not a good predictor of a student’s subsequent performance at medical school. I think there is an important distinction there.

    If the purpose of the interview process was to produce a particular type of doctor, why are UQ evaluating its effectiveness by only looking at academic achievement? Surely some form of post-tertiary follow-up study would be more appropriate.

  20. mbahnisch says:

    Yes, that makes sense.

  21. Feral Abacus’ comment = hammer+nail *whack*

  22. Kwakpwns says:

    TFA, yes i realise it is incredibly difficult to assess the best students for both the medical degree itself and the subsequent practice of medicine. I think what UQ discovered was that their dropout rates were not improving since the introduction of interviews, and the obvious question is therefore whether ; they serve a useful purpose. I (like i feel Mark does) hesitate to endorse a purely academic means of testing potential doctors because it is a position that requires a fair degree of communicative skills and tact. Nevertheless, at a time of chronic doctor shortages it is probably most prudent to find the system that increases the level of graduates of the degree. If a student drops out in third year because of the difficulty of the course, then that’s an entire medical place gone to waste. In any case, the assessment in your third and forth years does look at the way one interacts with patients.

    I hesitate to invoke blogger’s wrath by linking an opinion piece by the devil woman, Janet Albrechtsen, but here a piece she wrote earlier in the year:

    http://blogs.theaustralian.news.com.au/yoursay/index.php/theaustralian/comments/questions_not_even_a_doctor_should_answer/

    For the record, here is a reply by the Sydney (acting) dean of med:

    http://blogs.theaustralian.news.com.au/letters/index.php/theaustralian/comments/we_want_doctor_diversity

  23. kimberella says:

    Re – Martin’s choice of phraseology. Admission by exam results/gpa/ter is also “social engineering” just a different and more traditional kind.

  24. Chris says:

    Kwakpwns – you make a good point that part of the medical student’s assessment in later years is based on patient interaction, so if those results are not improving then the interview process is not filtering potential students properly.

    Mark – one of the problems with psychometric testing is that it can be manipulated by people to give the results that they are looking for. In fact that is one of the problems with the interview process at the moment – students are getting training in how to answer the questions in the interviews in the “proper ways”, instead of what they actually believe.

    Re: interviewing – the best way I’ve seen to test potential candidates (especially graduates) is to just employ them for 3-6 months (eg internships). Interviews can be good for sanity checks though – especially in small teams where its important that the person will get along with the rest of the group.

  25. christine says:

    The decision on medical school applicants seems to be largely about how to allocate large benefits to a small group of people when there are lots of people wanting to get in. Pretty much any way you do this is going to be unfair (or thought of us unfair) by someone. And it’s almost certainly incredibly inefficient, esp if there is a shortage of doctors. Worrying about the exact decision rules to allocate the benefits seems like a secondary problem, albeit one that is very important to those who just miss out.

    As for this business about admitting people because of a genuine vocational interest: got to say I think that would be incredibly easy to manufacture a vocation for a few days, enough to sound convincing in an interview. Also, I don’t think that was necessarily missing in the old high school admission system: of the 6 or so kids in my school year with the TE score to make it, only 1 (at most 2) did medicine, with the others doing, I think, journalism, law, science, and economics (I think one didn’t even go to UQ! Imagine!). So it wasn’t really the case that everyone with the marks automatically went into medicine.

  26. Sam Ward says:

    Interviews are an excellent way to work out what people are really good at sitting in an interview and speaking well.

    So they’re great for recruiting people who need to do this, like Politicians or Media Liasons.

    But for recruiting people to do actual work they are incredibly stupid. If you need a software developer for instance, then having an interview at all is going to cut out some of the best candidates, because not many people are good at both software development and public speaking.

  27. Enemy Combatant says:

    If a potential general practitioner possesses the cerebral wherewithal to handle the academic load, surely the next most important skill needed is the ability to communicate effectively with patients.
    The public would be well served if professionals/academics with the ability to assess this fundamentally essential tool in a potential GP’s bag, were well represented on interviewing/selection panels.
    Human whisperers make the best healers. One assumes, ever altruistically of course, that Modern Medicine is still more about healing that amassing corporate profit.

  28. Greg says:

    Interviews are only as good as the people conducting them, whether they’re the 13-question method (where do you see yourself in 5 years? what are your strong points? etc.) or behavioural/situational (which can be just as rote as the former method). The questions need to be formulated according to the role, which needs to be well-defined and -understood and contextualised. And while most people are smart enough to scrub up for the interview, face-to-face will almost inevitably assist in eliminating the compulsive gum-chewers, those lax in personal hygiene, and persons unable to adhere to standards of comportment. Pyschometric testing is a joke. I’ve taken at least three different tests multiple times and have seen a fairly wide set of results. I think this NY’er article is suggestive of their unreliability.

  29. FDB says:

    I agree with Yobbo (or would you prefer to drop the moniker now you’re a politician?).

    Interviews are great for people in service/PR jobs and the like, and for me this includes lots of clinical doctors, especially GPs. It shouldn’t determine entry to med though, because you don’t really need those skills to be a research doctor or perhaps surgeon, some specialties etc.

  30. David Rubie says:

    Sam Ward wrote:

    because not many people are good at both software development and public speaking.

    Just a nitpick – it’s hard to be a good software developer if you can’t interact in a reasonably social way with your customers/clients/users. Maybe twenty years ago you could still get away with being a totally anti-social, introverted geek but most places where software development is done need their coders to gather requirements too.

    There is a downside to the psycho testing: they used to do it (still might) at the millionaires factory for all prospective staff, including numeracy, problem solving and basic personality typing. Within the place, a unique sort of mono-culture flourished for a long while which meant that meetings were generally very confrontational games of one-upmanship, and you were never short of hard charging drinking buddies.

  31. Chris says:

    “Just a nitpick – it’s hard to be a good software developer if you can’t interact in a reasonably social way with your customers/clients/users. Maybe twenty years ago you could still get away with being a totally anti-social, introverted geek but most places where software development is done need their coders to gather requirements too.”

    There’s still lot of brilliant programmers out there with very poor social skills. But given there can be a couple of orders of magnitude difference in productivity of an excellent programmer compared to an average one there’s room for a few people around like that (just hide them in the basement, feed them caffeine and specifications and software will appear out the other end).

    With respect to personality like testing for potential medical students – because people going into uni are still quite young, personalities are still maturing (how much did you mature between the ages of 18 and 24?). On the other hand academic ability is pretty much established by then. Perhaps thats another reason why entrance interviews are not getting good results.

  32. FDB says:

    One problem is that on the face of it interviews are a great idea – let’s have doctors who are good listeners, have a genuine concern for their patients, are adept at explaining options, risks etc to people who may be distressed. The old “interview is only as good as interviewer” line doesn’t cover it. Unfortunately the process must anticipate complaints of unfairness, and if your only comeback is “our excellent interviewers thought you weren’t right for the job” this can’t work in a university. That’s why they substitute processes and metrics they can point to which try to quantify things. The complainant is left to confront whatever “world’s best practice” interview model was used, rather than the people involved.

  33. Dr S says:

    Quick word from a practicing specialist. Firstly, interviews are problematic primarily because they are used to reflect the social and ethnic mix of the community. That ends up as keeping out those of Jewish and Asian descent. Whatever the intention they end up being essentially racially prejudicial in their outcome.

    Secondly, the idea that interview based communication skills prior to training relate to capacity to project human warmth during a medical consultation is risible. Medical communication skill is a juggle between gaining trust and skilfully interrogating someone. These can be done simultaneously, and yes the gift of the gab helps, but in the end it is a learned skill. Capacity to perform the interview is probably a better correlate.

    Most important to all of this is that modern medical training is at least a decade of hard slog from the moment you walk in the front door to make it to being a GP. It is 13 years for a specialist physician, 16 or 17 for some of the surgical disciplines. For that time one is literally hammered with information. Dim, personable people do not do it well. The length of training also has an impact on graduate entry. My colleagues who went through Newcastle in its’ early days were significantly discouraged from specialist training by the thought of still sitting major exams in their forties.

    I have always felt that medical schools should take the brightest kids they can find, mainly so they will still have enough left in reserve to grow up normally despite medical school. Vocation tends to take care of itself.

  34. david tiley says:

    Here’s a compromise: Keep a certain number of places for people who do not have the highest tertiary entrance scores. Interview everybody. Use the results to a) find those people and b) get rid of any glassy-eyed sociopaths.

    Mind you, I would rather do the selecting by putting people into teams, sending them off bushwalking and then telling them when they are three days out that you have forgotten the food. The next half hour of running around should tell you how these people would cope with the obstacle course that Dr S so eloquently describes.

  35. kimberella says:

    Medical student Survivor! 😉

  36. Sam Ward says:

    “Keep a certain number of places for people who do not have the highest tertiary entrance scores.”

    Why? Affirmative Action for stupid?

    There are already more people applying with maximum scores than there are spots. What do you tell the 5 people cut with a score of 99.9 that are cut to make room for the 5 dumb kids?

  37. mister z says:

    Tell them it’s character building. Tell them to get a job! (Shout outs to PJK ’96)

  38. david tiley says:

    Why do you think the others are dumb?

    The plan goes like this: assume we have thirty places. A hundred apply, and fifty turn out to have scores high enough to enable them to do well. Within that group, we already have variation caused by external factors like school, weird shit in the year, heritage etc. So simply ranking them is not necessarily going to find the best.

    So, you interview everybody, because you may well want to weed out people who have been coached to within an inch of their lives, are incredibly socially disfunctional, or extremely likely to drop out. You won’t find that many, but it’s good to have the option.

    Then you fill 25 places on academic performance. The next five go to people you pick on interview. This is not primarily to do with diversity, although it could help. It helps you to search for the social skills which are important to medicine.

    It gives interviews the place they deserve, but not as the central process in the selection. I would argue that unfettered interviews over time simply replicate the class and cultural background of the people selected to give the interviews. Ask anyone who got into Oxbridge. Do you fit? is a bad criterion.

  39. Sam Ward says:

    It helps you to search for the social skills which are important to medicine.

    The social skills of a 17 year old child applying to a university degree which they won’t finish for close to 10 years is hardly relevant.

    Everyone applying is going to be a completely different person by the time they graduate.

    The fact is that TER scores are the only fair way to select students. As you say, all an interview does is confirm the biases of the interviewers.

    Some of those 17 year old kids with “poor social skills” and high TER scores are exactly that way because they wanted to be a doctor so badly they spent 5 years of their lives doing nothing but studying to ensure they got the TER they needed to be accepted. I went to school with a lot of kids like that.

    To then deny them the spot because they didn’t also get voted “most popular” is not only stupid, it’s unjust.

  40. Rain says:

    My daughter tried 2 years running through the GAMSAT, interviewed both times, bombed out at interview both times. First time with Wollongong, second time Syd Uni.

    She always wanted medicine, altruistic, community services volunteer since she was 14, selected through highly competitive rounds of interviews for the Australian Youth Volunteers at Sydney 2000 Olympics, a networker, team-player, team-leader etc all her life.

    She didnt make medicine from high-school, but did her first degree in related discipline of medical science, thinking of options in academic medical research, might be a good second-best to her first love to deal with people. She had several preferences, working with Indgenous remote communities, child, youth and women’s health etc.

    After her second knockback after high GAMSATs plus the following interview processes, she is very demoralised. After much thought and soul-searching decided it wasn’t worth a third attempt because of getting older etc, and the long years of training etc as others have mentioned beginning to slip away. Reluctantly, she accepted that her dream would not eventuate.

    Interestingly, the only career option she could find here in Australia, is working as an obscenely highly-paid sales rep, for a multinational pharmaceutical company.

    This corporate world also use similar interview testing techniques to the Universities grad med programs, but unlike the Unis, they always give formal feedback to all applicants, even unsuccessful ones. In my daughter’s case, they said she was a “highly talented natural” in interpersonal communication skills, and coupled with her academic medical science background was recruited for the corporate fast-track, with an eye-boggling starting salary, plus generous allowances, training etc.

    She abhors the ethics of the capitalist Sales & Marketing world, (ironically dealing with doctors, instead of patients) but the money, cars, perks, the professional networking opportunities at international medical conferences etc, is allowing her to rapidly get the money, the professional contacts etc, in order to send herself to Africa next year to try again somewhere else, where she might be “good enough”.

  41. mbahnisch says:

    Thanks for the comment, Rain – your daughter’s experience certainly sounds like an indictment of the current procedures.

  42. Beth Loggins says:

    I personally think that interviews are quite useless. Even situational interviews are useless, because verbally asking a question does not take into account the whole context of the situation. Also, studies have shown that people’s behavior are not necessarily consistent. For example, a person that acts a certain way during lunch time on a certain day usually acts the same way at lunch time on another day. But his or her behavior during dinner time, may be totally different. So a person’s behavior during an interview may be totally different from his actual behavior on the job. Performing well in an interview, I think for the most part, simply shows that the interviewee is good at interviews. But being interviewed isn’t the job itself. Instead of interviews, it might be more prudent to put candidates through actual situations that are similar to what will be encountered on the job, and observe their actions in those situations.

Comments are closed.

  • An error has occurred; the feed is probably down. Try again later.
%d bloggers like this: