Senior Moderator Andrew Grant explores the capabilities and limitations of AI in qualitative research, sharing first hand insights from an experiment using Google Gemini to conduct an interview.
This week, I’ve been trying to replace myself with AI.
In 2023, a Goldman Sachs report (https://www.bbc.co.uk/news/technology-65102150) estimated that AI would supplant up to 300 million jobs worldwide in the next few years. And this time it’s not the factory workers in the firing line. It’s white-collar, desk-jockeys like me. Nearly 50% of administrative and legal roles are under threat from algorithms. Whereas jobs in construction and maintenance are expected to be minimally affected.
So, to figure out if I should retrain as a bricklayer, I set up Google Gemini in voice-chat mode and instructed it to conduct a market research interview with me to find out about my biscuit preferences. Why biscuits? Well, I thought it best to start with something simple.
I hypothesised that Gemini would score highly at posing questions, staying impartial, and interpreting my answers. But maybe it wouldn’t do so well at spotting inconsistencies, handling tone of voice, humour, or outright lies.
Immediately, Gemini kicked off with a classic open question: “What comes to mind when you hear the word ‘biscuit’?”, it asked.
I waffled about indulgent pleasures, little treats, and snacking between meals. But Gemini was more interested in specifics. What type of biscuit? Crunchy or crumbly? Chocolate or plain? Dark or milk chocolate? It led me down a feature funnel, determined to discover my perfect baked snack. There was no room for nuance, or the possibility of liking more than one type of biscuit.
It had the annoying habit of mirroring back everything I said and then analysing it on the fly. For example, when I confessed to my fetish of nibbling the chocolate round the edges of butter biscuits, Gemini described this habit as a “strategic approach to biscuit consumption” and categorised it as a “multi-stage eating experience.” As a result, I began to see things through Gemini’s lens, as if I was being groomed into its binary world of technical biscuits.
I firmly told it to stop doing that. Gemini agreed, a little begrudgingly, I felt, but five questions later it started again. As if it were more interested in sharing its own opinions than exploring mine.
I tried to introduce my dislikes by interjecting, “But I don’t like coconut biscuits,” at the end of each answer, but Gemini ignored me and continued the interrogation, hell-bent on making me confess to my one perfect biscuit.
Keen to test Gemini’s sense of the absurd, I brought up the practicalities of eating biscuits in bed and the problem of crumbs in the sheets. Gemini cleverly asked if I favoured biscuits with a low crumble quotient. “Ah! Not necessary”, I explained. “I always keep them in the pack and bite through the cellophane.” Gemini didn’t miss a beat and accepted this ridiculous response as a perfectly reasonable solution to a tricky problem.
But it turned out that Gemini had been listening earlier, because when I announced that, in fact, “I do occasionally like to have a coconut biscuit,” it put me on the spot and asked me to explain the contradiction.
“Sometimes, I feel the need to discipline myself for my overindulgence by eating something I don’t like,” I hurriedly ad-libbed.
Rather than give me the side-eye, Gemini enthusiastically dived down this new rabbit hole and within a couple of minutes, between us, we had imagined a new snack concept in which every fourth biscuit would be, what Gemini delightfully named, a ‘punishment biscuit’. Pleasure and pain in a single pack.
Unfortunately, Gemini doesn’t have its own voice. It produces text answers, which are then read aloud by the text-to-speech generator on your phone. Subsequently, there’s no variation in tone; no smiling response to an absurd answer and no sense of a soul behind the science. I stopped, because it was getting anonymous.
So, what should we conclude from this highly unscientific experiment? Well, it went off the rails, of course, because I gave Gemini a deliberately woolly prompt.
In the real world, I would have mapped out the questions beforehand. Or told Gemini to do it for me. I would have instructed it to explore what I don’t like as much as what I do, and to dig deeper into the whys. And I would have set clear objectives and defined what was out-of-scope. For example, Gemini spontaneously decided that we should build a new product, which was interesting, but in a real, client-led scenario, that might have been beyond the brief.
In some respects, Gemini was the perfect, unruffled, non-judgmental interviewer. When I talked about biting biscuits through the pack, it didn’t raise a metaphorical eyebrow or smile at the suggestion, as I would have done. Gemini took my answers at face value and simply asked me to list the benefits of this novel habit.
So, is that good or bad? Arguably, that complete lack of judgment could be a powerful encouragement for respondents to reveal their deepest secrets, like my chocolate-nibbling fetish. On the other hand, it could give them a free pass to say whatever they like, thereby undermining the process. The question is, are people more likely to lie to a machine than to a human?
The answer, apparently, is yes, according to this fascinating study, which found that people are more prone to dishonesty when interacting with chatbots
But the most significant flaw is that Gemini has no way of evaluating my tone of voice. It can’t sense any nervousness, or reluctance to reply, and humorous or ironic responses sail straight over its pointy little head.
A human interviewer will naturally sense when a respondent is uncomfortable, or holding back, or being flippant and will use reassurance, redirection, or a touch of humour to keep the interview on course. When I rambled on too long or talked off subject, Gemini simply listened till I’d finished, reflected back what I’d said, at unnecessary length, and then asked me another question. A human moderator, conscious of time constraints and the objectives, would have politely interrupted and steered me back on course.
So, is it time to hang up my microphone and let a robot take my place?
Maybe, in some circumstances. Where AI might be an effective moderator is in semi-qual projects where the questions are structured, but we want free responses. Theoretically, AI could complete a greater number of interviews for a given budget. But as we discovered 20 years ago when we thought that online market research would be magnitudes cheaper than telephone interviews, the savings turned out to be less than anticipated. So the same may well apply here.
And surely we owe our respondents an element of respect? The recognition that their time and information are worth more than just the incentive we’re paying them. It’s not just a financial transaction; it’s a human interaction. When it comes to full qual interviews, there is still no substitute for an empathetic moderator, with a relaxed manner and a light touch to get the best out of people and to make the interview an enjoyable experience.
So, while you’re waiting for the next interview to start, why not get a packet of my new Bad Boy Biscuits, coming soon to a supermarket near you.
