A Bot’s Guide on How to Moderate

I have been observing the Artificial Intelligence (AI) community’s very impressive and clever development of Bots and expert systems for some time now.  Some of the biggest tech companies such as Google, Amazon and Apple have been putting serious money into this kind of AI. Bots can do many amazing things, some of which seem to have a research application. Some can convert text to speech, some speech to text, and some can summarise text data in a fraction of a second.

But can any of them moderate?


Some people in the research have suggested that qualitative research moderators will soon lose their jobs to Chat Bots. So, I thought I would help things along a little to delineate exactly what it means to be an online moderator for a qualitative research project, so we can reasonably assess which Bots will be best for the role.

 I have based some of this on the thoughts of AI Pioneer Marvin Minsky, especially his concept of ‘resourcefulness’. Minsky argued that humans are resourceful thinkers because we have ‘many ways to think’. He stated in (The Emotion Machine: Common sense Thinking, Artificial Intelligence, and the Future of the Human Mind[1]) that humans had limited the capacity of machines to think by limiting what they can do. He was critical of machines that had been built to suit a particular purpose, arguing that they should be more like humans and capable of divergent ways of thinking.

What a Bot would need to know

Clearly, the Bot would have to be taught the topic and the brief and the project background, but that should be a relatively simple matter. I am focussing here on the data collection side. These are some things that the best* online moderators and interviewers regularly do as a normal part of their role, that a Bot would need to learn.

  • If a line of questioning or an activity isn’t working, then change it.
  • Ask some people to expand on the point they have made because they have not been clear or because their comment is very relevant to the brief. 
  • Know when to step in to manage a group dynamics issue and when not to.
  • Suddenly realise the significance of someone’s comment, and so depart from the topic guide to pursue a new line of questioning.


In other words, Chat Bots will have to learn to adapt to the things that people do and say quite autonomously. They will also need to be taught how to comply with the Code of Professional Behaviour (COPB) – no turning into a potty-mouthed racist like Microsoft’s Tay!

They must ‘behave ethically and must not do anything which might damage the reputation of market, social and organisational research.’


This must also mean telling the truth surely? If we are using a Bot to moderate, then we must tell people what we are doing.

Try this quiz

Which of these do you think will happen if research participants know they are talking to a Bot?

(Multiple response)

  1. They will treat the Bot as if it was human.
  2. They will deliberately deceive or tease the Bot, to test it out.
  3. They will fail to engage with the project and give the minimum needed for the incentive.
  4. They will replace themselves with a Bot.
  5. Our moderators Bots will need to prepared for any of these.

And if they are not told?

One of the wonders of human conversation is how fragile it is. As Paul Grice showed, for human conversations to get past ‘hello’ we have to trust the people we are talking to. We start off with the implicit premise that the other person is being truthful, succinct, polite and relevant.  Because all this is implicit, we are naturally highly sensitive to any hint of a breach, any hint that the person is lying or being ‘economical with the truth’ or for some reason telling us more than we need to know.

Mashable reported an exchange between an irate customer and a customer service ‘rep’ (or most likely a Bot) who failed to answer the question that the customer had asked (i.e failed the ‘relevance’ test), denied being a Bot (failed the truthful test), kept on saying the same thing even when the context changed (failed the ‘succinctness’ test). All of these were hints that ‘Danielle’ and ‘Sophie’ were not human. And then of course, humans being humans, it went viral.

So, following Minsky, if our industry wants to use Chat Bots then we will have to develop Bots that can think divergently, act autonomously and truthfully within the confines of the Code, and to be human enough that people will trust them enough to tell them things.

*For me the ‘best’ moderators are those people who actively engage participants online. I do know that some people have chosen to automate their own moderating process, so they input all their questions into the system ahead of time, and then pretty much exit the building.  Perhaps these are the best jobs for the Bots?

I will be exploring the topic of Ai, Automation and Agility in the Market Research fields in an upcoming webinar hosted by NewMR on 18th August 2016. Click Here to register. 

[1] Simon & Schuster; Reprint edition (November 13, 2007)

Tags: Market Research, Qualitative Research , Technology, online qualitative, Moderating

Print Email