History might explain why Faithfuls find it so difficult to root out Traitors in the hit BBC show, suggests one member of the Celebrity Traitors cohort, who were record-breakingly bad at the game.

The roundtable – where contestants discuss who should be cast out – was somewhat “frightening” because of the “velocity in which something goes from a suspicion to belief, to faith, to condemnation”, said the broadcaster and historian David Olusoga.

He spoke as part of a panel at Hay festival on Tuesday, alongside his fellow cast members Clare Balding and Harriet Tyce.

Asked what overconfidence in identifying “traitors and hidden enemies” had led to historically, Olusoga said: “It’s at the heart of what happens in Stalin’s Russia, I think it’s at the heart of what happens in the Salem witch trials at the end of the 17th century, of the Spanish inquisition.”

The idea of “denouncement” – that “what we are swayed by is a voice that is confident, that begins the rumours” – was the “most important thing”. Olusoga said the best example of this was the “myth” of the Gestapo: arrests and imprisonments “weren’t started by surveillance by the Gestapo, they were started because somebody phoned them up, or someone sent them a letter, or somebody popped in and had a word with them.

“It was somebody saying, ‘I think it’s them, I heard them’,” he continued. “It was setting up the rumour, and once that snowball is running, it passed seamlessly from rumour to official process in a way that’s absolutely terrifying.”

He spoke of the frightening “speed” with which suspicion solidified into conviction at the roundtable. Balding agreed, saying that you could “feel the wave coming towards you, and you’re thinking, ‘I can’t fight this wave, I don’t know how to defend myself’.”

Olusoga responded that what Balding described could be seen in footage of Russian show trials. “When people have been set up and framed, and they’ve had denouncements and people shouting about them as enemies of the people, they go silent, they shrink. They don’t shout and scream and beg, and protest their innocence. They diminish. And you could see when that wave comes at you, we didn’t make great defence cases, we didn’t stand up and go, ‘This is ridiculous, can’t you see? I’m obviously a Faithful’.”

Balding said that she was “possibly too polite” at the roundtable, adding that “deflection is a really effective mechanism of defence, and it’s why politicians do it all the time, and I think we need to be really aware of that”.

Asked whether any in-groups and out-groups had formed on the show, Balding said that a group began to form comprising herself, Charlotte Church, Ruth Codd, Paloma Faith and Olusoga, but she believed a Traitor saw it emerge and thought “bang, we’re getting rid of this”.

“It goes back to people being identified as witches back in the day – women who know their own mind and will say it, and aren’t going to be swayed by groupthink. Celia [Imrie] would have been one as well. We’d have been in big trouble in a time when they were burning witches.”