‘I do wonder whether we’ve entered a phase in our politics when support by people who say atrocious things maybe doesn’t have the impact that it once did’