‘And in a lot of ways, I was intimidated by the claim that anyone who notices it is an anti-Semite’