‘If Donald Trump loses the House and the Senate, I would worry. I’d be worried if I were him’