‘Yes, it’s his decision, but you rarely go against what the Secret Service is advising’