‘I will tell you, in the first term, President Trump spoke to me very seriously about all three Supreme Court vacancies, and I said no three times’