‘If they ever took power, and they could, they would pack the Supreme Court, it would end up with hundreds of justices’