The "Patriots" panelist admitted that the financial reward is far from high, shared the circumstances that led to his being removed from the screen for a period, and explained why he returned despite everything: "I have a role"