The comedian made the claim while appearing on "Jimmy Kimmel Live."
"You gotta be real careful around here," Allen noted. "You get beat up if you don't believe what everybody else believes. This is like '30s Germany."
Allen, 63, plays an outspoken conservative on the sitcom "Last Man Standing" and is one of the few actors in Hollywood to profess having right-wing leanings.
When Kimmel asked him about attending the inauguration ceremony the "Home Improvement" star's eyes bulged and he stammered: "I was invited, we did a VIP thing for the vets, and went to a veterans ball, so I went to go see Democrats and Republicans."
"Yeah I went to the inauguration," he added.