But Congress could still give the IRS more power and resources to investigate taxpayers and collect.
Federal lawmakers are considering plans that would increase fines for tax preparers, increase questions for taxpayers regarding the deduction for home mortgage interest, revoke passports for some who owe back taxes, and allow private debt collectors to pursue people with unpaid taxes.
To fund things like highway projects, Congress is looking for ways to bring in more revenue without raising taxes, so it is considering ways to crack down on tax deadbeats.
But the plan concerns people like Edward Durham who say the IRS has abused the power it already has.
"It just smells wrong," said Durham. "There might be a time in the future when that would make sense but not right now."