Lots of people are trying to force you to care about the things they care about.
True, some of these issues are really important. But if you don't care, you don't care.
You don't have to pretend. And you don't owe anyone an explanation.
It is not a man's duty, as a matter of course, to devote himself to the eradication of any, even the most enormous, wrong; he may still properly have other concerns to engage him.
See, a lot of self-righteous people can't imagine how anyone could make a different calculation about the most important causes in the world. The person who donates to hungry children finds it absurd that others donate to animal shelters while there are still needy kids out there.
But this is just another variation of how specializing makes things more efficient. If everyone works on the causes they care about most, their passion and efforts will go the furthest.
But Thoreau continues that even if someone doesn't want to help eradicate a wrong, it is "his duty, at least, to wash his hands of it, and, if he gives it no thought longer, not to give it practically his support."
If you're not contributing to wrong, that is enough.