We all know the benefits of a daily dose of physical activity by now. Not only does exercise tone your body so you can wear your favorite jeans, it even helps with your memory. But when it comes to retaining new information, working out may not do much for people who were exposed to high levels of mercury before birth, according to a new study that was just published in the journal Environmental Health Perspectives.

Most adults get exposed to some level of mercury through their diet, especially if they eat a lot of seafood. The metal often gets into water as a result of industrial pollution, and it makes its way up the food chain when fish eat plankton or smaller fish that are already contaminated. Because it accumulates this way, the United States Food and Drug Administration has set a limit on the amount of mercury (1 part per million) that cannot be exceeded in fish intended for human consumption.

Generally, two groups are more sensitive to the effects of mercury — tiny fetuses, and people who are regularly exposed to mercury.