Our 26th president, Teddy Roosevelt himself, often took nude plunges into the Potomac. And men's high school and college swim teams would even train in the nude well into the 20th century.
Outside of all-male groups, men were expected to cover up. Clothing that one wore specifically for swimming was firstly a modesty garment, and a practical garment second. Swimsuit regulations were the strictest in the U.S. in the late 19th and early 20th centuries, when men were prohibited by law and custom from going bare-chested in public. Woolen one-piece suits with cutoff sleeves and legs were the common style. In the 30s, that began to change, and men actually organized protests arguing for the right to go topless, many of whom were cited for public indecency. Finally, in 1936, the topless ban was lifted in Westchester, New York, and other cities and states followed suit (no pun intended!), mostly for economic reasons.