On Tuesday, government figures showed that foreign demand for Treasurys fell by the largest amount on record in December.
China cut its holdings by $34.2 billion to $755.4 billion, losing the top spot in terms of foreign ownership of Treasurys to Japan.
Japan also cut exposure, cutting ownership of Treasurys by $11.5 billion to $768.8 billion, a much slower pace than China.
"I am surprised China has not dropped more," Rogers told CNBC.com.
Asked if the US should be worried about this trend, Rogers, who does not hold US Treasurys, said: "Of course. The US should be worried about everyone lightening up – not just China."