The two men—the CEO and senior data scientist, respectively, of a startup called Civis Analytics—would stay late at work, drinking bourbon and watching returns come in. Their office, a repurposed industrial space in Chicago's West Loop, would rattle every time the L train rumbled by.

As much as Wagner and Shor were following the political horse race itself, they were also watching to see how the race's oddsmakers were doing. The US polling industry has been suffering a crisis of insight over the past decade or so; its methods have become increasingly bad at telling which way America is leaning. Like nearly everyone who works in politics, Wagner and Shor knew the polling establishment was liable to embarrass itself this year. It wasn't a question of if, but when—and how badly.

It didn't take long to find out. About 10 days before the Iowa caucuses in February, two major polls came out: One put Hillary Clinton ahead by 29 points; the other, as if it were tracking an entirely different race, showed Bernie Sanders leading by eight. In the Republican contest, Donald Trump topped the state's final 10 polls and averaged a seven-point advantage. On the night of the caucus itself, the Civis office in Chicago was crowded with staffers gathered around a big flatscreen TV for a viewing party. They all watched as Clinton—and Ted Cruz—won the state.