Libertarian presidential nominee Gary Johnson said it's "game over" on any chance of winning the White House if he does not make it to the debate stage.

"Winning the election, yes, I would say game over," Johnson said on "Fox News Sunday" with Chris Wallace.

Johnson would need to be at a 15 percent threshold to join Hillary Clinton and Donald Trump on the debate stage. The Commission on Presidential Debates announced that the five polls it would use to determine that threshold are ABC-Washington Post, CBS-New York Times, CNN-Opinion Research Corporation, Fox News and NBC-Wall Street Journal.

Johnson said he is at 10 percent in those five polls but has seen an increase of about 4 percentage points over the past six weeks.

"We're optimistic we're going to actually get into the debates," the former New Mexico governor said. "We're spending money right now in many states, and in five states right now we're at 16 percent. So I'm just really optimistic."

Earlier this month, the presidential debate commission directed venues to prepare for a three-person forum.

Johnson said getting to the debate is just phase one.