Judge Aaron Persky handed Stanford University swimmer Brock Turner a six-month sentence in June 2016.
But county judges in California are elected, and if a petition to remove them from office garners enough signatures a vote will be held.
Such elections are rare - the last time a US judge was recalled was in 1977.
Tuesday's vote in Santa Clara County marks the first time a Californian judge has been removed in this way for more than 80 years.
Reacting to the result, the campaign to unseat Judge Persky said the voters of Santa Clara were the winners.
"We voted today against impunity for high status perpetrators of sexual assault and domestic violence," chair of Recall Persky, Michelle Dauber said.