Introduction

Any legitimate election involves secret voting but public vote counting.  This is “rule one” in elections.  Maintaining the privacy of the voters (so that they can't be retaliated against in employment or the like for their voting pattern) runs contrary to the need to secure the election system, because unlike a financial transaction there's no possible “receipt” that the voter can use to prove how they voted (2).

In a classical “paper vote, hand count” pre-technology election (say, Tombstone AZ in 1880) the ability of the public to effectively watch the counting process was unquestioned.  Any attempt to count the vote in secret could well have led to gunfire, as actually occurred in 1946 in Athens Tennessee:
The Battle of Athens Tenn 1946 (Video):



As elections have gone high-tech and computerized, the monitoring for them has to follow into the world of high-tech.  In this article I'm going to show you an actual election that looks visibly wonky that took place on March 13th 2012 in Coconino County AZ.

This is going to be long because I need to bring you up to speed.

The 3/13/12 Coconino County Election:

This was a series of town elections run by the Coconino County elections office under contract.  The election gear involved was “Diebold type” (3).  For our purposes this means four things:

1)    The security of the central tabulator (4) software seriously stinks.  The core vote tally database runs as an application talking to a Microsoft Access (“MS-JET”) database.  If you open the data with the approved vote processing program (“GEMS” for “Global Election Management Software” (5)) everything looks secure.  There's a password, audit log, etc.  Great.  But open the same data in a copy of MS-Access and the security vanishes.  No password needed to get in, every element of the election can be hand-tweaked and no audit log entries are left while working in MS-Access. Nice.

2)    When mail-in votes are being processed, the scanners are very primitive.  They are “mark sense” scanners that record, for example, that a fill-in-the-dot bubble was marked at position 20 over, 28 down.  It “knows” that's a mark for candidate “x”.  It doesn't scan the whole page or store a picture image of the ballot.  Worse, the scanner stations don't do any local recording of what votes were cast �" not electronic, not paper.  There's no memory card inserted in the slot for one.  There's a paper “cash register tape” but it records how many ballots were run through, NOT what the votes are.  Vote tallies flow over an old-fashioned serial port wire (yes, RS-232) to the central tabulator which is the sole (and easy-to-hack) location where the vote totals are kept.

3)    In Arizona, once a vote has been electronically tabulated, you can't hand-count it.  Period.  With two exceptions: if it's extremely close (within 1/10th of 1%) you can get a repeat electronic count, and if that doesn't match you might possibly get some hand-counting.  And in some races there's a 2% hand-count spot-check �" if both major political parties agree.  This is becoming rare, and doesn't even affect most very local races.

4)    Arizona requires “federally certified” voting systems.  That means that in theory, they've been checked out by a private test lab (6).  This also means that they're “legally frozen” they can't be changed from the initially approved setup.  So all operating system updates are banned, as is any anti-virus programs.  The only way they can survive is completely offline and disconnected from anything else.  This is called “air gap security” and it works well so long as it isn't violated.  Data going in (candidate names, precinct numbers, etc.) are typed in.  Vote totals come in from other bits of certified gear (touchscreen voting machines and optical scanners).  Data going out (election results) are supposed to be burned to CD-ROMs (a CD “burner drive” is provided for this purpose) and those CDs go over to an Internet-connected station for uploads.  Nothing is supposed to go back in from any non-certified system, Internet-connected or otherwise.

For our purposes we'll focus on the Flagstaff Mayoral primary and the Tusayan recall elections.  Let's look at the issues with each:

Flagstaff: this was an all-mail-in election �" no precinct voting.  Red flag there because a serious variance between the precinct and mail-in votes can be an indicator of trouble as it's more difficult (but not impossible) to hack both.

Tusayan: late on election day a flood of provisional votes came in totaling 20% of the overall vote �" an incredibly high number and enough to swing several races.  Most voting was by mail.

I was called in to look at this election by a guy who lost the Flagstaff primary, Paul Reilly.  He thought that he campaigned very hard and was the sole Democrat running in this fairly left-wing (partially college dominated) town.  By the official results he lost big.

The Observation Process Post-Election-Day

I filed a public records request for the electronic “debris” from the election.  The exact structure of this kind of request will vary based on the voting system gear involved �" drop me Email if you want details for your county (and yeah, I can find out what any county or city is running). 

The initial incoming data revealed something important: somebody was using a USB memory stick (aka “USB flash drive”) to move data in or out of the server.  And they were using it a LOT. (7)

Here's what the “Windows System Event Log” from the central tabulator said was going on the day BEFORE election day (right after they scanned about 2/3rds of the mail-in votes for the Flagstaff race):

3/12/2012    5:27:27 PM    Removable Storage Service    Information     None    134    N/A    COCONINO    Received a device interface ARRIVAL notification for device: ?USBSTOR#Disk&Ven_SanDisk&Prod_U3_Cruzer_Micro&Rev_4.05#0000187DA572D9A8&0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

3/12/2012    5:51:16 PM    Removable Storage Service    Information     None    135    N/A    COCONINO    Received a device interface REMOVAL notification [deleted the “device” info from here down, it's the same]

3/12/2012    6:08:34 PM    Removable Storage Service    Information    None    134    N/A    COCONINO    Received a device interface ARRIVAL notification

3/12/2012    6:09:18 PM    Removable Storage Service    Information    None    135    N/A    COCONINO    Received a device interface REMOVAL notification

3/12/2012    6:12:40 PM    Removable Storage Service    Information    None    134    N/A    COCONINO    Received a device interface ARRIVAL notification

3/12/2012    6:13:57 PM    Removable Storage Service    Information    None    135    N/A    COCONINO    Received a device interface REMOVAL notification

3/12/2012    6:34:14 PM    Removable Storage Service    Information    None    134    N/A    COCONINO    Received a device interface ARRIVAL notification
 
3/12/2012    6:36:17 PM    Removable Storage Service    Information    None    135    N/A    COCONINO    Received a device interface REMOVAL notification

This is from the System Event Log, lines 913 through 932, in chronological order (note timestamps).  During a period a bit over an hour long a USB flash memory device (“Cruzer Micro” by Sandisk corporation) was inserted and removed four times.

This is a problem, folks.

First issue: remember that “air gap”?  Well that's been thoroughly violated.  Instead of bothering to burn CDs they moved data in and out with this critter which is a definite known viral infection source if the other computer it's being bounced back and forth to is infested.

But much worse is the pattern of insertions and removals.  Let's look at it again:

·    Comes in at 5:27pm, stays for about 24 minutes, leaves and goes elsewhere for almost 20 minutes.

·    Comes back at 6:08, stays for a minute, leaves.

·    Comes back for a minute or two each at 6:12 and 6:34.

So what's going on here?

Well on the second night of vote processing (just before polls close) we see the same thing start and then continue into the evening.  The day after election day we see the most activity, running from just after high noon to around 5:00pm.

Those events are “sort of explainable”: the results being compiled as they come in from the precincts and provisional processing are being written out to USB memory stick as it occurs, to be uploaded to the Internet on another station.  OK.  I can see that.  It's stupid as hell, risky, illegal (violates the hell out of the certification rules) but yeah, a lazy elections admin might do that.

But the night before election day?  About 26 hours before polls close?

NO.   Hell no.  There's no possibly quasi-lawful explanation for the memory card insertions and removals of 3/12 as shown above.

A single one might be explainable as “we're backing up the day's data”.  Yet again they should have burned it to CD-ROM but OK, I can see one.

Four?  Across an hour-plus?  No.

What it does look like is the critical central tabulator data file is being copied out and carried elsewhere �" where it can be hacked on any station with MS-Access on it, including a laptop, etc.

Or put another way, it doesn't look like anything else.

What I'm doing now is waiting for additional records I've requested from this county before trying to go further figuring out what these clowns are doing.  Meanwhile I posted these initial results, and had to go into the county elections HQ to clarify some records requests.  Let me show you how that went:
Arizona Election Officials GONE WILD (Video):



Sigh.  As an aside, the right to record officials in a public space is legally unquestioned in Arizona �" and to their credit the Flagstaff PD actually realized that. (8)

It's not that I care much about whether or not I can film in their lobby.  This matters during the most critical election periods �" if I or anyone else sees something wonky I need to be able to whip out a camera and document it without some half-wit screaming about their non-existent right not to be filmed as public officials doing a public function in a public place.

Because the cop “got it” (thank the deity of your choice) it's possible this mob of election lunatics caught a clue and won't be filmed acting out like this again.  One can hope.

Anyways.  This is what modern election observation looks like.  Geeks are needed to look at the electronic debris.  People with guts have to know the election laws inside and out and demand the outside observation (as minimal as it is) found in AZ law.

Oh, and I wish I could tell you which other computer this memory stick was moving back and forth to.  I should be able to �" under AZ rules there's supposed to be a camera pointed at the central tabulator.  But the morons (or criminals?) pointed the camera at the scanner stations, which reveal just about nothing.

Reforms Needed

The biggest change needed in AZ law is that we need to link this review of the electronic records of elections to the formal rules for monitoring elections.  A challenge to a bad election has to be filed within five days of the final election results (the “final canvas”) yet public records can be delayed for between 10 and 30 days.  This is the single craziest part of AZ election laws.

We also need the right to do pre-election and post-election public reviews of the election gear, much like how winning race cars are taken apart looking for hidden nitrous oxide systems, oversize fuel tanks and other forms of cheating.

Until then, election observation is going to continue to be a cat-and-mouse-game with election officials who are either screwing up left and right and don't want that seen or are outright cheating �" which often looks damned similar.

Welcome to my world :(

 

1. I am a member of the Board of Directors at BlackBoxVoting.org �" a national non-profit well known in the area of electronic voting system investigation and reform, active with Bev Harris (now Executive Director) since mid-2003 before the formation of the .org.  I am also a member of the Board of Directors of the Southern Arizona chapter of the ACLU (not writing here in connection with them) and also a member of the Pima County Election Integrity Commission, a body that advises the Pima County Board of Supervisors (again, not representing them in this writing).  I also serve as Treasurer of the Pima County Libertarian Party.

 

2. There have been various cryptographic “high tech” proposals to ensure that a particular voter's vote was accurately counted but as we'll see, none of these are foolproof because the process behind the “crypto” has to be public.  Put another way, as we'll see even basic electronic transparency is a total failure today �" adding an additional high-tech layer doesn't solve the fundamental problems and introduces new issues along the way.

 

3. This means the stuff started out as “Global Election Systems Inc.” (GESI) which was bought by Diebold in 2002 and turned into “Diebold Election Systems Inc.” under the same management.  Once they suffered badly enough in the PR department with a number of horrible failures, Diebold renamed them “Premier”, then tried to sell the mess to competitor ES&S, then the US-DOJ declined that deal on monopoly grounds, then what was left got sold to a Canadian company name of “Dominion”.  Which also bought Sequoia which was briefly owned by a Venezuelan outfit with ties to Chavez.  Sigh.

 

4. In most election systems this is the one central computer that tallies the votes from the various smaller systems.  It also controls how the races are laid out, ballot styles and types, etc.  In many cases it's a “one stop shopping place” for fraud.

5. The MS-Windows program icon for GEMS is a fist holding a globe.  It's a more colorful variant of the corporate logo for Dr. Evil in the Austin Powers comedy series.  Not making this up!

6. Yeah, but there's more bad news.  There's only been four test labs ever approved by the feds.  Of those three have been thrown out for poor performance: Wyle and Ciber (formerly “Metamore”) both based in Huntsville AL and Systest.  Wyle and Ciber are in Huntsville to be next to the Redstone National Arsenal where they usually test military aerospace software systems...such as ICBM control units.  I hope they do a better job there!  ibeta is the only lab never thrown out of the process, which makes sense as they normally test software that a lot of people would complain about if it went belly-up.  Video games.  Except they quit voluntarily in disgust at the whole mess in late 2010.

7. The System Event Log goes back to mid-2011 and it's used that far back if not further, in every election.

8. Technical note: I had a junk “obvious camera” in hand (or set on the countertop) in most of these, and then set my Android 4G Tmobile smartphone to record and live-stream the results to a server in Sweden via the bambuser application (http://bambuser.com).  Starting at 1:18 in you can see the results of the camera live-streamed to a tablet computer sitting on Paul Reilly's coffee table in his living room.  Election director Patty Hansen makes a grab at the obvious-cam at one point �" even if she'd smashed it I'd have had a record of it on the cell phone.  This is an example of state-of-the-art “cop-watching” and it changes the whole game.  You need a 4G data connection to play this game correctly.

Visit BlackBoxVoting.Org, where Jim March is on the Board of Directors - a national non-profit well known in the area of electronic voting system investigation and reform.