Ed Brayton is a journalist, commentator and speaker. He is the co-founder and president of 
For the second time this year (here's the first), a court has issued a ruling that may void a large number of drunk driving convictions because the company that makes the breath test machine refuses to turn over the source code so that defense experts can test the accuracy of the machines.
Minnesota may be forced to drop thousands of driving-while-impaired cases and change the way it prosecutes others in the wake of a state Supreme Court ruling Thursday, prosecutors and defense attorneys agreed.The state's highest court ruled that defendants in drunken-driving cases have the right to make prosecutors turn over the computer "source code" that runs the Intoxilyzer breath-testing device to determine whether the device's results are reliable.
But the company refuses to turn over their code for testing:
But there's a problem: Prosecutors can't turn over the code because they don't have it.The Kentucky company that makes the Intoxilyzer says the code is a trade secret and has refused to release it, thus complicating DWI prosecutions.
"There's going to be significant difficulty to prosecutors across the state to getting convictions when we can't utilize evidence to show the levels of the defendant's intoxication," said Dakota County Attorney James Backstrom.
"In the short term, it's going to cause significant problems with holding offenders accountable because of this problem of not being able to obtain this source code."
This is the same company that refused to turn over their source code after a Florida court made a similar ruling in January. If the police want to use those results, they're going to have to insist on using only breath test machines made by companies that will reveal the source code. Given the notorious unreliability of the machines, it is a pretty obvious matter of due process if the defendants cannot perform their own testing on them.
Politics
Comments
I don't understand why you would need the source code to determine reliability. Can't you just test the devices directly?
Posted by: Ace of Sevens | May 8, 2009 9:45 AM
Who blinks first? I am guessing that a large number of canceled orders for "secret" breath test machines may force a certain Kentucky company to rethink their marketing efforts.
Simple question: We would like to use the results from your test machines as evidence in court. Will that be possible?
Answer: No, we will not release our source codes or otherwise allow independent examinations of our methods and reliability.
City: Ok, thanks anyway. Bye, now.
Posted by: threetorches | May 8, 2009 9:47 AM
Speaking as a code-monkey, I also am not clear on what source code analysis buys you, as opposed to straight black-box testing. I don't know how many kilo-lines we're talking about (possibly not many), but the only thing I can imagine doing with it is locking up some geeks with the code so they can reverse-engineer it and look for possible bugs. Maybe they can check whether the part that converts raw sensor data to the official (and legally admissable) reading is correctly implemented?
How are these machines usually validated? Someone has a few drinks, and simultaneously blows and gives a blood sample for comparison? (And can I have that job? ;-)
Posted by: Eamon Knight | May 8, 2009 9:56 AM
"Why can't you test directly?"
How do you know what to test? There might be things that the software doesn't consider, or deals with wrongly. How do you know? All you have is a single number result. So the magic box says 0.81 alcohol so you are guilty. Where do you start testing? Does it use a molecular weight scheme such that certain foods might bias results? Does it use electrical conductivity such that the amount of exercise may alter results due to lactic acid or something in the breath? Without the source code or the equivalent, how would you know what to look for?
If the machines were really good and solid to the tenth of a percent, or the concentration was so high that just normal testing would be fine, then I don't think you would have a right to look. That is in the decision. It is the edge cases where specifics may matter.
Posted by: Markk | May 8, 2009 10:08 AM
I'm also a code monkey and yeah, code analysis isn't everything but it is a component. If it's not flushing the cache properly, if values are getting confused, if it's not re-calibrating or if it has any one of thousands of bugs then the results may not be reliable. For my first job I worked in an environmental testing laboratory and worked on the calibration and reporting tools. I certainly didn't see all aspects of the business but I did see that through simple mistakes it was possible to end up with persistent errors. Black-box style testing can give a general feel but it can't easily tell you how they operate under field conditions and it can't help you determine if there are circumstances in which the results get 'confused'.
Think of the problems with electronic voting booths. I'm sure Diebold tests them extensively before releasing them into the wild, but what a single lab can do is very different than what happens which hundreds of millions of untrained people use them.
That said, my feeling is that any bugs are likely to be rare unless you can demonstrate these problems in practice. Analyzing source code is also a time-consuming, expensive process and I'd bet that any bugs are unlikely occur except in very rare circumstances which, considering the lower presumption of guilt in these cases, is probably not going to help anyone. Sounds like they're grasping.
Posted by: Tyro | May 8, 2009 10:14 AM
While I am for public disclosure of source code I really am not sure what it will prove in this case. My understanding of the source for these types of devices is that it isn't that sophisticated. Even if there are errors found in the code (and there will be) is that enough to prove the machine was giving erroneous results? There are so many other things that can go wrong (unintentionally or deliberately) during an arrest that can you pin down a wrong reading at that point in time to an error in the code? And if the source code is prestine (unlikely) how do we know the mechanical parts of the machine are properly validated and are working? Source code review is not going help with that.
(in my experience companies don't like sharing source code not because of 'trade secrets' its because its a bloody mess)
Posted by: yoshi | May 8, 2009 10:14 AM
Ace & Eamon:
Probably because, technically, the question isn't whether the machine works, but whether it is a reliable meter of the degree of alcohol in the defendant's blood at that time.
If the code has any flaws which might cause it to give an inaccurate reading, even if such inaccuracies rarely occur, that is some evidence which a defendant is entitled to present to show that the reading which was obtained is not a reliable measure of BAL, and therefore, grounds for the judge or jury to find the state hadn't made its case. (Exactly what those defects might be, I can't say.)
Mearly testing the machine won't do, because it might be that during the test, the conditions which might cause the machine to malfunction are not present, so it can't eliminate the possiblity that the conditions causing malfunction were present at the time of the reading of the defendant's breath.
Posted by: Learned Foot | May 8, 2009 10:21 AM
Let's not forgot about all the flaws like integer overflows that have been found in voting machines--look for news reports of negative vote counts or voting machines "counting backwards." I know of only one analysis of breathalyzer source code and while some of the items reveal a lack of understanding of source code (e.g., 60% of lines generating lint complaints don't mean 60% of the lines have bugs), there seems to be room for serious flaws:
http://www.sandiegodrunkdrivingattorney.net/2007/08/successful-dui-breath-test-machine.html
Posted by: John Smith | May 8, 2009 10:33 AM
Having the source code is very much like having a theory of how the device work (instead of reverse-engineering it, that is like having just an hypotesis)
With the source code you can design experiment (that is, tests) to verify the device.
An example could be the precision of the software variables used.
(For those unfamiliar with programming, software stores information in different kind of variables.
A number, for example, can be stored in a FLOATING-POINT variable, which can contains a great deals of decimals, like 4.23553248383 or 7,123827722.
Or it can be stored in an INTEGER variable, which can contains
only integers, like 4 or 7.
The information of how data are stored allows you to test the device for precision).
Posted by: diegopig | May 8, 2009 10:43 AM
Re John Smith @#8: Thanks for that link; yes that does sound scarey. I'm actually trained as an electrical engineer, so I hope I have some awareness of issues beyond the code itself. To me the most alarming finding in that list is their apparent abuse of the A/D convertor. You can have the best internal algorithms in the world, all perfectly implemented, but if your input data is flakey for whatever reason, it's all GIGO.
Posted by: Eamon Knight | May 8, 2009 11:03 AM
Black box testing can validate that a device worked or didn't in the specific test conditions. But field use will never exactly match testing conditions.
In a different case, where the defense did get the source code, here are some of the highlights of what they found. Note this was a Draeger AlcoTest 7110, not the same device or manufacturer as the one we're talking about in this thread. It's simply to prove an example of the type of information that can be gained by examination of source code. Full disclosure, Draeger is a competitor of the company I work for.
Posted by: Abby Normal | May 8, 2009 11:20 AM
Speaking as a former embedded-systems code monkey, I can add that the source code is necessary but not sufficient. Machines like this are almost always based on custom hardware designs, so, besides the firmware source code, you'll also need all the details of the hardware, the manufacturers' specs of all the major parts, and all the design documents that describe how the hardware and the firmware work together.
I can also add that all of this will be far beyond the abilities of any ordinary jury to figure out. Knowing how to work Excel or Word, or even knowing how to write programs on a Windows box, will not help you understand firmware.
Posted by: anon | May 8, 2009 11:37 AM
What iPhone doesn't have an app for that??? - DJ
Posted by: DingoJack | May 8, 2009 11:39 AM
that firmware analysis of the Draeger device is most interesting to me. the kinds of errors identified have a familiar feel, even though i don't write firmware.
they're the sort of things i'd expect to see happen when a product is developed by a too small programming team, without sufficient structure to the business process that's driving the development --- with feature lists and deadlines changing in mid-flow and marketing or sales being given more attention than development.
they're the same general sorts of errors i keep finding in my own code, or leaving in for lack of time to beat them out. fortunately, my programs are in-house only, never delivered to anybody else.
i'm coding that way because i'm writing stuff i'm really too junior to be designing and implementing the sorts of applications i'm writing by myself, and i don't have the office-politics pull to put my foot down when the goalposts move on me. hence, errors i know can occur don't get detected, other errors don't get prevented that might be, sufficient (or, sometimes, any) testing is not done (because that would take time, and the code "just needs to be written" --- usually by yesterday), and so on. code gets written by trial and error because the requirements are never stated clearly enough, and the actual design is fuzzy at best, as well as ever changing.
but my code won't have to produce legal evidence to stand up in court; the only people who'll be hurt by its shortcomings are the employees of my company, to include myself. you'd think a breathalyzer maker would take a bit more care.
Posted by: Nomen Nescio | May 8, 2009 12:31 PM
Wow, is Diebold hiring? I quit CoSci after high school but even I wouldn't screw that up.
Posted by: Brandon | May 8, 2009 12:44 PM
I have to honestly say that there's no reason source code shouldn't be publicly available as part of the audit trail of any device whose evidence is likely to wind up in court.
Posted by: Brian X | May 8, 2009 1:13 PM
Am I being too cynical in believing that the impetus behind this effort to get the source code released is likely coming from defendants looking for a way to get off on a technicality?
I am sure there are a few legitimate cases where an injustice may be happening, but it doesn't take looking at the source code to confirm that the devices are working fine in a large majority of cases and scenarios.
Drunk driving is a serious business, and I hope that this kerfuffle doesn't result in thousands of dangerous drivers walking.
Posted by: tacitus | May 8, 2009 1:27 PM
It's certainly possible that the people complaining were, in fact, drunk, but the mere suspicion that someone has committed a crime isn't enough; you have to actually prove it, and that means we have to prove beyond reasonable doubt the reliability of the tests.
Note that this doesn't imply a jury would be required to parse the code; it implies that we could have dueling expert witnesses.
Posted by: Anthony | May 8, 2009 1:46 PM
tacitus: The last study I saw on the subject (I don't have a cite handy, though) was that about 70% of DUI convictios were for BACs of .10 or less. In other words, for people within .02 of the limit in most jurisdictions. With that small a tolerance, all variables on the performance of a testing unit are more than "technicalities:" They are an assurance that the defendant is getting a fair trial.
Posted by: kehrsam | May 8, 2009 1:55 PM
It would be interesting and maybe contentious if the code is utilizing heuristics. If so, the quality of the guess may be arguable and even the fact that the code classifies analysis by educated guess might disqualify the results in such a legally significant context. Classification rules would also be subject to legal rebuke. However, for such a small device with such a specialized and limited role, I find it unlikely that any embedded software would use intelligence methods.
Posted by: James Taylor | May 8, 2009 2:05 PM
The articles on this never say anything about voiding convictions. All it says is that this decision may affect cases currently being prosecuted. I would imagine most convictions are safe; for one, most of those convicted likely did not request the source code during their trials. Even if they did, the defendants have to make the case that the source code is relevant. The Minnesota Supreme Court decision is based on two different cases, but only in one of those cases did they say the defendant had showed the code was relevant. Of course, attorneys in current cases now know how to present the issue.
Posted by: CS | May 8, 2009 2:07 PM
If I was a juror, I'd want to know the equipment used gave valid
readings if calibrated properly.
Proper calibration is also another sore point. I've been on a jury where that question came into play.
Posted by: rnb | May 8, 2009 2:25 PM
Given the known problems with the accuracy of breathalyzers, why are the police not obtaining blood tests on arrest? The breathalyzer result provides probable cause for the arrest - the blood test would then provide much more reliable confirmation. Relying on breath tests is just asking for trouble.
Posted by: Bill Poser | May 8, 2009 3:08 PM
Another interesting aspect of this relates to the ownership of the code. I skimmed/read the Minnesota opinion last week and the court implies (maybe even stated) that the State of Minnesota owned the code pursuant to the original RFP under which the machines were bought. In fact, I got the impression (being too lazy to go back and investigate fully) that an earlier decision by the court had already concluded that the source code was in the prosecutor's possession/custody/control (i.e., that they didn't have an excuse for not producing it).
I would assume that state public safety departments probably acquire their equipment using similar methods (i.e., similar RFPs). Thus, the company may very well have a problem that they have sold the same code to multiple customers. Their unwillingness to give it up probably indicates a lack of confidence in the defensibility of the code (and, knowing enough about the law of evidence and a smidgen of how breathalyzers work, I am confident that any expert worth his salt could utterly destroy the credibility of a breathalyzer reading). But, they are also probably unwilling to give up the code to any one state because doing so will amount to a confession that they sold the same code twice, which would invalidate their numerous contracts and require them to to cough up a lot of the money they have collected selling breathalyzers (the first customer would demand license fees for all the subsequent sales, and the subsequent buyers would demand their money back since they wouldn't own what they thought they had bought). In other words, it would put the company out of business.
I have no sympathy for them. If they can't be trusted to write/understand a basic contract, why do we trust them to write scientifically valid software upon which we will stake people's liberty?
The next logical step is for the states to pool together and fund a working group to develop a reference hardware platform with open source firmware that the states could then contract with various companies to manufacture. However, glancing around the webs (e.g. http://www.motorists.org/blog/breath-alcohol-analysis-how-reliable-is-it/) it looks like maybe there first needs to be a more rigorous scientific debate as to whether this type of testing can ever be reliable, no matter how well we construct the tools.
(I am not trying to be an apologist for drunk drivers, I thing DUI/DWI are serious crimes and should be prosecuted/punished, but only using valid means. Also, if the above linked site is woo and the science is more settled than my 5 minutes of Googling reveals, then I apologize, and am more than willing to hear the counter arguments)
Posted by: automandc | May 8, 2009 3:42 PM
Remember, reliability is simply repeatability, whether correct or not, and accuracy depends upon the assumptions the designers made - if the machine produces repeatable results within the parameters of the designers' assumptions, then the device can be said to be both accurate and reliable. That doesn't mean it has produced evidence of a certain level of alcohol in the blood - for that, you need to know about the design.
As a programmer, I can tell you that having the source code isn't going to tell anyone about how accurate or reliable the machine itself is, but it *will* tell you about the assumptions the designers made, and that is actually more important. If, for instance, it is discovered that the designers made a medically-unsupportable design decision, then that would suggest that the device may have other design flaws. My guess is that the company is afraid of any design flaws being found and harming the device's ability to provide evidence in court.
Posted by: Ryan Egesdahl | May 8, 2009 3:50 PM
then they seem to be shooting themselves in the foot with their current actions, which apparently have exactly that same effect. if your guess is correct, then it's no longer relevant whether or not the code is actually flawed; if the company that produced it are going on public record acting as though it were, it might as well be.
Posted by: Nomen Nescio | May 8, 2009 4:47 PM
It should be a matter of public policy that the source code for voting machines, breathalysers, and any other machines that are paid for with public money and serve a public use should be open source, or at the least, available for public examination.
Posted by: Kevin | May 8, 2009 5:30 PM
Breath analyzers do not measure the blood alcohol content , they measure a family of chemicals in your breath (methyls? possibly). They make several generalizations to corrolate BAC to breath Methyls, few that come close to reality. 1st not all of the Methyls you exhale are from breaking down alcohols that you drink several are from metabolic breakdown of food or medicine, cops are to ask if you have hypo or hyperglycimia(?)or on any medicine before they start a breath test because the manufactures know that for example diabetics can blow .28 with no BAC and many medicines invalidate or obscure correct breath test results. 2nd, they make a generalization that your lungs will exhale an exact fraction of the Methyls in your blood. Medicial instrament manufactures know that neighter the average lung function or an individuals specific lung function are constant and can vary by more than 60%. 3rd these machines are temprature sensitive but at least one model only self calibrates once after each recharge when it may be in an A/C room not in the field . 4th air speed through device is essential to accurate measurement, but from black box testing of several models if velocity is between the min an max set points it starts a test regardless of air speed during the middle or end of the test which can have a .03 difference in output. These are some of the errors in design that have come to light from looking at source code
Posted by: pmiller | May 8, 2009 6:46 PM
Posted by: llewelly | May 8, 2009 10:57 PM
Isn't it hard to breathalyze dead people?
Posted by: Wesley R. Elsberry | May 9, 2009 6:43 AM
Public money to pay for a device that essentially tests whether a citizen is guilty or not should be held to extremely high standards. Medical devices must undergo scrutiny from the FDA which includes source code analysis. Testing the device (dynamic testing) will give you a good picture of its reliability but it's only as good as the number of scenarios that can be tested. Even simple devices may have thousands if not millions of different scenarios - some scenarios are difficult to create. Source code analysis (static analysis), if you use the right tools, will help you get a more complete picture and will help find many potential flaws. Many of them are unlikely to ever bear fruit in practice, but some can be major problems. A complete testing strategy should include a static component.
Posted by: Andy | May 9, 2009 9:16 AM
The 6th Sense was going to be about a ghostly Highway Patrolman prowling around punishing those that drive drunk, but the studio thought the tag-line was a little lame*, they went for a complete re-write, and the rest is history. - Jeremiah
*"I breathalyze dead people" :)
Posted by: 'ere I am JH | May 9, 2009 10:44 AM
This is an interesting case. I'm yet another firmware monkey, so I can sympathize with both sides. I do biometrics, so all of my code ends up giving answers based on probability and lots of testing.
Yes, looking at the code will give you an idea of how sloppy it is. If it's like most closed commercial products (especially those for government clients), it's probably going to have some pretty scary stuff in it. That being said, anything that affects the accuracy with any reasonable probability is almost certainly going to show up in properly conducted black box testing.
That's not to say that they should be doing that kind of testing for the court case. They're supposed to do that testing before they standardize on the device. That way, when somebody challenges them, they can say something like, "In 99.6% of our cases, the reading was +/- 0.003. Do you have a reason why you should be different? One that we can test?" If not, they're stuck arguing that they're just really unlucky.
Posted by: Troublesome Frog | May 9, 2009 2:33 PM
IAYACM, (... Code Monkey), and I have written code to exacting specifications for both medical and FAA certifications. It sounds like this code would not pass the most cursory verification effort, which should be a requirement for anything intended to produce evidence of guilt admissible in a court of law.
For that matter, nothing based on Windows (with the unknown vagaries of its operation) should be certified either.
Posted by: Engineer-Poet | May 11, 2009 1:15 PM