Comments

1. #1 Lucas

   December 29, 2007

   This “bug” is hardly significant, and arguably is not even a bug. If this is a bug, then the graph of something like y=Re[I+x] should be empty because you need complex numbers to calculate it. This is really a matter of definition, and I don’t think that Wolfram made a bad decision here. I think if they had designed the graphing engine to work as Pais implies it should, then it would lead to all sorts of obnoxious problems in rendering graphs. It would make everyone’s life harder, and (more generally) graphs would have *less* information in them. That seems like a bad thing for mathematical research.

   Also, you don’t need to look at the source code to surmise how this works. I think it’s pretty obvious how Mathematica is computing things.

   As for open source software, I agree totally that using it *as a step in a published research result* is essential, but if you just want to work out an example, then what’s the problem? Also, my understanding about SAGE (the open-source Mathematica alternative) is that it has nowhere near the capabilities of Mathematica. I would be able to verify this myself, but the OS X version of SAGE seems to be quite broken (i.e. I can’t get it to work).

   This is not a screed against open source software. In this case, the closed-source software is massively superior to the open-source software–Mathematica is a technical marvel and a huge time saver. On the other hand, SAGE (or something like it) may surpass it one day, and I think that would be a good thing. They could probably start by getting the Mac OS X binaries to work correctly, or fix the broken Makefile. Grrr…

2. #2 Who Cares

   December 29, 2007

   What open source replacement do you suggest for Mathematica/Matlab?
   Seeing that I’ve been converting proof of concept matlab files into C/C++ just to avoid this from happening.

3. #3 Rob Knop

   December 30, 2007

   A couple of packages to look at :

   GNU Octave — designed to be a matlab replacement

   Maxima — a symbolic algebra system

4. #4 Greg Laden

   December 30, 2007

   To you mathematicians: I’m not telling you that I think you are using the wrong software. Use the software that works. But in science, you need to document and explain your methods and it sounds like Mathematica is unsuitable for science. So you can play with Mathematica but then, I guess, for the final “proof” you need to use your abacus or whatever.

   What I am saying is that you (the community of mathematicians) have let things get out of hand in the same way that scholars in general have let things get out of hand with the publication of our journals. You bought into the pretty pictures and were lulled by the sales pitch.

   Suitable opensource software may or may not be available to you … I know for all of the statistical work I do such software exists, but maybe not for math. But it does not just show up.

   You have to build it.

   This means you. Not the guy standing behind you.

   So what are you going to do about it?????

5. #5 Anonymous

   December 30, 2007

   Now, Miquel Pais (via the Quantum Pontiff) reports a bug in Mathematica that can’t be understood because you can’t look under the hood.

   
   This is a disaster.
   

   As Lucas points out above, the “bug” is perfectly easy to understand without looking under the hood, and it is arguably the right way to do things. If I were writing a computer algebra system, I would make it do exactly this, and I would argue with anyone who said it was a bug. I don’t expect a bug fix from Wolfram…

   So what are you going to do about it?????

   Nothing, becaue 99% of scientists and mathematicians don’t care about this issue. And they are perfectly right not to care about it. What do you do if something seems funny in the results from Mathematica or Maple? You send a bug report to tech support. This is the same thing ordinary users do when they find a bug in Maxima (the nearest open source equivalent), except that “tech support” is a mailing list of volunteers. Only an exceedingly tiny minority of researchers are comfortable enough with both computational algebra and heavy-duty programming to take a look at the source code and try to understand/fix it themselves. I’d be shocked if it is one in ten thousand users, at least for the commercial systems that actually have a lot of users. Almost nobody is capable of doing it, and the few who could usually don’t want to. (Otherwise they’d already be spending their time on Maxima development.)

   So what’s the advantage of open source? For commercial software, you have teams of experts who are paid to be responsive. For open source, you hope someone cares enough to look into your problem quickly. It usually works impressively well, but at its best it merely equals what you get automatically with commercial software. (I’ve used Maple/Mathematica pretty seriously since the late 80′s, and in my experience they are extremely helpful when bugs arise. They know they need a reputation for reliability, so they have no tolerance for mathematical errors. This is very different from most other types of software.)

   Open source certainly has its advantages, such as price, but this isn’t one of them.

   It’s nice to dream about a world in which you can take personal responsibility (intellectually) for every tool you use in research, but it’s impossible. Hardware and software are simply too complex for any single human to understand everything. What you end up needing is just a helpful and responsive community of people who do understand any given issue, and open source has no advantage there. People may be proud of how their open source research tools have no black boxes, but they are still leaving the inspection/understanding to someone else.

6. #6 Sam Centipedro

   December 30, 2007

   What a strange argument! A handful of people decide that perfectly explicable behaviour in one piece of software might be a bug (it isn’t, which makes it more amusing), therefore (in a leap worthy of creationists) we should all use open-source software! Do I hear cuckoos?

   Applying this argument, no scientist should have any equipment or supplies in the laboratory that has not been made by their own fair hands! How can you be sure that your balance doesn’t have a fault that makes it give wrong readings for some weights? Yes, you check it on standard weights, you know the supplier is highly reputable, but how can you guarantee it on all possible weights?

   While open source is a nice idea, this evangelism is rather tiresome. Put simply OSS doesn’t usually cut the mustard compared with the commercial stuff. I wish it did.

7. #7 Who Cares

   December 30, 2007

   Thanks Rob

8. #8 Guber

   December 30, 2007

   I suppose that you are knowledgeable enough to be able to “look under the hood”, and verify that all of Maxima’s algorithms are correct? Or at least fix the bugs you find? Because of you’re not, then Maxima is just as much of a black box for you as Mma is.

   BTW, simplifying Sqrt[x]^2 for x in Reals to x is not a bug.

9. #9 Ron

   December 30, 2007

   A commentor named Anonymous is telling the writer of this post that he can’t “look under the hood.”

10. #10 Ron

    December 30, 2007

    If you go back and read Dr. Laden’s argument, it is partially based on the following problem cited in David Joyner and William Stein, the paper at the end of the links if you follow them. Maybe instead of attacking Dr. Lade you should read the article he is referring to:

    –begin quote

    Suppose Jane is a well-known mathematician who announces she has proved a theorem. We probably will believe her, but she knows that she will be required to produce proof if requested. However, suppose now Jane says a theorem is true based partly on the results of software. The closest we can reasonably hope to get to a rigorous proof without new ideas) is the open inspection and ability to use all the computer code on which the result depends. If the program is proprietary, this is not possible. We have every right to be distrustful, not only due to a vague distrust of computers but because even the best programmers regularly make mistakes.

    –end quote –

11. #11 The Real Anti-Ron

    December 30, 2007

    Ron, smart of you to go deeper into the argument but if you go one level deeper you get this:

    Most of the documentation provided for Mathematica is concerned with explaining what Mathematica does, not how it does it. But the purpose of this page is to say at least a little about how Mathematica does what it does. Some Notes on Internal Implementation gives some more details.
    You should realize at the outset that while knowing about the internals of Mathematica may be of intellectual interest, it is usually much less important in practice than one might at first suppose.
    Indeed, one of the main points of Mathematica is that it provides an environment where you can perform mathematical and other operations without having to think in detail about how these operations are actually carried out inside your computer.

12. #12 CS

    December 30, 2007

    Mathematica attempts to simplify expressions in several functions. In this case, it simplifies (x^a)^b to x^ab before evaluating. It does this to shorten evaluation time for such functions as plotting and integrating. If you do not want expressions to be simplified, look into using Hold[] or the Compiled->False option. (Also, from the documentation for Sqrt and Power, it is stated that Power[Sqrt[x],2] is simplified to x.)

    Besides, the apparent “bug” here is that the given expression should be undefined if intermediate states are restricted to the real numbers. Mathematica handles complex numbers, in which case there is no problem with the expression. How is that a “bug”? What, is complex analysis not yet considered a valid branch of mathematics?

    The current behavior is explained in the documentation. Open source would be great, but if people can’t be bothered to read the documentation, how can we expect that they will actually parse through the code whenever something unexpected happens?

13. #13 Anonymous

    December 30, 2007

    Maybe instead of attacking Dr. Laden you should read the article he is referring to

    Oh, I have read it (I’m a mathematician and read it when it first appeared in Notices of the AMS). I don’t agree with it, and I think Stein and Joyner have an axe to grind, considering that they are both heavily involved in the development of SAGE (Stein started SAGE and Joyner is an active developer). Basically, I think Stein and Joyner have a valid philosophical point, which is that it would be nice if we could inspect or modify anything we wanted. If Wolfram decided he was so rich that he was going to release Mathematica as open source and continue to pay people to work on it, we’d all be better off. However, the underlying philosophical point is fairly minor and is simply not important for the vast majority of mathematical and scientific work. It’s foolish to sacrifice tremendous practical benefits for a tiny theoretical gain. Of course, Stein and Joyner want us to: they are among the tiny group of experts who would benefit the most from open access to the source code, plus they benefit greatly from having many people use SAGE.

    Plus I’m not attacking Dr. Laden, who seems like a fine guy. I’m just attacking some of his less well thought out ideas.

14. #14 Lucas

    December 30, 2007

    Rob: I agree with that quotation completely. For this reason, I think that if someone were to release a proof which used proprietary software X, then it would be looked upon with even more skepticism than even proofs using computers normally are. However, I doubt that Mathematica or Maple are so clever that it would come to this–indeed for the computer assisted proofs like the 4-color theorem or Kepler’s conjecture, the main problem is size of the computation. For the 4-color theorem, the computations are not particularly hard to do by hand, there are just a *lot* of them. I don’t think that Mathematica exceeds the cleverness of mathematicians, who, if the situation calls for it could reproduce the closed source cleverness of Mathematica. (On the other hand, it once took me 5 hours to evaluate an integral by hand that Mathematica did in less than 1 second.) The philosophical point is still valid, though Laden picked a very poor example to try and make it.

    “Applying this argument, no scientist should have any equipment or supplies in the laboratory that has not been made by their own fair hands!”

    But mathematics is different from other sciences. We have something called proof, and we require to be absolute. Consider something like the Riemann hypothesis, which has been verified for an enormous number of potential instances. If it were a physical question, it would be much better supported than general relativity, but mathematicians are agnostic on its truth. I’ve seen one number theorist say in all seriousness that he has no idea if it’s true or not, and he had studied the issue quite a lot.

15. #15 greg laden

    December 30, 2007

    This is developing into a very interesting discussion. Funny, all the mathematicians I know personally are much better behaved and more polite.

    I certainly do not want to defend the idea that this is a bug. I have no idea. But I do think that the arguments that you can have a black box doing your calculations just like lab scientists have black boxes doing their electrophoresis (or whatever) are valid. In fact, they are absurd. It is not true that this approach would be considered acceptable. Increasingly, lab work is outsourced or done with machines that are essentially black box like. I’m particularly familiar with work done along these lines in endocrinology and stable isotope research.

    The researchers that send samples out without being able to do the work themselves, or who treat their equipment like a black box and do not understand (and are not able to examine, adjust, etc.) what goes on under the hood are the bad ones … they are perhaps even crossing an ethical line.

    I am shocked to find that some mathematicians do not worry about this issue. Shocked I say!

    Perhaps many of you are too young to remember, or perhaps simply not well educated in the relevant areas, the MacClade fiasco. Do you? If not, go find out about it. MacClade was a piece of black box software that for quite some time produce phylogenetic analyses (cladograms) that conformed in all ways to expectations. there was no way to tell if the cladograms were correct or as advertized. You could not infer the inner workings by looking at the results. When it was discovered that there was a flaw in the software, and that many of the perfectly normal looking results over many months (a couple of years maybe?) of time and across numerous peer reviewed papers were simply wrong, shit, really bad shit, hit the fan. Some people in tenure-track jobs had their careers trashed, people with tenure were better protected.

    Perhaps mathematicians have special powers that allow then to know without looking what is going on inside the black box. But I doubt it!

16. #16 Lucas

    December 31, 2007

    I think that most mathematicians don’t have the time or inclination to look into how even open source software packages work. There is certainly no hope at all that mathematicians (as a group) would understand in intricate detail the often closed-source internal workings of their computers. Do you understand how the CPU in your Linux box works in intricate enough detail to be certain of its workings? On the other hand, when the source is open, the scientists with the inclination will frequently check its correctness (if the software is frequently used in research).

    Mathematicians in my experience do care about this. I’ve never seen a peer-reviewed paper in a major journal which uses Mathematica or Maple outside of the WZ algorithm. (This algorithm generates hand-verifiable certificates of correctness.) Admittedly, my areas of interest don’t often intersect with Mathematica’s or Maple’s main capabilities. I do know that Haken and Appel’s proof of the 4-color theorem is still a bit unsettling to me despite the fact I understand how it works, the code is completely open source, and the result has been replicated by several researchers in several different ways. I still feel like the 4-color theorem is just a little suspect because no one has ever really understood *why* it is true.

    I reiterate that using closed-source programs is fine for research, just so long as they don’t appear in the finished product. The situation is much the same as using heuristics, which every mathematician uses constantly. Heuristics likewise don’t appear in peer-reviewed papers.

    P.S. I didn’t mean to be rude, and apologize if my remarks came off as rude.

17. #17 greg laden

    December 31, 2007

    Lucas,

    That is very interesting, and it is interesting to learn how proprietary software fits into actual research in math.

    I don’t think one has to necessarily personally understand or analyze under the hood. For instance, I get the basics of how stable isotopes are measured in samples, I’ve seen it done, read the literature, but I can’t personally tell you if a certain lab is doing right or the details of how they are doing it. But I know a couple of people who absolutely can do this, and I can ask them.

18. #18 Anonymous

    December 31, 2007

    Mathematicians in my experience do care about this. I’ve never seen a peer-reviewed paper in a major journal which uses Mathematica or Maple outside of the WZ algorithm… Admittedly, my areas of interest don’t often intersect with Mathematica’s or Maple’s main capabilities.

    This must be an issue of research areas. Most of my papers, and many of the papers I’ve read, have user some computer algebra, but the referees have never even mentioned it as an issue (let alone complained about it).

    I think one issue is that it is rare for any computer algebra code to rely on unique features of a certain system. Usually it is pretty easy to translate it to any other system, and that tends to reassure people (since they know that if they suspected a problem, they could always run it on their favorite system to check, even though in practice they don’t do this).

    outside of the WZ algorithm. (This algorithm generates hand-verifiable certificates of correctness.)

    In many cases they are indeed hand-verifiable, but I’m pretty sure that most WZ certificates appearing in published proofs have never been checked by hand. This is another thing that is psychologically reassuring (and therefore frequently gets mentioned) but rarely gets taken up. Whenever a calculation is short enough to be doable by hand in principle, the author is likely to highlight this, but here’s one piece of evidence: I’ve seen many papers in which the authors claim something is short enough that it could be checked by hand, but I don’t recall any in which the authors announce that they did check it by hand.

    I do know that Haken and Appel’s proof of the 4-color theorem is still a bit unsettling to me despite the fact I understand how it works, the code is completely open source, and the result has been replicated by several researchers in several different ways.

    I agree that the Appel and Haken proof was unsettling, but not primarily because of the computer use. Rather, it wasn’t computerised enough, in the sense that elaborate hand calculations and case analysis were needed to verify that the computer part was actually complete and matched up with the human part. This interface between humand and computer proofs is by far the least reliable part. Humans are good at building conceptual proofs, and computers are great at systematically checking something simple, but this interface combined the weaknesses of both. The only record of it consisted of a lengthy sequence of hand-drawn diagrams published with the 1976 proof, and I do not believe anyone except Appel and Haken ever checked it carefully. For many years, there were rumors of gaps. This is a very good reason to feel unsettled.

    The only other proof I am aware of is by Robertson, Sanders, Seymour, and Thomas from 1995. It massively simplifies the Appel-Haken proof and in particular cleans up the interface and systematizes the proof to the extent that it is overwhelmingly easier to check. (In fact, they found this proof after trying to check the Appel-Haken proof and giving up.) I am confident that this proof is understandable and reliable.

    This is a good example of why the overall strategy for a huge computer-assisted proof has to be chosen very carefully and sensibly.

    However, there are almost no computer-assisted proofs of this scope. There are a handful of others (such as the Kepler conjecture or the projective plane of order 10), but most proofs that use computers are much less complicated and need very little computer time or custom code.

    I still feel like the 4-color theorem is just a little suspect because no one has ever really understood *why* it is true.

    I wouldn’t call it “suspect” but of course I agree with you. There are two issues here: what constitutes a proof, and what constitutes a beautiful, illuminating proof. Computer-assisted proofs may have some beautiful aspects, but they are almost never as illuminating as completely conceptual proofs. On the other hand, they are frequently every bit as reliable as human proofs.