Mark C. Chu-Carroll at Good Math, Bad Math has an excellent post in which he mercilessly dispatches the misbegotten idea that infinity is a number. It’s not.
How do we know? He explains it very well, but the fundamental thing to remember is that number is a word, and like all words it has a specific meaning. The words we use in mathematics and science have to be very carefully defined to make sure we’re all talking about the same thing, and the definition of infinity is simply not even remotely compatible with the definition of number.
To take an extreme example, if you define dragon as “A small carnivorous mammal (Felis catus or F. domesticus) domesticated since early times as a catcher of rats and mice and as a pet and existing in several distinctive breeds and varieties.” then my parents own two real live dragons. “But the definition is wrong!”, you exclaim. No, definitions can’t be wrong. They simply are. The only problem is when people use definitions that aren’t consistent. You run into these problems with trying to say that infinity is a number. Define “number” in such a way as to include 1, 2, 3 and all the rest, along with the usual relationships between those numbers and you’ll find your definition cannot include infinity. It just can’t happen. Might as well try to say your cat is a number.
Now if you are suitably clever you can do some finagling and come up with a way to treat infinity in a manner that doesn’t fit the field axioms but still has its uses as something similar to a number. But you don’t have anything resembling ∞ + ∞ = 2∞, so it’s still doesn’t fit the usual definition of number.
Do we ever have these definitional issues in physics? Sure, all the time.
This question was on an exam for the 201 class for which I do some TAing. Say you have a ball held a meter over a table, which is itself a meter above the floor. The potential energy is mgh, but which h should you use? One meter or two meters? It was a short answer question, which should have tipped the students off that there was more to it than just one or the other.
The answer is of course that it doesn’t matter. You can chose whatever reference point for potential energy that you want, so long as you’re consistent. Pick a definition for potential energy and stick with it and you’re golden. This is first-semester material, but it crops up all throughout physics at every level. In electrodynamics and other subjects, we often have even more elaborate choices for how we talk about potential. We have all kinds of screwball gauge choices we can use to help solve problems. You can define the gauge pretty much however you want so long as you pay attention to maintaining consistency in your definition, and so long as your gauge choice fits the definition of gauge choice in the problem at hand.
All this is an extended way of saying that a lot of misunderstandings about physics and mathematics could be resolved by clarifying definitions.