Sometimes people walk around with only half a brain, or a large portion of their brain disconnected, or simply having never developed, or an extra large brain, and we usually take little notice. But when there is a five or ten or twenty percent difference between two groups of people we are quickly willing to use that to decide (as in the Bell Curve) that those people with the (on average) smaller brain are inferior. The fact that all the well known studies comparing groups of living people that show such differences have been shown to be bogus (i.e. made up or doctored data) is often ignored.
Anyway, the following is the abstract of a 1998 paper by M. Henneberg that is still relevant of some interest:
1. The hominid brain has increased approximately three times in size since the Pliocene, but so has the brain of equids. The tripling of hominid brain size has been considered as an indicator of increased mental abilities, as it coincided with the production of tools, weapons and other artefacts of increasing sophistication. No indicators of the increase in equid intelligence are known. Intraspecific correlation between brain size and variously measured ‘intelligence’ is, in modern humans, very weak if not completely absent. With the exception of size, there are no major differences between the anatomy of ape and human brains.
2. A study of 297 estimates of body height, 626 estimates of bodyweight and 276 estimates of the cranial capacity of hominids dated at various periods over the past 5 million years shows that the increase in hominid brain size was paralleled by an increase in body size.
3. In a sample of 45 variously dated fossil hominids, brain size correlates isometrically with body size.
4. Since the Late Pleistocene (approximately 30 000 years ago), human brain size decreased by approximately 10%; yet again, this decrease was paralleled by a decrease in body size.
5. Therefore, it may be concluded that the gross anatomy of the hominid brain is not related to its functional capabilities. The large human brain:body size ratio may be a result of the structural reduction of the size of the gastrointestinal tract and, consequently, its musculoskeletal supports. It is related to richer, meat-based diets and extra-oral food processing rather than the exceptional increase in the size of the cerebrum. The exceptional mental abilities of humans may be a result of functional rather than anatomical evolution.
There are earlier and later papers that indicate or support similar ideas, but this is the nicest summary.