Larger brains have evolved a number of times. It seems that there has been a trend over several tens of millions of years of evolution of larger brains in various clades, such as carnivores and primates. There is probably a kind of arms race going on among various species in which a larger brain is an asset.
However, as you imply, a really large brain (like the extraordinarily large human brain) seems to be very rare. One of the reasons for this is that there are at least two major kinds of costs of a large brain that outweigh the benefits. One kind of cost is the energetic expense of having this large brain. Over 10% of the day to day energy demands of an adult human go to the brain. The total energy requirement of an infant can be over 60% while the brain is both a relatively large proportion of the infant’s body, and is undergoing a great deal of growth. The brain tissue is very picky about things like the temperature it requires for normal function and the kind of nutrient it needs.
The other negative, which is also a positive, is that behavior mediated by cerebral function (roughly, “thinking”) requires learning. Organisms with larger brains are not born “knowing” what to do. This is a benefit because it allows for more flexibility in behavior and it allows for kinds of behaviors that probably can’t be “programmed” genetically. The down side is that a lot of learning needs to happen, and it has to happen in the right way, or you get a dysfunctional individual.
The challenge of behavioral biologists interested in humans is to ascertain how the costs and benefits of a large brain interact with the environment of adaptation in which we see this large brain emerge over evolutionary time.
How many species of early hominid existed at any one point in time?
Sahelanthropus tchadensis, Orrorin tugenensis, Ardipithecus ramidus, Australopithecus anamensis, Australopithecus afarensis, Kenyanthropus platyops, Australopithecus africanus, Australopithecus garhi, Australopithecus aethiopicus, Australopithecus robustus, Australopithecus boisei, Homo habilis, and Homo georgicus is a short list of species that lived from about 6.5 or so million years ago to about 1.8 million years ago. Many of them lived during the “golden age of the hominid” between about 4 and 2 million years ago. There are others not listed here that are being or have recently been proposed.
With some clades of primates, we know that the species differences are seen entirely or almost entirely in the non-skeletal parts. This suggests that this number of species is a severe underestimate of the actual numbers. We also know that many primates have moderate size or small ranges, and this list comes from only a handful of localities across the region in which hominids lived, so there must be more species living in areas not sampled by the fossil sites. And, the rate of discovery of new species has not slowed down, or at least, it is reasonable to say that we can expect that the more that we look, the more we find, for the foreseeable future.
So the prospect that several, perhaps quite a large number, of hominid species would have existed at one point in time across most of Africa and possibly west Asia, is a near certainty. You would need a guidebook to figure out what you were looking at if you went back in a time machine to this region.
How many of them interacted and what was this interaction like?
Another common feature of primates is that many social primates do get into inter-species groups. There are various advantages to this. As long as food competition is minimal, mixed species groups may form to decrease predator pressure, for instance.
My personal feeling is that there was likely a fair amount of interaction between different hominid species. One reason for this is that not long after the first appearance of chipped stone tools, we see them in the record over a fairly large area. My guess (and it is only a guess at this point) is that stone tool use is more widespread at, say, 2.2 million years ago than it could have been if it was only practiced by one species. Interaction among species would facilitate the spread of this technology. That is only a guess, of course, and there are a lot of other ways to look at the data.
What caused those that disappeared to do so?
Perhaps there was a mass extinction related to increaced drying of the environment between 2 and 1.5 million years ago. Perhaps the rise of Homo erectus led to all of the others being outcompeted (or eaten for food!?) I find it interesting that relatively little is written about this. This may be in part because the time period we need to understand has fewer well dated deposits with fossil than we would like to address such questions.