In extreme cases, prosopagnostics are unable to recognize family members, and even their own face.
Prosopagnosia (commonly known as face blindness), often occurs as a result of damage to a region of the brain called the fusiform gyrus, located near the inferior (lower) surface of the temporal lobe at the midline. The damage may be due to head injury, stroke, or various neurodegenerative diseases.
However, it still remains unclear whether or not prosopagnosia is a subtype of a more general inability to recognize objects, and whether or not the brain contains a specialized facial recognition system.
Until recently, it was believed that prosopagnosia occurred only as a result of brain damage, but it is now known that there are congenital forms of the condition. Choisser suffers from congenital face blindness. In the introduction to his book, he writes:
I was born with a condition that makes it difficult for me to recognize faces. There is a small part of the brain that is dedicated to that job, and though it is small, when it comes to recognizing faces, it is very very good. In me, that part doesn't work, making me blind to all but the most familiar of faces. To help you understand this, let me compare it to two conditions you are probably more familiar with.
People who are "tone deaf" are not deaf to tones. They can hear tones, they just can't tell them apart. People who are "color blind" can see things that are in color. They just can't tell colors apart. Similarly, I can see faces. I just can't tell them apart.
(Via Science of Insanity)
My youngest brother has congenital prosopagnosia, but they still think it was from brain damage, possibly caused by hypoxia when the umbilical cord wrapped around his neck during labor.
He has no problems recognizing other objects but he has visual processing issues that interfered with his ability to read until they were corrected (at which point he went from being illiterate to reading significantly above his grade level in a matter of weeks) as well as some mild aphasia.