Comparison of Human and Neanderthal Skulls - image source
While early brain development in the two species is similar, major differences occur later and point to significant divergence in brain function.
Neanderthals and modern humans, who may have co-existed as recently as 30,000 years ago and likely interbred to a minor degree, are separated from one another by only a few genetic differences. Their brains are comparably sized, with Neanderthal brains being slightly larger on average. In a paper published in the November 9, 2010 issue of the journal Current Biology, researchers have detailed the findings of a systematic comparison between the brains of Neanderthals (studied by making virtual models from fossil braincases) and those of modern humans. It is hoped that by analyzing the differences between human brains and those of their close cousins, the reasons for the unprecedented evolutionary changes that took place in the human lineage can be made clearer.
Elongated Skulls and Rounded Skulls
The skulls of Neanderthals and modern humans, while similar in size, differ sharply in shape. As infants, both species' skulls are elongated, probably in order to facilitate passage through the birth canal, which is similarly shaped in both species. In fact, the skulls of all hominin species, as well as those of chimpanzees, are likewise elongated, making the oddly shaped skulls of humans a later development and something of an anomaly.
Shortly after birth, the skulls of human infants begin a transformation from the ancestral elongated shape, becoming much more rounded, like a globe. This divergence of skull shape and the associated differences in brain development may be a partial clue to unlocking the mystery of humans' so-called "runaway" brain growth. The differences may also say a great deal about Neanderthals' behavior and mental capabilities.
Neanderthals and Higher Brain Function
Researchers used the fossilized skulls of eleven Neanderthal individuals (including one infant) to create visual imprints of their brains for comparison to modern human brains. While it is not entirely certain what difference the shape of the brains makes, it is likely that the large divergence that occurs in human brains in the first year of life has a significant impact on the mental functioning of the species.
Neanderthals and modern humans are genetically very similar, but the nature of the brain divergence points to a comparable divergence in function. Researchers speculate that Neanderthals, while highly intelligent, were probably not capable of the sophisticated communicative skills that humans possess, and may have displayed less of humans' social and emotional complexity. This tentative conclusion was reached on the basis of data not only on the differences in brain shape, but also in small genetic differences between the two species that may have affected the pattern and speed of brain development.
Source
Choi, Charles Q. "Brain Scans Reveal Difference Between Neanderthals and Us." LiveScience. https://www.livescience.com/11078-brain-scans-reveal-difference-neanderthals.html.
Gunz, Philipp et al. "Brain Development after Birth Differs between Neanderthals and Modern Humans." Current Biology. http://www.cell.com/current-biology/abstract/S0960-9822%2810%2901282-0.