The main players, their measurements, and their mistakes

The famous Dutch astronomer, Jacobus Kapteyn and his collaborators, started a major project of counting stars around the 1900s, to determine once and for all the structure of the MW. He counted stars on photographic plates as function of apparent magnitude (brightness), in several directions.

His analysis basically confirmed Herschel's picture: the MW is a flattened elliptical system, with stellar density $n$ decreasing away from the centre. $n$ drops to half its central value at 150 perpendicular to the MW plane, and 800 in the galactic plane. The Sun is at 650 from the centre. This is in fact wrong but for an interesting reason.

Kapteyn assumed that the flux received drops as $1/r^2$. One reason this could be wrong is because of absorption or scattering of the stellar light, in which case the flux will drop faster than $1/r^2$. Of course, not taking absorption into account would give you the wrong answer, and Kapteyn was fully aware of this. How can we test whether absorption is important? For Rayleigh scattering off atoms, there is a colour dependence: blue light is scattered more than red light. As a consequence fainter and hence more distant stars would be expected to be redder on average. Now Kapteyn found this indeed to be the case, yet the amount of reddening was too small for absorption to be important. The mistake in the reasoning was that the absorption is due to dust, not Rayleigh scattering. Dust absorption is much less colour dependent, and so the small reddening measured by Kapteyn actually implied a much greater dimming of the distant stars. Consequently the size of the MW as well as the position of the Sun in it came out wrong.

The American astronomer Harlow Shapley estimated the distances to the brightest globular clusters using their RR Lyrae variables, and found they were not centred around the Sun, but around a point 15kpc away in the direction of Sagittarius. This was in conflict with the Kapteyn universe - it seemed difficult to reconcile the Sun to be near the centre of the MW, yet the centre of the globular cluster system not to coincide with the centre of MW? There was no clear way how to reconcile these different data, and so the issue remained unresolved for some time.

Meanwhile, another famous Dutch astronomer, van Maanen, found that some of the nebulae had proper motions, implying they were nearby, within the MW. (He compared the positions of the nebulae on plates taken several years apart). It was generally accepted these nebulae were probably proto-planetary systems, or something like that. van Maanen himself, but also Edwin Hubble later redid the proper motion measurements with better data (using a better telescope), and both failed to reproduce the proper motions. van Maanen probably just made a mistake! In the 1930s, Hubble made his even more famous discovery that the spectral lines of atoms in other galaxies are shifted to the red, by an amount proportional to the distance to the galaxy.

So it was a combination of mistakes (van Maanen), and not taking into account the importance of dust (Kapteyn), that led to the confusion. In the end, Hubble's better data clinched the matter. He resolved Cepheids in Andromeda, one of the bigger nebulae, and found a distance of 300kpc and so clearly outside of the MW. In one great swoop, the universe because much bigger, and in addition it was found to be expanding! It is difficult to underestimate the significance of this discovery.

The 1920s were of course a period of great economic depression. But from the scientific point of view, they must have been tremendously exciting. In only a couple of years, the general theory of relativity was formulated, quantum mechanics revolutionised physics, the MW was recognised to be just one of millions of other galaxies, the Universe was found to be big and expanding

It was not until the 1930s that Trumpler found what had been wrong with Kapteyn's results. He studied Open Clusters in the MW plane. He assumed them to be all of about the same size, and so got an independent relative distance for them. He used another estimate of their distance, based on `spectroscopic parallaxes'. This involves predicting the luminosity of a star from its spectrum. He showed that the more distant clusters (as judged from their angular size), appeared to have unusually faint stars in them. Or equivalently, that the more distant clusters (as judged from their photometric parallax) were larger. Clearly, something was wrong here. He correctly interpreted this, as due to the fact that the light from these more distant stars was strongly attenuated. By comparing the reddening with the attenuation, it was clear that a given amount of reddening corresponded to much more dimming than you'd expect for Rayleigh scattering. And hence Kapteyn's scales were wrong.

Before I summarise this in a time-line, just think of the fact that Einstein formulated his theory of general relativity, on which the current cosmological models are still based, before this puzzle was resolved. In fact, our view of the Universe around Einstein's time, was just plain wrong. I guess he rightly paid little attention to what the astronomers of his days told him!