I'll be the first to admit that sysgen creates too many "unicorns" which require torturous "explanations".
A thought about the Main World sysgen:
It was designed to work within certain parameters and does it quite well. And when I say this I am, of course, looking at the original rules and text found in 1977 -- before there was an OTU and players for two years were left to their own devices to make up whatever they wanted.
First, it is a "prod to the imagination" to help the Referee create worlds full of exotic location and edge-case logic like those found in the SF stories from the 50s to the mid-70s. It is not built to build a selection of worlds that are uniform or logical in the sense that many world-builders (like Hans) might enjoy, or even "Hard Science Fiction" as the term is used today.
It should be noted that the rules make clear that if the Referee knows what he wants a world to be like he should just make that world and not used a system of random generation.
Second, location: The implied kinds of worlds we find in Books 2 and 3 are world that are geographically and culturally isolated from each other. And as Book 4 sums up, "
Traveller assume a remote, centralized government..." The worlds, then, do not work with the assumption of any sort of centralizing or cosmopolitan influence.
Each world is, then, "It's own thing" and designed to be such. with the PCs and other travellers being the extraordinary folks who travel the stars and bring word and news of these other worlds or influencing politics between worlds (as spies, mercenaries, whatever). There is no assumed standardization across the worlds as to the way things work. A naval or scout base might belong to the world it orbits, belong to a nearby world, or belong to the remote government. Only when the Referee decides the justification for the base's presence is this matter settled.
Whether these "remote" worlds are on a frontier, or have grown fallow, or are being developed is up to the Referee. The key, though, is that the system is no generating worlds that would make sense at the hub of an imperial government or most locals where a centralized government has strong influence.
Third, the number of unicorns is very large because we often use it outside the scope of its design. The sysgen was built to create only 40, or perhaps 80 worlds.
From Book 3, 1977 edition:
"Initially, one or two sub-sectors should be quite enough for years of adventure (each sub-sector has, on the average, 40 worlds)..."
I know that if I were in the position of having to justify all the unicorns in a game preserve of 640 worlds (a sector), let alone all of the The Third Imperium I'd probably go nuts with frustration about having to justify all the unicorns as well. But the rules as written and designed never assumed this. There are settled worlds beyond this subsector of play -- but there is no need to roll them up or define them or justify them.
Because the number of worlds is very specific, we don't have
countless weird cases. We have the really unique and interesting cases in a specific patch of space that the Referee has built. Because the setting of play is limited in scope the Referee can find interesting ways of making all the pieces interlock and be interesting with each other. The logic is built
for this particular setting of the subsector and does not have to make sense across all subsectors are all iterations of all
Traveller settings.
None of this is to say that Referees should trap their
Traveller settings or logic in the assumptions of the Boo 3 Main World sysgen rules. My only point is that over time the needs of the OTU -- and the logic of players that wanted the OTU to make sense as presented in the fluff -- the rules got stretched to the point of breaking.
This, however, is not a fault of the system but of expectations that don't want or don't need the kinds of setting material the system was designed to produce.
In my view, however, the toolkit aspect of the rules allows a Referee to re-jigger the Main World sysgen system tables easily enough. If he wants a prod to the imagination, but a different range of outputs, it would be easy enough to create the tables to do this.