There has to be a reason for folks to not only go someplace, but for them to stay.
The US is littered with dead, and even today, dying towns. Simply waiting for the elder residents to pass away. The West is famous for its boom and bust ghost towns from the late 19th and early 20th century, and the Rail Town that basically followed the railroads as they pushed east and west. There are towns in the midwest dying today as the farming industry changes and moves away from the communities.
As soon as the local resource dried up, the towns vanished.
We in the US are blessed by an efficient transportation system that lets small amounts of people head out in to the wilds, "far away" from civilization, yet not be cut off from it. Yea, it may be 50 miles to nearest grocery store, but it's a "cheap drive" to get there and back. As energy prices rise, some folks may well be rethinking their remote settlements unless they have some modicum amount of self sufficiency.
Here in the West, another primary driver is simply water. But even that is not enough necessarily for folks to stay.
The oil workers may be average Joes, and they may have extended duty, but they do not Live there. They live at home, someplace else. They're paid for their hardship of extended deployments. They don't bring their families with them, and save for a curious boy, their families don't want to be their with them. The environs are very utilitarian.
But once a well head goes dry, boom, they're gone. They leave. Off to the next opportunity.
The remote, desolate areas of the planet have very small population densities, particularly by todays metropolitan standards, and in fact these remote populations are shrinking as many young people see the outside world, move away, and never look back. These populations will slowly die out, as the only thing keeping them there is culture and history, all of which changes with each generation.
It's easier to assemble widgets in a factory to provide for your family than walk out on the pack ice, rifle and spear in hand.
Circumstance no longer traps these people. It's no longer a major trip to better climes and cheaper resources.
To quote the ever lovable and sensitive Sam Kinison:
See this? This is SAND! You know what it's going to be 10 years from now? IT'S GOING TO BE SAND! You live in a f------ desert! Nothing grows here! Nothing's going to grow here! We have deserts in America, we just don't live in them, A------!
And he's right -- to a point. The local high desert is exploding as a bedroom community, driven by high housing prices in the valleys. Las Vegas is booming as well by being a ginormous magnet for capital, but actually having few resources. There are economic incentives for living in Las Vegas, and many folks in Southern California when they look at their options for a home being in the California desert versus the Nevada desert may well look at the housing prices and other economic drivers and say "Well, if I'm going to live in the desert, I may well not pay the taxes as well" and head of to Vegas.
California is a nice place to live because it's a nice place to live. Our deserts are mostly very loosly populated with a few small communities surrounding water spots, just like every other desert on the planet.
If you look at the layout of the major population centers, particularly historically across the northern hemisphere, you'll note that the cities are basically in places where it's comfortable to live. As my Dad says, why should I pay to heat or cool the outside when I can live someplace that's properly warm or cold. Witness the "Snow bunnies" that migrate from the NE to Florida every year.
So, there needs to be some driver of the economy for folks to stay. If an asteroid is tapped out of minerals, only external trade will keep it alive (whether through exploiting other asteroids, or whatever). The service industries are almost ALL locally based, and reliant on the local industrial base for capital. While some of you may drive out of your local area to see a dentist or doctor, most folks won't. The value doesn't necessarily outweigh the costs. And an asteroid better have a damn good doctor/lawyer/etc to make a very expensive spaceflight worthwhile.
So, yes, some folks will live in inhospitable environs. The pioneers will go and set up shop, and the services will follow. But once the capital dries up, the pioneers will move on, taking their families with them. And once they go, the service sector collapses readily behind, leaving nothing but an abandoned, lifeless rock with a locked starport.