The one that most closely mimics "reality".
Now all we have to do is figure out what said "reality" is.
If we consider "reality" to be a Newtonian Universe with some Geedeedubian discoveries and technology enhancements, then I think the TNE, FFS, and BL is likely the most realistic system we have.
It is founded on the basic "throw things at stuff and see if it pokes through" combat system, that scales from a side arm to auto cannons to lasers with incredible range. Add in some radiation damage and "armor, what armor" meson tech, and the rest just grows from there. Obviously it's imperfect, and we've learned things in the past 20 years, and it's not even a unique system in the Traveller Universe, or gaming in general.
But the principles are sound, and it's the most developed we have when you consider the work from FFS to TNE basic to BL with its detailed movement and record system. What it lacks is the operational characteristics. We haven't really seen that formalized anywhere (I believe) since TCS and FFW. Besides operations is more a field of budgets and scheduling than physics. But for putting bullets and blasters downrange to targets, TNE et al is pretty solid. Mind, none of this addresses any playability issues folks may have with the ruleset.
I think TNE handles damage well, I like its high penetration, low damage laser model. The biggest gap in TNE is the lack of handling for collisions at speed. Ships move REALLY fast, and the closing velocities of potential collision objects are potentially REALLY, REALLY fast, making impacts very energetic.
But TNE punted on that problem cleanly by making missiles bomb pumped lasers rather than exploding balls of shrapnel. Now, "close" and "impact" are measured in 1000's of kilometers, so possible impact can be considered "unlikely". Otherwise we'd have a flurry of guided kinetic kill weapons and we could just all go home and be done with it.
So, this aspect is missing, but I don't think it's actually missed.
TNE has a usable sensor model. We can debate nits on ranges and what not, but that really falls in the "we already know what you are, now we're debating price" column. Make sensors too powerful and we may as well not have a model at all (everyone sees everyone at all reasonable weapon ranges, so why bother). Make it too weak, and it's sub hunt in the fog. "Sulu, Z minus 10000 meters!"
As for what would a space battle look like?
Simply, the larger force wins. Why does the larger force win? Same reason the larger force always wins -- with a one to one trade, the larger force has units on the field at the end, and the smaller force doesn't. As the larger force atrits the smaller force, the larger force proportionally gets that much larger while the smaller force gets that much smaller.
In space, there's no advantage to defense. You can't build fortifications, you can't hide behind trees, and you can not stop the assault. There is no terrain to slow down the attacker, no way to way to force their positioning (i.e. defend the pass). The faster ships fleet will control the range to their advantage, so in this way a smaller fast fleet may have some advantages of a slower larger fleet, but the problem remains that if your target is in your danger space, then you are in theirs. If you can hit them, they can hit you. Save under specific circumstances, notably constant acceleration that prevents facing changes and perhaps white outs that prevent detection and lock, there are no blind spots. And in the case of white outs, larger fleets have the advantage as they can share locks. If one ship detects, all of the ships do. White outs are concealment, not cover.
3 dimensions offer no real advantage. 3D helps in a gravity well, high ground has advantages. Gravity wells help when velocity is expensive, but in Traveller, that's not really the case. The high G drives make velocity rather cheap. So, most battles I think would be outside of major influence of a gravity well. Why would a defender want to have to crawl out of one to meet their attacker if they don't have to? That would be a surprise issue, and I think the 100D limit is far enough to make surprise a non issue. In TNE Jupiter is about 5 hexes in diameter (TNE uses 30Kkm hexes), which puts 100D at 500 hexes. That's a long time, depending on the approach vector (which in theory can be anything since, in theory, vectors are maintained in Jump -- a ship can exit jump with a 500 hex velocity, giving effectively no warning, but also no time on target either).
High Guard fueling scenarios pop in, but truth be told, the fueling fleet may well be able to put off fueling to engage the attackers, since most of the fuel may well be for jump, rather than maneuver.
The perfect tactical defense situation is a running battle, where the defenders are faster than the attackers, the defenders are behind the attackers, and the defenders are slowly catching up while the attackers continue to accelerate and thus not be able to change facing to put their spinals in arc. This lets the defenders continually fire and attrit the attacking force with impunity. Note this when you're stuck in your far trader trying to out run a 3G or 4G Corsair.
I just don't ever see that happening. Barring some time pressure, the attacker simply will engage the defender, overwhelm said defender (larger force wins), and move on their merry way, licking their wounds.
In the end, this is why I think B5 is the way that it is. Operationally, the battles are tactically uninteresting. As I've said before, fleet composition determines the victor before the game even begins. The dice determines the cost. Close fleets will start close, but once one side gets a lucky shot and starts to gain advantage, they will in the end mostly retain and expand upon that advantage until victory.
Space battles offer none of the really rich tactical options of maneuver, positioning and tactical timing that other battle spaces do.
This is why I think fleet battles are really more an operational game, getting your overwhelming forces in the right place at the right time and dealing with all of the time lags of jump. It's more "Risk" than "Car Wars".