If "Canon is as canon does" is a requirement, then the ever popular BR v BB debate will continue, since the Imperium (apparently, I'll let the canonistas argue those points) deployed and promoted both doctrines. I guess the new rule set will need to come up with mechanics and reasons why both are viable from a design point of view, and then why one is chosen over another from a doctrine/logistics point of view.
I believe the "Critical" nature of the combat system must be retained. Effectively ship design centers upon the basic premise of "lasting long enough to get a lucky", and larger ships "get lucky" more often against smaller ships than vice-a-versa.
At the same time, you don't want a free trader and a pirate ship exchanging blows and having one of them spontaneously explode after a single round of combat. At the small ship level it needs to be more about scraping weapons off of each other and reducing drive ratings etc than out and out killing blows.
This would likely be managed by the basic fact that the smaller ships can't carry anything "strong enough" to do that kind of instant damage to each other.
But the larger ships will certainly need that kind of firepower. The bay weapons and spinals being the most likely sorts to enable this capability.
I believe the TNE system is actually pretty close to this, the biggest failing of the stock system is that the missiles don't work well against larger ships, as missiles are basically a bunch of close range lasers, and lasers don't have the capability to crit large ships. In TNE lasers are "death by a thousand cuts". Dangerous to small ships, not so much to larger ships. If you can't crit it, you can't kill it -- at least not quickly.
Mind, save for Battle Rider, TNE is pretty much a "small ship" game, since there was nothing published in terms of large ships. Even the Regency Source Book's ships enforcing the quarantine are were, what, 700-1000 tons? No large 10-20K ton cruiser or riders or BBs documented by the time it all ended. FF&S could make them, of course.
There needs to be crossover between ships weapons and ground vehicles. While T-Factor spinals may not be shooting at TL-15 Intrepid grav tanks, it would be good to know how they work from in the Ortillery role. And, again, while we may not be shooting spinals at tanks, I can easily see wanting to beset a laser turret on one while trying figure out what will happen when the tanks main fusion gun starts trying to punch holes in to the side of my ship. Can a RAM grenade breach a Free Trader hull or airlock hatch?? Things like that.
I don't have a real issue with the abstract combat systems. Folks talk about movement, 2D, 3D, etc. But maneuver offers so little value in current Traveller combat. Maneuver only makes sense when their's something to maneuver around, that is, it only matters when terrain is involved.
In deep space, there is no terrain, save for planets, maybe asteroids. Fighting in asteroids is, despite the excellent piloting of Han Solo, either insane due to density, or irrelevant, again due to density. If the rocks are dense enough to get in the way, they're likely dense enough to destroy the ships flying among them. And if they're not that dense, well, they don't matter anyway.
Range matters, ship facing ALMOST matters (it doesn't matter to meson guns, for example). TNE used whiteout from missile explosions as, effectively, "chaff pods" affecting detection. Blobs of energy blocking "line of site" for detection DMs. But ships aren't stealthy enough to really take advantage of these whiteouts. They certainly help, but more like smoke for a squad of infantry. It's difficult for a ship to duck behind such a whiteout and then "vanish" from the EM spectrum and break lock. Fire a missile, blow it up for noise and then go dark so as to be mostly undetectable when the noise clears. And against multiple ships, its utterly worthless. Ship A loses lock, Ship B has lock, gives lock to Ship A -- Fire when ready griddly.
The rules simply don't work very well for that, but it could be an interesting mechanic.
Maneuver can possibly be used with these mechanics, but as realistic as vector movement is, it's really not very agile. It's hard to turn, hard to stop, etc. It's awkward enough that even if there was a planet in play, you're either behind it or you're not, and you're certainly not going to "pop up", take a shot, and then "pop down", ducking and dodging behind it.
Space combat is a game of riflemen in a bull ring.
This ties in to the other issue: the likely pre-determination of the result, and how little there is the players can do about it.
HG's abstract system is mostly pre-destined. When the fleets show up, the result is pretty much pre-determined. We as players may not immediately see who is going to win, but at the same time, we, as players, if played reasonably, can't do much to affect the outcome. The fleet organization and composition decided that for us up front. One fleet has enough of an advantage early on with which to prosecute the battle, or the fleets are even enough that the decisions are up to the dice alone.
It's kind of like the card game "War". That game is decided when the deck is shuffled. We as players don't know the result until we play it out, but as participants, we can only really watch it unfold without really affecting the outcome. An HG fleet battle is similar.
Even the more detailed maneuver based systems are well determined just by showing up, since the systems of play aren't really deep enough to give the players options in how to conduct the battle. In range? Blast away. Out of range? Decide whether to close or run.
TNE mixed that up a little with limited fuel, and G-turns, etc. But, again, you're either in range or you're not.
I have no idea what can be done about that, I think it's simply the nature of "realistic" space combat. Consumables can be important, as they effect decision making. "Use it now, or use it later but possibly lose it completely". Limitation affect choice. But energy weapons don't have that problem, pretty much only missiles and sand caster have "ammo". And Sand Casters are basically ablative armor, giving your ship time to run away. But if the attacker is faster than you, you're basically a dead duck unless you have somewhere to run to or cavalry coming. Eventually he can laser your ship out of sand casters. At that point it's almost a race against the dice.
I think folks would enjoy a larger strategic system, ala TCS or Pocket Empires. Again, HG is perfect for this because the game is a logistics challenge to get the fleets in the right places at the right times, vs the details of ship v ship action. When the Big Fleet shows up against the Small Fleet, Big Fleet wins, takes some damage, roll to see if the admiral was killed.
Anyway, those are my thoughts of overall issues with ship combat in Traveller.