Ship combat in and of itself doesn't seem that engaging, unless it ties in with the experiences of the people in those ships. Otherwise, it's just two tin cans shooting at each other in space.
You're exactly right.
As both a reader, writer, and gamer, it's
hard to do ship combat in a way that is engaging. I think a lot of it is that space combat is boring and in the 40-odd years since Traveller has been made, nobody has ever tried to address it.
You can anthropomorphize the ships in question (and some authors have with varying degrees of success). However, I've always found it a bit dissatisfying: In games like Traveller, space combat occurs at ranges outside of visual range, so there's little to describe about the enemy ship's condition - it's where I find the parallels between, say, something about combat in the Age of Sail break down.
In addition, there's a distinct lack of that other crutch that writers use to make sea combat interesting: There's no "terrain" in space - it's like a duel in an empty parking lot with computer-aimed guns. You can't have one starship suddenly flying out from a bank of clouds in a "nebula" or a gas giant atmosphere or out from behind an asteroid (unless you're playing in that kind of sci-fi).
There's a story in a (really old) Dragon magazine that I read as a child (did I just date myself, I guess so): "Passing In The Night" by Rob Chilson in Dragon #102 (it's available for free online, but I am not 100% sure of the legality of the pdf, so I won't be posting the direct link).
This story has really stuck me as there's no sequences of dogfighting, you never see the enemy except as a sensor signature; nobody looks out the porthole and sees the enemy ship. The entire sequence is this deadly waiting game of sending missiles off and waiting for hours while the missiles reach the enemy (or don't), meanwhile they've shot their missiles too and you watch them close in and you see if your anti-missile defenses work (or don't).
While Traveller combat is different post Nuclear Damper, I think the emphasis in that short story is similar to that of Traveller space combat, especially if you want to write about it - lots of emphasis on the anxiety and waiting. I think due to Traveller's wargaming origins, there's a certain fatalism to the space combat system, especially if the two combatants are not warships and lack much in the way of the countermeasures/protection.
Instead, I think you could put emphasis on how different space combat is, while the world is basically going to be in the ship for the most part. I'll share my efforts to make a space combat system that is more interesting for all the players, not just the "pilot" and the GM while everyone else as might as well go get food ("We'll be back in 30, tell us if we live or die then.")
* Everyone puts on an environment suit when a questionable contact comes up on sensors.
* The may be ship depressurized during combat. Naval veterans on the crew likely have been trained in "suiting for dummies" so the crew buttons up, and then they go through the airlock in turns so they know everyone's suit has been put on properly and works before the fecal matter hits the fan; someone's suit might be revealed to be broken so he (or she) has the unenviable position of knowing if the ship is hit and depressurizes, they have twenty seconds to reach the head that doubles as a pressure closet - easy to do normally, less easy to do in the panic of combat when the ship is possibly doing a 2G burn.
* In most editions of Traveller, there's a seconds of delay between even a laser firing and when it hits the target, slower weapons take much longer to hit their target. In a civilian ship, this could literally lead to a situation where some bridge officer has a stopwatch, the pilot has to activate the maneuver system (powered on battery backup, so "burns" are limited while engineering tries to get the reactor working again). The ship would have to have to play a game of chicken, where they maintain a predictable course to try and get the other ship to shoot their beam laser at them, then make a hard burn between the time the laser is fired and when it hits them to get out of the way with the crew knowing (for instance) the enemy ship is 6 seconds away, they figure out the enemy ship can charge to fire their beam laser once every X minutes. So the countdown is for the X minutes, then it's up to the captain to decide when the enemy ship has likely fired to order a hard burn.
* If the ship is hit, it's likely that microdebris and possibly atmosphere will form a kind of 'cloud' around the ship. Near-misses on the ship may only be detectable because this debris ionizes and glows, showing how closely they were missed (on a civilian ship, with their poor or possibly damaged sensors, crewmembers may actually be assigned to look outside the portholes to look for this - in fact, if you really want to wow the reader, the crew might intentionally vent gases out of the ship if their sensors aren't well-tuned to detect this kind of thing so they can tell when the enemy ship fired "Laser glow, port lower!" "How intense? Jon, stopwatch!" "Six!" "Um...just a second..." "Five!" "Now, bosun!" "Four!" "Looks like 20 megajoule range!" "In Vilani please, bosun." "Three!" "Five sixes Dekk and four eights Tarr!" "Two!" "Jeanne, two gee hard burn evasive!").
* If the G-compensation suddenly goes on the fritz, it's likely to happen in the middle of combat. Writing about how suddenly "down" lurches sickeningly, cups of coffee, lose equipment, much of it stuff that nobody thought would go flying suddenly going sliding/tumbling across the bridge could be an actual hazard - in addition to those crewmembers who didn't properly buckle down (because you know, in the TU people are most likely used to comfortable rides even in 2G maneuvers because of G-compensation) - crewmembers can get injured, possibly even killed by such debris.
* Fusion reactors don't explode and starships don't sink under the sea. I think there'd be a lot of ambiguity about when a ship is actually "destroyed" without any convenient moments where a freighter explodes in a dramatic fireball. How do you know if your opponent is actually disabled and unable to fight or just playing possum? In fact they might be doing both - if it takes you hours or something to approach a disabled ship, perhaps their crew (in their spacesuits) have gotten their fusion reactor back on-line in the time it takes you to close. It is likely unrealistic for a starship to simply keep pounding a hulk until it falls to pieces, so what happens when one ship or the other stops firing and maneuvering and sensors say the power plant is off-line? Do you simply just leave it and fly off? Do you send in drones to check out the hulk or do you have to close yourself and inspect (and possibly get ambushed)? Do you hail the ship and demand its surrender (and is there some sort of "spacer law" that handles such surrenders, like nobody wants to be cast adrift in some uninhabited system so perhaps there's etiquette regarding surrenders where the defeated side are interred in Low Berths while the winning is honor-bound to take them to a spaceport?).