Salting your SEC files

[robject]

These days I tend to salt my SEC files with just enough XML to section the data. This gives my code some directives for more intelligent parsing (key-value pairs and UWP lines, both solved problems) without throwing out the whole purpose of SEC files (human readability and edit-ability).

As I've said before, SEC files are poor databases. Their best use is for humans to edit sector data. Thus a non-intrusive salting of XML makes them easy for machines to create and parse without shackling human editors. Frankly, a light salting is all that's needed to simply import and export sector data from real databases, anyhow.

Document
The top document element is <sec>. It has no real data requirements, but I often put "permanent" or "machine relevant" stuff in here anyway, just for good measure.

<sec name="Core" 
     milieu="Second Survey" 
     desc="AOTI recapture" 
     gen="Wed May 10 14:41:53 2006">

Header
Next comes the header. I just wrap whatever data the SEC file keeps here with XML tags and move on. Okay, sometimes I'll format the content into a key: value list, but typically this is done for us anyway.

<header>
Sector: Core Sector
Alt:    Sector fG
DOS:    core
Coord:  fG
Ref:    9960 Ring/Ray 62832
</header>

Subsectors

Again, I keep it simple, simply wrapping the existing data.

<subsectors>
A: Apge
B: Perite
C: Ameros
... etc ...
N: Ch'naar
O: Dunea
P: Saregon
</subsectors>

References

Sometimes there is reference data, with the names of articles which talk about worlds in this sector. I put that in a <notes> section.

<notes>
ref 0125 Trevor (SM)  "Traveller News Service" 
ref 0140 Reference (TD)  "Reference, Center of the Imperium" 
ref 0523 Dinhe (SM)  "Traveller News Service" 
... etc ...
</notes>

UWP Data

Finally, just a simple wrapper around the UWP lines and I'm done.

<data>
New Ramma     0108 D465540-6  S Ag Ni              703 Im K6 V M6 V G6 V
Saarinen      0113 B566212-8    Lo Ni Rs           513 Im K6 V M5 D
Rena          0123 D596211-7  S Lo Ni              403 Im K4 V
Gulistan      0124 CAB2358-7  S Lo Ni Fl           820 Im K6 II
... etc ...
</data>

</sec>

Yeah, don't forget the final </sec>.

And that's all your SEC file needs. Let XML do the heavy lifting of breaking the file up into sections. The rest is a solved problem.

 

[Hemdian]

Like

(Damn, must be spending too much time on Facebook.)

 

[robject]

LOL! (How's that for a tweenaged Facebooky response?)

At the end of the day, it doesn't matter, of course. SEC files are SEC files, programs are few and far between, as are programmers. We do what we have to do. Selah.

 

[robject]

Xboat data

For xboat data, I used to put a colon at the end of a UWP line, followed by a comma-separated list of destination hexes. It was a horrible solution, but it worked.

Nowadays, I prefer to put jump route segments in their own section. I like supporting multiple stops per segment, because it reduces the line count of the file, and it implicitly supports a simpler one-segment per line format as well. The downside to multiple-stop entries is that legs in a route tend to get buried. So maybe pairs are better after all.

I also like putting in world names, if they're unique. Very easy to find that way.

<routes>
red: Regina 2007 2005 1904 Pixie 2202  # comments are OK after the pound sign
red: Boughene Efate 1307 1106 Ruby
red: Emerald Jewell Mongo
</routes>

 

[simonh]

Interesting stuff, I'd not considered embedding XML in a SEC file, bit it makes a lot of sense. I'm not generally a fan of using XML for file formats intended to be human readable or editable (or at all of they can be avoided) but this minimalist approach using it to add annotations is nice.

I have experimental SEC file importing working in StarBase (not yet in the beta release version available for download), because I can imagine users having SEC format data they want to import and then develop further in the app. The main problem with SEC format export is that StarBase can contain a lot of data that wouldn't fit into the SEC file format. Route data is one such. Your approach goes some way to helping with that, if there are tools that will accept and use that data.

I must admit I don't fully understand the route data format though. Is the intention to allow specifying nodes on the route either by hex or by world name interchangeably? If so, since there are no delimiters how do you handle world names with a space in them? I suppose the software would have to detect that and use a hex code instead. Can you think of any way to handle routes that connect to worlds in other sectors?

Simon Hibbs

 

[Nathan Brazil]

Sorry

 

[robject]

If so, since there are no delimiters how do you handle world names with a space in them? I suppose the software would have to detect that and use a hex code instead. Can you think of any way to handle routes that connect to worlds in other sectors?

Simon Hibbs

World names with a space in them would pose a problem. I think the best solution, aside from using the world's hex number, is to insert an underscore between the words, e.g. Marnie's_World, then implicitly a program would have to know to decode that back into separate words.

I handle routes leading to other sectors by referencing the hex number relative to the current sector. So for example Mora has a line leading into Deneb Sector; let's suppose that's Deneb 0134 (I don't know the real number). In the Marches SEC file I would show that link as 3334.

Crossing in the other direction, we can go negative. If the link is just across the border there's no problem using a hex location such as 0001. Going negative looks messy (e.g. -03-01) but it works okay, and usually only one number goes negative.

 

[simonh]

I think a delimiter would work better generally for world names. Of course it does have its own problems, you have to choose the delimiter carefully.

I am considering adding 1000 to all hex coordinates in StarBase, so the grid centre is notionally 1000 hexes 'up' and 'left' of the map area. That would allow for easy representation of locations outside the map area, but for SEC files there is no easy solution.

Frankly the SEC format is not great, but for Traveller data sets its what we have. However for extensions like trade and other links, we don't need to be bound by the constraints of the SEC file conventions too much.

Simon Hibbs

 

[Nathan Brazil]

Back at the turn of the millenium, I worked for a company that generated cell phone bills for the then major carriers and their subscribers. So were were processing over 2 billion call records a month. Anyways, we had developed a standard record layout where we took the official call records from the various antenna's and land line switches and integrated them in to one record each billable segment of the call. So besides keeping the original call segment on our homegrown record layout, we added portions to the layout for our needs. Then we created a new billing system and had to keep compatability with legacy and external systems. So I suggest a layout which keeps the old .SEC layout but extend it for compatability. One suggestion.

1-25: Name
27-30: HexNbr
32-40: UWP
42: Bases
44-68: Remarks & Comments (Characters 1-25)
70: Zone
72-74: PBG
76-77: Allegiance
79-98: Stellar Data (Characters 1-20)
101-125 Remarks & Comments (Characters 26-50)
126-155 Stellar Data (Characters 21-50)
This way old programs do not have to change their input system unless they looking for data beyond column 98
New programs read in the two data fields and concatenate them together into one field within their program.

 

[Nathan Brazil]

Sometimes formatting data looks crappy because of cutoff in mid comment so an alternative:
1-25: Name
27-30: HexNbr
32-40: UWP
42: Bases
44-68: Remarks & Comments (Original)
70: Zone
72-74: PBG
76-77: Allegiance
79-98: Stellar Data (Original)
101-150 Remarks & Comments (50 character format)
151-200 Stellar Data (50 Character format)

 

[Nathan Brazil]

Sometimes formatting data looks crappy because of cutoff in mid comment so an alternative:
1-25: Name (Original)
27-30: HexNbr
32-40: UWP
42: Bases
44-68: Remarks & Comments (Original)
70: Zone
72-74: PBG
76-77: Allegiance
79-98: Stellar Data (Original)
101-150 Name (50 character format)
151-200 Remarks & Comments (50 character format)
201-250 Stellar Data (50 Character format)

 

[simonh]

Eventually you get to the point where maintaining compatibility with the old format is just too limiting. Yo're better off moving to a new more modern and flexible format, and just using a separate importer or exporter for the legacy format.

One big reason for moving to a modern format is the ability to use widely available libraries for data format handling. If you implement a new format in XML or JSON for example, you immediately gain access to robust, performant and flexible libraries on multiple platforms.

The problem is standardisation. For better or worse SEC files are what we have, but as soon as you start extending the format they you need to standardise your enhancements otherwise they become just as useless as a new format alongside a SEC file compatibility module would be.

Simon Hibbs