Sector Data File Formats (.sec)

inexorabletash · Mar 28, 2009

Okay, I think I have stellar data close to ready to go. There are a handful of worlds with bogus data, but it's not too bad.

I ended up writing a recursive descent parser for the starsystem data to capture Malenfant's Stellar Generation that encodes the system structure as a superset of the classic data - several files use that.

Here's the grammar I ended up with:

Code:

// Basic: 
//   system     ::= star ( w star )*
//
// Extended: Malenfant's Stellar Generation
//
//   system     ::= unit ( w companion )*
//   companion  ::= near | far
//   near       ::= unit
//   far        ::= "[" system "]"
//   unit       ::= star | pair
//   pair       ::= "(" star w star ")"
//
//   star       ::= type tenths w* size main?
//                | dwarf
//                | unknown
//   type       ::= "O" | "B" | "A" | "F" | "G" | "K" | "M"
//   tenths     ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
//   size       ::= "D" | "Ia" | "Ib" | "II" | "III" | "IV" | "V" | "VI" | "VII"
//   dwarf      ::= "DB" | "DA" | "DF" | "DG" | "DK" | "DM" | "D"
//   unknown    ::= "Un"
//
//   main       ::= "*"
//
//   w          ::= " "

(The extended grammar includes all productions of the basic grammar, so I just parse as extended.)

Errata:

"Un" - this is present as a star code in Mendan sector data from the Challenge #49 (Mike Mikesh) for 0221. Possibly a typo, but I interpret it as "Unknown star type" i.e. there's something weird going on.

Using this grammar and scrubbing my data files, and ignoring LRX and some strange single-letter suffixes (h, f), I end up with just these errors - corrections appreciated:

Code:

Zhodane
Suffix Data Error: 'F3 V M4 V M9 D M4'
* Oythepru      2034 A66586B-C  Q Ri Mr              324 Dr F3 V M4 V M9 D M4
Suffix Data Error: 'M9 Zh G6 D'
*               2924 E333223-B    Lo Ni Po           923 Zh M9 Zh G6 D

Core
Suffix Data Error: 'K8 VI M4'
* Night           0839 A5749C9-F N Hi In                   320 Im K8 VI M4 

Ley Sector
Suffix Data Error: '2 V A6 D'
* Tender Mercy  0314 E447210-5    Lo Ni              200 IL 2 V A6 D

A small number of errors, which is encouraging, but many sector files have no stellar data so that doesn't mean much.

inexorabletash · Mar 28, 2009

Just a random note:

I haven't seen any SEC files that use the "TF" codes (Traffic and Facilities) from The Traveller Chronicle (TTC) #5 (variant/house rules by Guy Garnett). These were printed with UWPs in subsector listings in TTC after that point, like so:

Code:

name   hex  uwp       b code      z pbg tf al stellar
Foobar 0102 A123987-F S Hi Cp Xb  A 123-45 Dd G7 IV

Any fixed-width plain-text format needs to accommodate such extensions.

inexorabletash · Mar 29, 2009

FWIW, site updates live:

http://travellermap.blogspot.com/2009/03/stellar-data.html

TWTP · Mar 29, 2009

Brown Dwarfs

inexorabletash said:
Code:

// type ::= "O" | "B" | "A" | "F" | "G" | "K" | "M"

FYI: you may want to add the brown dwarf types: "L", "T" and "Y". This way for people who wish to use them, your code will still be valid and useful.

see: http://en.wikipedia.org/wiki/Brown_dwarf for more basic info

robject · Mar 29, 2009

Space Hamster said:
Fixed width fields are the old way of doing file formats. XML is not overkill. It is the logical successor to the fixed width fields. Most modern programming languages have build in support for reading and writing XML files. XML is by far easier to read than a fixed width field file as you know what values your working with because it has the information wrapping the values.

Fixed width fields are so 80’s. Get with the 2000’s and embrace variable length fields.

As an amusing side note, my modules actually output a Frankensteinian hybrid: major sections are delimited by XML tags, but the "data lines" are plain text. How's that for painful?

I do think that the main reason I haven't gone straight to YAML is that there is all this legacy data floating around. But, now we do have applications that digest the legacy data (even if we have to pre-filter some of it, as inexorabletash has done), so I suspect that soon we really will have a 3.0 standard.

In fact, I suggest that once inexorabletash shares his XML schema, that it will be the de facto standard. At that point I can even write YAML structures with comfort, because they're just a data serialization language, and the critical point isn't the language but rather the underlying structure.

Wow - the more I think about it, the more I think that's the solution.

OK Inexorabletash,, show me the structures you use, and I'll adopt them. I'll simplify and update my Perl modules in CPAN and we'll be on our way.

Hemdian · Mar 29, 2009

I'd forgotten all about it until just now but Mark Preston developed a candidate XML schema back in 2002 he called AstroML.

inexorabletash · Mar 29, 2009

robject said:
As an amusing side note, my modules actually output a Frankensteinian hybrid: major sections are delimited by XML tags, but the "data lines" are plain text. How's that for painful?

I've actually become quite enamored of that design. My site uses a hybrid approach, where the metadata - everything but the UWPs - is in nicely structured XML, and then it references the UWP as an external file for convenience (but it could be inline as well).

In fact, I suggest that once inexorabletash shares his XML schema, that it will be the de facto standard.

It's not actually generally applicable as-is. You can check it out by visiting http://www.travellermap.com/res/sectors.xml - that's effectively the metadata store behind the site. A more specific example is http://www.travellermap.com/res/Sectors/farfrontiers.xml which excises just one sector, referenced by the main file.

At that point I can even write YAML structures with comfort, because they're just a data serialization language, and the critical point isn't the language but rather the underlying structure.

Heh. In my day job, my co-workers are working on an IETF draft that tries to work a level up from the serialization format (http://tools.ietf.org/html/draft-hamrick-llsd-00) - it's actually hard to get folks to ignore the bits-on-the-wire and realize that the higher level structure is more subtle and what matters, and serialization formats are trivial by comparison.

OK Inexorabletash,, show me the structures you use, and I'll adopt them. I'll simplify and update my Perl modules in CPAN and we'll be on our way.

I don't think it's that easy, but let's discuss the roadmap.

First, I think we need to document the current standards. It should be crystal clear that no-one is trying to make those documents obsolete. Supporting 30 year old files is a Good Thing. Existing SEC files are robust and stable, and a good interchange format, let's preserve that. Marc's vision from Challenge #26 is still true today.

Second, we need to loop in other software producers (e.g. Heaven and Earth, Galactic, various libraries, and so forth). There's no point trying to design a standard that no-one will use. That said, a one-size-fits-all is also not necessarily a goal. It should be clear that any proposal is for an interchange format - it captures the common set of data in an extensible way, and isn't necessarily anyone's native format.

Third, we need to identify that common set of data. I see this extending in three dimensions from "plain old SEC":

Extending the UWP for a world with additional data fields per-system
Within a hex: encoding the full system structure (orbits, UWPs for each body, etc)
Beyond a hex: routes, borders, allegiances, subsectors, etc.

Additionally, most facets of the data needs to be taggable with metadata - sources, credits, and other references. Specifying an era for the data is also relevant.

Since I am not producing data, I have only been focused on some of these (beyond a hex and metadata), so I am not a good source for identifying the specific needs.

robject · Mar 30, 2009

inexorabletash said:
Since I am not producing data, I have only been focused on some of these (beyond a hex and metadata), so I am not a good source for identifying the specific needs.

Granted, but travellermap could be serving the data.

I've got a serious worry: how do we interpret the legacy stellar data? If there are three stars listed, what are the relative positions of the 2nd and 3rd stars? Can we tell if the 3rd is far, or if the 2nd is close, 'cuz I can't without employing some shenanigans...

SEC Parsing

The most popular format, that used by Genie/Sunbane, Galactic, Heaven & Earth, and travellermap.com, are structurally identical, and therefore are parseable by the same logic, to wit:

Code:

my $uwp = "my world name   0101  A123456-8 A             111 Im K0 V M1 VI";
# also: "my world name   0101  A123456-8              A 111 Im K0 V M1 VI";

#         World    0101     A123456-8    A         A      111    Im   F1 V...
$uwp =~ /^(.*?)\s*(\d{4})\s+(.{7}-.) \s?(.)(.*?)([AR])? (\d{3}) (..) (.*)$/;

$name       = $1;
$hex        = $2;
$uwp        = $3;
$base       = $4;
$comments   = $5;
$zone       = $6;
$pbg        = $7;
$allegiance = $8;
$stellar    = $9;

Beyond a hex: routes, borders, allegiances, subsectors, etc.

Joshua has suggested some elements of this already, which I've also used. Here's how I'm currently annotating my sector files:

Code:

<sec>

This is how I start a sector file. The only reason I'm abbreviating is because I like abbreviations.

Code:

<meta
    alt="Sector iH"
    coord="iH"
    dos="alphacru"
    milieu="Second Survey"
    loc="10080 Ring/Ray 62864"
    sector="Alpha Crucis"
   />

Because I wasn't sure where to store this metadata, I created a "meta" block for it. Other suggestions are welcome.

Joshua lists subsectors in a very thoughtful way. Whereas I had a single element with a set of fields from A to O, he creates an element for each subsector name, which allows him to include a reference -- a great idea, by the way, one which I will be adopting:

Code:

<subsectors>
   <subsector index="A" source="Travellers' Chronicles #5">Detsiaiem</subsector>
   <subsector index="B" source="Travellers' Chronicles #5">Ienji</subsector>
...etc...
</subsectors>

After that, I have a "world reference" section, in case I want to jot down where world data comes from. It's not well thought out, but it's a start.

Code:

<world_reference>
   <ref hex="2021" name="Depot"     source="RS">Imperial Navy Depots</ref>
   <ref hex="2117" name="Sabmiqys"  source="WBH">Using World Data</ref>
   <ref hex="2406" name="Githiaski" source="JTAS">Contact: The Githiaskio</ref>
   <ref hex="2421" name="Antares"   source="VV">Vilani Space: Then and Now</ref>
   <ref hex="2425" name="Ansenz"    source="SM">Traveller News Service</ref>
</world_reference>

Next comes the "data", whose content is the plaintext sec data:

Code:

<data type="application/x-net-farfuture-traveller-sec-v2">
Irkigkhan       0103 E470100-4    De Lo                   920 Im M V 
Shana Ma        0104 E324610-5    Ni                      111 Im F V 
Niizediju       0106 B850864-9 N  De Po                   924 Im F V +M V
Azimuth         0202 B797300-7 N  Lo                      720 Im M V 
...
</data>

Joshua suggested the "type" field, which here tells us that the data is SEC v2. (My code assumes that v2 is the default, so I typically leave out the type field).

Another thoughtful addition from Joshua: adding the ability to reference a sector file via URI, instead of having to embed it into this file. Here's how that might work:

Code:

<data ref="vland.sec" type="application/x-net-farfuture-traveller-sec-v2" />

The "ref" means this file should use the sec data found in the file "vland.sec", and the "type" of course means it's version 2 format.

Next would come the border lists, which are plaintext lists of hexes.

Code:

<borders ...>
   <border ...> 0101 0102 0103 0104 0105 ... </border>
</borders>

Then the route lists... I'm of two minds about this. Dale Kemper uses simple hex pairs to lay out directed routes, while I prefer re-using the hex list format used in border data. Regardless, the outer tags remain the same:

Code:

<routes ...>
   <route ...> 0305 0306 0406 0308 </route> ...
</routes>

And finally, the sector file ends with:

Code:

</sec>

forthekill · Mar 30, 2009

That's a pretty good setup for sec data in an XML format, although you use a lot of attributes, which I think need to be used sparingly, if at all (and which I personally detest).

I can see some value in having the plain text sector data in one tag like that, but I really think if you're going to bother using XML that the individual data are in separate tagged XML fields.

Regardless of what I think it's a great example of the kind of data that needs to be in a "universal" sector XML data format, and is something for us to chew on in the discussion. It also provides a good comparison to what inexorabletash/Joshua is using. And I never really thought about source material and reference tags, because I don't really use real/canon Traveller data, so it is of less interest to me, but I realize the need for those who trade in the sometimes mysterious OTU data.

o:

Is anyone else out there using XML for storing sector data?

jfetters · Mar 30, 2009

Yeah, I have a format that I created a few years ago for a program I wrote (which I need to put back on the download list). The source program enabled generating the sector in XML, then allowed you to export to HES or SEC formats via XSLT. Unfortunately, I haven't had any cycles to revisit or refine, as I wanted to transform from the text formats into my XML, but I think Joshua has that well in hand!

Anyway, here is the world XML format:

Code:

[URL="file:///C:/Users/jamesfe/Desktop/test%20Sector.xml#"][B][FONT=Courier New][COLOR=#ff0000]-[/COLOR][/FONT][/B][/URL] [COLOR=#0000ff]<[/COLOR][COLOR=#990000]sector name[/COLOR][COLOR=#0000ff]="[/COLOR][B]test[/B][COLOR=#0000ff]">[/COLOR]
[URL="file:///C:/Users/jamesfe/Desktop/test%20Sector.xml#"][B][FONT=Courier New][COLOR=#ff0000]-[/COLOR][/FONT][/B][/URL] [COLOR=#0000ff]<[/COLOR][COLOR=#990000]worlds[/COLOR][COLOR=#0000ff]>[/COLOR]
[URL="file:///C:/Users/jamesfe/Desktop/test%20Sector.xml#"][B][FONT=Courier New][COLOR=#ff0000]-[/COLOR][/FONT][/B][/URL] [COLOR=#0000ff]<[/COLOR][COLOR=#990000]world name[/COLOR][COLOR=#0000ff]="[/COLOR][B]A 0101[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] hex[/COLOR][COLOR=#0000ff]="[/COLOR][B]0101[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] planetoids[/COLOR][COLOR=#0000ff]="[/COLOR][B]2[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] gasGiants[/COLOR][COLOR=#0000ff]="[/COLOR][B]4[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] popMult[/COLOR][COLOR=#0000ff]="[/COLOR][B]9[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] allegiance[/COLOR][COLOR=#0000ff]="[/COLOR][B]Na[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] subsectorCode[/COLOR][COLOR=#0000ff]="[/COLOR][B]A[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] primaryStar[/COLOR][COLOR=#0000ff]="[/COLOR][B]F6 D[/B][COLOR=#0000ff]">[/COLOR]
 [COLOR=#0000ff]<[/COLOR][COLOR=#990000]UWP[/COLOR][COLOR=#0000ff]>[/COLOR][B]B373665-B[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]UWP[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]starport[/COLOR][COLOR=#0000ff]>[/COLOR][B]B[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]starport[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]size[/COLOR][COLOR=#0000ff]>[/COLOR][B]3[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]size[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]atmosphere[/COLOR][COLOR=#0000ff]>[/COLOR][B]7[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]atmosphere[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]hydrosphere[/COLOR][COLOR=#0000ff]>[/COLOR][B]3[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]hydrosphere[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]population[/COLOR][COLOR=#0000ff]>[/COLOR][B]6[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]population[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]government[/COLOR][COLOR=#0000ff]>[/COLOR][B]6[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]government[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]lawLevel[/COLOR][COLOR=#0000ff]>[/COLOR][B]5[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]lawLevel[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]techLevel[/COLOR][COLOR=#0000ff]>[/COLOR][B]B[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]techLevel[/COLOR][COLOR=#0000ff]>[/COLOR] 
 [URL="file:///C:/Users/jamesfe/Desktop/test%20Sector.xml#"][B][FONT=Courier New][COLOR=#ff0000]-[/COLOR][/FONT][/B][/URL] [COLOR=#0000ff]<[/COLOR][COLOR=#990000]tradeCodes hex[/COLOR][COLOR=#0000ff]="[/COLOR][B]0101[/B][COLOR=#0000ff]">[/COLOR]
 [COLOR=#0000ff]<[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR][B]Ni[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]</[/COLOR][COLOR=#990000]tradeCodes[/COLOR][COLOR=#0000ff]>[/COLOR]
  [COLOR=#0000ff]</[/COLOR][COLOR=#990000]world[/COLOR][COLOR=#0000ff]>[/COLOR]
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]world name[/COLOR][COLOR=#0000ff]="[/COLOR][B]L 3028[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] hex[/COLOR][COLOR=#0000ff]="[/COLOR][B]3028[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] planetoids[/COLOR][COLOR=#0000ff]="[/COLOR][B]1[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] gasGiants[/COLOR][COLOR=#0000ff]="[/COLOR][B]0[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] popMult[/COLOR][COLOR=#0000ff]="[/COLOR][B]5[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] allegiance[/COLOR][COLOR=#0000ff]="[/COLOR][B]Na[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] baseCode[/COLOR][COLOR=#0000ff]="[/COLOR][B]N[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] subsectorCode[/COLOR][COLOR=#0000ff]="[/COLOR][B]L[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] primaryStar[/COLOR][COLOR=#0000ff]="[/COLOR][B]F4 D[/B][COLOR=#0000ff]"[/COLOR][COLOR=#990000] secondaryStar[/COLOR][COLOR=#0000ff]="[/COLOR][B]M4 D[/B][COLOR=#0000ff]">[/COLOR]
 [COLOR=#0000ff]<[/COLOR][COLOR=#990000]UWP[/COLOR][COLOR=#0000ff]>[/COLOR][B]A2008B8-A[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]UWP[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]starport[/COLOR][COLOR=#0000ff]>[/COLOR][B]A[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]starport[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]size[/COLOR][COLOR=#0000ff]>[/COLOR][B]2[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]size[/COLOR][COLOR=#0000ff]>[/COLOR]  
 [COLOR=#0000ff]<[/COLOR][COLOR=#990000]atmosphere[/COLOR][COLOR=#0000ff]>[/COLOR][B]0[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]atmosphere[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]hydrosphere[/COLOR][COLOR=#0000ff]>[/COLOR][B]0[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]hydrosphere[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]population[/COLOR][COLOR=#0000ff]>[/COLOR][B]8[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]population[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]government[/COLOR][COLOR=#0000ff]>[/COLOR][B]B[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]government[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]lawLevel[/COLOR][COLOR=#0000ff]>[/COLOR][B]8[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]lawLevel[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]techLevel[/COLOR][COLOR=#0000ff]>[/COLOR][B]A[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]techLevel[/COLOR][COLOR=#0000ff]>[/COLOR] 
 [URL="file:///C:/Users/jamesfe/Desktop/test%20Sector.xml#"][B][FONT=Courier New][COLOR=#ff0000]-[/COLOR][/FONT][/B][/URL] [COLOR=#0000ff]<[/COLOR][COLOR=#990000]tradeCodes hex[/COLOR][COLOR=#0000ff]="[/COLOR][B]3028[/B][COLOR=#0000ff]">[/COLOR]
 [COLOR=#0000ff]<[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR][B]Na[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]<[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR][B]Va[/B][COLOR=#0000ff]</[/COLOR][COLOR=#990000]tradeCode[/COLOR][COLOR=#0000ff]>[/COLOR] 
  [COLOR=#0000ff]</[/COLOR][COLOR=#990000]tradeCodes[/COLOR][COLOR=#0000ff]>[/COLOR]
  [COLOR=#0000ff]</[/COLOR][COLOR=#990000]world[/COLOR][COLOR=#0000ff]>[/COLOR]
  [COLOR=#0000ff]</[/COLOR][COLOR=#990000]worlds[/COLOR][COLOR=#0000ff]>[/COLOR]
 [COLOR=#0000ff]</[/COLOR][COLOR=#990000]sector[/COLOR][COLOR=#0000ff]>[/COLOR]

Space Hamster · Mar 30, 2009

coliver988 said:
depends. I work with banks, credit card companies, telcos for data exchange. ALL use fixed-width fields. I don't see that changing anytime soon, and I've been doing this since 1986. Inertia has a lot going for it

The list you just mentioned is probably all using COBOL and Fortran. Our legacy applications use COBOL and trying to get them off of fixed width files into an actual database as a means to store data takes an act of congress. :rofl:

I said modern programming languages.

inexorabletash · Mar 30, 2009

robject said:
Code:

my $uwp = "my world name 0101 A123456-8 A 111 Im K0 V M1 VI"; # also: "my world name 0101 A123456-8 A 111 Im K0 V M1 VI"; # World 0101 A123456-8 A A 111 Im F1 V... $uwp =~ /^(.*?)\s*(\d{4})\s+(.{7}-.) \s?(.)(.*?)([AR])? (\d{3}) (..) (.*)$/;

There are more annoying wrinkles to consider - some files use G for a zone code (Green, i.e. not Red or Amber), Zhodani worlds can have F/Forbidden and U/Unabsorbed, and TNE files have other codes I'm not hip to.

But yes, agreed. I plan to begin work on a formal retro-specification for the "v2" format. I'll post it to the Traveller Wikia and open it up for review. It's unlikely to be ready for eyeballs for a few weeks at least as I'll be out of the country most of the next week.

Next would come the border lists, which are plaintext lists of hexes.

Click to expand...

I stole the format wholeheartedly from J. Greely's MSEC format. It would need to be well documented, and the way it deals with out-sector borders is a hack. A gloriously beautiful hack that might be worth preserving for all time, but nonetheless a hack, so it bears consideration before enshrining.

Then the route lists... I'm of two minds about this. Dale Kemper uses simple hex pairs to lay out directed routes, while I prefer re-using the hex list format used in border data.

Click to expand...

I also use pairs (as does Greely's MSEC) for one big reason (beyond being lazy) - the need to encode out-sector routes (i.e. those that begin or end beyond a sector). Borders can conveniently be represented accurately by only going one hex beyond the sector unambiguously and encoded using pseudohexes like 0000 and 3341, but routes may extend beyond what can be encoded. MSEC and I do this by adding sector offsets (e.g. "3240 -1 -1" meaning the lower left hex of the sector to spin-coreward). However, this can be encoded into a list using novel syntax (e.g. "... -1:-1:3240 ...") or supporting two formats. Suggestions welcome.

I also like being able to tag individual items with metadata ("this route was published in TD#9, article authored by Joe Fugate"). Assuming we settle on XML, I'd like to find a good metadata syntax standard. XML namespaces can make life hellish, but this is one good use for them.

Re: attributes vs. elements - ah, the endless debate. Meanwhile, non-XML fans point and laugh at the tag soup.

I mentioned to rob off-forum that something that could serialize to either JSON or XML would make me happy.

Anyway... I'll also take a stab at writing something up on this for the wiki for comments.

coliver988 · Mar 30, 2009

Space Hamster said:
The list you just mentioned is probably all using COBOL and Fortran. Our legacy applications use COBOL and trying to get them off of fixed width files into an actual database as a means to store data takes an act of congress. :rofl:

I said modern programming languages.

Could be, but I've not used COBOL for the last 10 years. And we use fixed width files for a variety of reasons. We also use SQL - depends on the system and the needs. But the fixed width files are a LOT simpler to deal with, take significantly LESS space than SQL, and work very well for data transfer. It is a lot more complicated to transfer SQL (you can do CSV but then you lose the metadata/comments We use SQL internally and export what we need into either CSV or flat fixed files, depending on intent). And fixed-width used for most of our data transfers from a few dozen other companies who are NOT running COBOL or Fortran (yes, I've asked) as well as the few switches we use (I work for a CLEC) and those switches send the files in fixed width (although not pure ASCII - there are some binary things in there, and some really odd Nortel things. but they are fixed width)

Admittedly some of the languages I use you may not consider modern, but they work, are still in production and are maintained and extended. And the reason a lot of the old programs are still around after 30, 40 or more years (I maintain a system originally written when the CLECs were formed w/the AT&T breakup back in the early 80s [it's been through a few hardware/OS but the same language for 20+ years]) is that they are well written and work.

Fixed has a LOT of advantages to it, and works fine for the SEC stuff with the extensions as expressed here. ANY data transfer system works as long as both the producer & the consumer know what they are getting. While I use XML for a lot of things, I think the SEC files don't need it.

Bottom line this is a thread for the SEC file format, which is a fixed width. You often have to use what you've got, and that format (all variations of it) have lasted the test of time for a long time. It is a common format with a LOT of sites & software using it. Some of us would like to extend it for additional things. I actually do use XML for some of my stuff, but as I'm the only one with that particular format, and it changes as I want to do new things, it is not useful for anyone else but me.

edit: But if we do get a standard XML, I'm all for that as well. Just fixed really should not be bashed simply because your Java/PHP or whatever script language of the day can do things differently. So I'm an old C-style programmer with an actual computer science degree - not one of those programming degrees they put out now (and get off my lawn!)

robject · Mar 31, 2009

jfetters said:
Yeah, I have a format that I created a few years ago for a program I wrote (which I need to put back on the download list). The source program enabled generating the sector in XML, then allowed you to export to HES or SEC formats via XSLT. Unfortunately, I haven't had any cycles to revisit or refine, as I wanted to transform from the text formats into my XML, but I think Joshua has that well in hand!

Anyway, here is the world XML format:

I like this, but in a modified form, as a good candidate for Joshua's "system information" format -- i.e. expanded data for all worlds within a system. My tweaks:

(Apologies to those who dislike attributes. Some unrolling is fine by me.)

NOTE This format is next to worthless if it's not useful.... some of these fields may be silly, but I'm just brainstorming now.

Code:

<system planetoids="2" 
	 gasGiants="4"
        rockballs="4"  
        stars="2"
        subsector="A" 
        hex="0101" >
   <primary type="M8 III">
      <name>Murcheson's Eye</name>
      <companion type="F D">
         <name>Mote</name>
      </companion>
	  <worlds>
        <world orbit="3">
         <name>Fetters Alpha</name>
		 <UWP code="B373665-B"> 
		   <starport code="B" /> 
		   <size code="3">3000</size> 
		   <atmosphere code="7">7</atmosphere> 
		   <hydrosphere code="3">30%</hydrosphere> 
		   <population code="6" popMult="9">9,000,000</population> 
		   <government code="6">Captive</government> 
		   <lawLevel code="5">Moderate</lawLevel> 
		   <techLevel code="B">11</techLevel>
		 </UWP>	 
         <tradeCodes>
           <tradeCode code="Ni">Non-industrial</tradeCode>
           <tradeCode code="Ri" />  		
         </tradeCodes>
	     <zone code="R">Interdicted</zone>
         <allegiance code="Na">Non-aligned</allegiance>
        </world>
       ...
      </worlds>
   </primary>   
   <secondary type="G D" orbit="8">
      <name>Speck</name>
   </secondary>
   <secondary type="F0 V" orbit="100" />
</system>

[/QUOTE]

jfetters · Mar 31, 2009

I really like that direction there, Rob. I was looking at one of the Firefly 'Verse maps the other day and thinking it would be nice to render that in a format kind of like this.

My initial suggestion would be that the UWP has to be the primary mandatory element or attribute for a system. And also determine where the other mandatory system attributes need to reside (attributes in this case as in characteristic, not XML). As I recall, that is how I ended up with elements in my XML (at the time I just didn't want as many elements and didn't mind the attributes).

Anyway, once the mandatory system attributes have been determined, the rest of the system details can be added if applicatible. That way, if there is a system imported from a SEC file, which won't have fine-grained system details, there aren't a bunch of potentially out-of-band data points in between the root of the <system> group and <world>, like star names and orbits, but still allow them to be added in later.

So, something like this:

Code:

<system planetoids="2" 
	        gasGiants="4"
        rockballs="4"  
        stars="2"
        subsector="A" 
        hex="0101" 
        uwp="B373665-B">
   <primary type="M8 III">
      <companion type="F D"/>
       <worlds>
         <name>Fetters Alpha</name>
         <UWP code="B373665-B"> 
		            <starport code="B" /> 
            <size code="3">3000</size> 
		            <atmosphere code="7">7</atmosphere> 
		            <hydrosphere code="3">30%</hydrosphere> 
		            <population code="6" popMult="9">9,000,000</population> 
		            <government code="6">Captive</government> 
		            <lawLevel code="5">Moderate</lawLevel> 
		            <techLevel code="B">11</techLevel>
		        </UWP>	 
         <tradeCodes>
           <tradeCode code="Ni">Non-industrial</tradeCode>
           <tradeCode code="Ri" />  		
         </tradeCodes>
         <zone code="R">Interdicted</zone>
         <allegiance code="Na">Non-aligned</allegiance>
        </world>
      </worlds>
   </primary>   
   <secondary type="G D"/>
</system>

jfetters · Mar 31, 2009

coliver988 said:
And fixed-width used for most of our data transfers from a few dozen other companies who are NOT running COBOL or Fortran (yes, I've asked) as well as the few switches we use (I work for a CLEC) and those switches send the files in fixed width (although not pure ASCII - there are some binary things in there, and some really odd Nortel things. but they are fixed width)

Admittedly some of the languages I use you may not consider modern, but they work, are still in production and are maintained and extended. And the reason a lot of the old programs are still around after 30, 40 or more years (I maintain a system originally written when the CLECs were formed w/the AT&T breakup back in the early 80s [it's been through a few hardware/OS but the same language for 20+ years]) is that they are well written and work.

Yeah, I remember that. I worked on a project almost 10 years ago with a Telco and remember the formats for those. Nothing wrong with them, especially with regards to what the system that uses them does. We had to transform them into XML in order to do other things with them, but yeah it was certainly nothing outlandish.

Fixed has a LOT of advantages to it, and works fine for the SEC stuff with the extensions as expressed here. ANY data transfer system works as long as both the producer & the consumer know what they are getting. While I use XML for a lot of things, I think the SEC files don't need it.

Bottom line this is a thread for the SEC file format, which is a fixed width. You often have to use what you've got, and that format (all variations of it) have lasted the test of time for a long time. It is a common format with a LOT of sites & software using it. Some of us would like to extend it for additional things. I actually do use XML for some of my stuff, but as I'm the only one with that particular format, and it changes as I want to do new things, it is not useful for anyone else but me.

edit: But if we do get a standard XML, I'm all for that as well. Just fixed really should not be bashed simply because your Java/PHP or whatever script language of the day can do things differently. So I'm an old C-style programmer with an actual computer science degree - not one of those programming degrees they put out now (and get off my lawn!)

I don't think there is anything wrong with SEC or HES in and of themselves, generally speaking. I like them as they are human-readable as well as readable by several other Traveller mapping programs. But if you want to do anything more with that, you can't. So if data in older formats can be transformed into a more updated format that allows for detail expansion, I think it's a good thing. It doesn't necessarily invalidate the old format but allows a developer to create tools to create edit and manipulate more kinds of system information. And you can still extract that data and transform information into a SEC or HES if you want.

robject · Mar 31, 2009

jfetters said:
I don't think there is anything wrong with SEC or HES in and of themselves, generally speaking. I like them as they are human-readable as well as readable by several other Traveller mapping programs. But if you want to do anything more with that, you can't. So if data in older formats can be transformed into a more updated format that allows for detail expansion, I think it's a good thing. It doesn't necessarily invalidate the old format but allows a developer to create tools to create edit and manipulate more kinds of system information. And you can still extract that data and transform information into a SEC or HES if you want.

Hear, hear.

SEC files have the benefit of being a long-established standard (reference Joshua's weblink). But where application needs more data than the SEC format provides, then that suggests a new standard, and the way we do things now is to use structured, tagged data.

I've tried to extend the SEC format before, and I never liked the results; they always felt like a kludge. My two biggest kludges were to tack on route information at the end of each world line, and to try to use a "star grammar" which let me tell whether a star was close, near, far, or a binary companion. And just try to get those standards adopted... not going to happen.

So, as Joshua explained, the SEC format won't be going away. However, there are two other formats to standardize (system-level structures, and sector-level structures), and those won't be fixed width. I've posted suggestions for both, using Jim and Joshua's XML structures as a starting point (and I note that Joshua borrowed some of his structure from others as well). So I'm ready for a new pair of standards to coalesce.

To Summarize, it appears that we currently have one standard, and that is the fixed-field format on a single line for a single world. We do not have an adequate standard for representing sector- and supersector-level data, and we do not have a standard for representing system-level data. But maybe we can agree on some facets of them.

My initial suggestion would be that the UWP has to be the primary mandatory element or attribute for a system. And also determine where the other mandatory system attributes need to reside (attributes in this case as in characteristic, not XML). As I recall, that is how I ended up with elements in my XML (at the time I just didn't want as many elements and didn't mind the attributes).

Anyway, once the mandatory system attributes have been determined, the rest of the system details can be added if applicatible. That way, if there is a system imported from a SEC file, which won't have fine-grained system details, there aren't a bunch of potentially out-of-band data points in between the root of the <system> group and <world>, like star names and orbits, but still allow them to be added in later.

That works for me. I did another pass; see what you think about it. I expanded some fields.

Code:

<system mainworld="Fetters Alpha"
        hex="0101"		
        uwp="B373665-B"
		bases="A"
		zone="R"
		planetoids="2" 
		gasGiants="4"
        rockballs="4"  
        stars="2"
		allegiance="Na">

  <primary type="M8 III">
    <companion type="F D"/>
	<worlds>
	  <world>
	    <name>Fetters Alpha</name>
		<UWP code="B373665-B"> 
		  <starport   code="B" /> 
		  <size       code="3">3200</size> 
		  <atmosphere code="7">
		     <density>Standard</density>
			 <taint>Tainted</taint>
		  </atmosphere> 
		  <hydrographics code="3">30%</hydrographics> 
		  <population    code="6" popMult="9">9,000,000</population> 
		  <government    code="6">Captive</government> 
		  <lawLevel      code="5">Moderate</lawLevel> 
		  <techLevel     code="B">11</techLevel>
		</UWP>
        <bases code="B">
		   <navy>Naval Base</navy>
		   <scouts>Scout Way Station</scouts>
        </bases>		
        <tradeCodes>
           <tradeCode code="Ni">Non-industrial</tradeCode>
           <tradeCode code="Ri" />  		
         </tradeCodes>
         <zone code="R">Interdicted</zone>
         <allegiance code="Na">Non-aligned</allegiance>
        </world>
      </worlds>
   </primary>   
   <secondary type="G D"/>
</system>

Space Hamster · Mar 31, 2009

Hi there,

I would be careful with how much information you put into the file. What I mean is that whatever you put into the file you will want a standard for. For example, you have a property Government with a value 6 being a captive government. Is the table for government types the same for all versions of Traveller? If we put expected data into the files, then that data would need to be available for people (application developers) to know what acceptable values for that field is. It might be necessary to have a XML file with the valid data for the fields to be available with the standard template. I do not know if this would be acceptable with Mark Miller.

I do not know how much difference there are between versions of Traveller for system data. I think if you do multiple sections off the root node you can achieve a flexible system. Have your first section be your base that is common amongst all versions of Traveller and what is in the current files. Then you can have sections that are for version specific data. This way any tool that is reading the file can read the base section and any version specific section needed.

Code:

<TravellerRootNode>
     <CommonSectorData>
          Stuff…..
     </CommonSectorData>
     <CTSectorData>
          Stuff…..
     </CTSectorData>
</TravllerRootNode>

This will allow growth and applications can be set to read data for a specific version of Traveller. Another advantage of this (XML not my specific example) is that people can add to the standard without breaking existing applications (new applications developed to the XML file type). This is as long as they only add fields not remove or change.

Gadrin · Apr 1, 2009

Code:

<system mainworld="Fetters Alpha"
        hex="0101"        
        uwp="B373665-B"
        bases="A"
        zone="R"
        planetoids="2" 
        gasGiants="4"
        rockballs="4"  
        stars="2"
        allegiance="Na">

[FONT=arial,helvetica]

Quick question about bases="A" if you had a system with more than one base what would the data look like ?

bases="S N" or "S,N" or something else ?

Multiple data items are stored in a space delimited list (ala classNames in CSS) or comma separated value list, etc.

>
[/FONT]

coliver988 · Apr 1, 2009

Gadrin said:
Code:

<system mainworld="Fetters Alpha" hex="0101" uwp="B373665-B" bases="A" zone="R" planetoids="2" gasGiants="4" rockballs="4" stars="2" allegiance="Na">

[FONT=arial,helvetica]

Quick question about bases="A" if you had a system with more than one base what would the data look like ?

bases="S N" or "S,N" or something else ?

Multiple data items are stored in a space delimited list (ala classNames in CSS) or comma separated value list, etc.

>
[/FONT]

I think you could have multiple bases:

Code:

<system mainworld="Fetters Alpha"
        hex="0101"        
        uwp="B373665-B"
        bases="A"
        bases="N"
        zone="R"
        planetoids="2" 
        gasGiants="4"
        rockballs="4"  
        stars="2"
        allegiance="Na">

[FONT=arial,helvetica]
[/FONT]

Sector Data File Formats (.sec)

SOC-14 1K

SOC-14 1K

SOC-14 1K

SOC-12

SOC-14 10K

SOC-14 1K

SOC-14 1K

SOC-14 10K

SOC-11

SOC-14 1K

SOC-12

SOC-14 1K

SOC-14 1K

SOC-14 10K

SOC-14 1K

SOC-14 1K

SOC-14 10K

SOC-12

SOC-14 1K

SOC-14 1K

Similar threads