Extended System Data

[robject]

Fellow software engineers and others,

I've always played around with the UWP's format, adding a field here, compressing a field there, moving things around, just to see what might work. Needless to say, I've had more failures than successes.

My latest idea is encoding system data. I've always wanted a format that resembles the extended system data found in the back of Book 6, Module 8, and so on -- something that faithfully represents a set of Traveller data for an entire system which is also very human-friendly.

Today, I'm considering YAML, a compact, human-friendly data serialization format which modern dynamic languages can parse relatively easily (and other languages can parse somewhat less easily).

Rather than bore you further, I'll post a sample.
</font><blockquote>code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">0921:
uwp: Hrunting 0921 B463747-9 M Ri 313 Sw M V :0922
system:
- Hrin 0.0 M V
- Naga 0.4 Y200000-0 Va 0 Sw:
- Aba 20 Y000000-0 As 0 Sw
- Ningul 45 Y000000-0 As 0 Sw
- Hrunting 1.0 B463747-9 M Ri 3 Sw:
- Niddhog 200 Y100100-9 Va 1 Sw
- Tyr 2.0 C000447-9 M Belt 4 Sw
- Friye 4.0 YF00000-0 LGG 0 Sw:
- Friye 1 40 Y100000-0 Va 0 Sw
- Friye 2 80 Y200000-0 Va 0 Sw
- Friye 3 120 Y200000-0 Va 0 Sw
- Friye 4 150 Y200000-0 Va 0 Sw
- Friye 5 200 Y000000-0 As 0 Sw</pre>[/QUOTE]

 

[StuartFerris]

I've already implemented something similar for the HE2 System Module beta. The data is an easy to read format and can be loaded into the Module for editing.

</font><blockquote>code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">HE0 File Format (V1.0.5)
------------------------

SYSTEM UWP

Hex Location - 0304
Name - Serk
UWP - B89A866-C
Trade Code - Wa
Population Multiplier - 7
Planetoid Belts - 2
Gas Giants - 3
Base Code - G
Allegiance - Vf
System Nature - Primary
Primary Star - M4 V

SYSTEM DETAILS

Primary Habitable Zone - 0
Primary Maximum Orbits - 7
Primary Empty Orbits - 1
Primary Captured Planets - 0

PRIMARY SYSTEM - M4 V

0 (0.2) - Serk B89A866-C Wa
1 (7) - Shabush YS00127-B
2 (8) - Aglashlanlur YS00265-B
3 (40) - Ginamiiduug YS00564-B Co
1 (0.4) - Aadnigazursha Y000000-B
2 (0.7) - Esha YB00000-B
1 (30) - Uglen YS00000-B
3 (1.0) - Alernishim YC00000-B
1 (5) - Uggargasmine H100000-B
2 (8) - Pashigamam Y100000-C Ml
3 (9) - Liishkhi Y500000-B
4 (40) - Lushigegushe Y740000-C Re
4 (1.6) - Shilegishasha Y000000-B
5 (2.8) - Gakakakhinka YC00000-B
1 (1) - Nanikirgale YR00000-B
2 (2) - Kugdumgaa YR00000-B
3 (4) - Arshimzigam Y300000-B
4 (7) - Ungagiginik Y624000-B
5 (11) - Arguur Y663116-B
6 (30) - Kuukidiikuki Y543145-B
7 (35) - Mikagamgir Y100000-C Re
8 (40) - Irshashuusade Y300000-B
9 (300) - Arla Y450000-B
6 (5.6) - Empty Orbit</pre>[/QUOTE]

 

[kaladorn]

I think I like the H&E variant a bit better in terms of readability. Of course, I'm still not a fan of terseness at the expense of readability. Disk space is cheap these days, memory not much further off. The reasons for using arcane codings have (to my mind) essentially vanished. YMMV.

 

[RainOfSteel]

Originally posted by kaladorn:
I think I like the H&E variant a bit better in terms of readability. Of course, I'm still not a fan of terseness at the expense of readability. Disk space is cheap these days, memory not much further off. The reasons for using arcane codings have (to my mind) essentially vanished. YMMV.

If you have enough room to display everything long-hand, then, displaying it all that way is a good idea.

But when you want to show lots of worlds at once, then other display formats are more appropriate.

In an ideal world, the database tables holding our world/stellar info would all include a fully descriptive value for each possible range of values for each attribute.

Then, when it comes to display time, and you only need to display one world, you can display the full long-hand directly (it will likely fill the screen, at least, if you include full information; and if you include only basic info, then only a few worlds can be displayed at once).

And when you need to display lots of worlds, you can display it short-hand UWP-style (translating the long-hand info to short-hand dynamically right before display-time).

 

[Vanguard]

From a database perspective, I'd actually want to do it in reverse. Your data tables will have a short code or ID number for a type of govt, tech level, etc., and a separate table matches those codes up to their long descriptions.

When you want a bunch of data on one page, you can just output the codes (If you assigned them with readability in mind). If you want something more easily readable by your players who havn't learned to decode UWPs and such, you can do a lookup on the other table to get the long descriptions and display those. There also could be a third column in your lookup table for abbreviated descriptions.

You might have been meaning that you're not wanting to abbreviate numerical data (population or world size for instance) into classification codes. I'd agree with that argument, we'd want to digest the more exact data into classification codes when necessary, rather than lose the exact data by storing a classification code.

 

[RainOfSteel]

Vanguard,

It's a translation lookup one way or another.

In small databases like those I'd expect Traveller fanns would use, it isn't going to matter much.

In a large database, it would depend on which lookup would get done more often. I'd structure it whichever way reduced the number of lookups needed. The cpu-time needed to join the tables would vastly exceed the cpu-time needed to grab larger columns out of a single table.

Further, for Traveller fanns who were digging through any provided product's (fan-provided, of course) tables, those that hate the code-and-lookup of the basic UWP/USP/UCP system, would again be faced with it while looking at the raw tables. By having the base tables loaded with the long-hand information, less-than-totally database savvy fanns would get along with the database a lot better.

 

[RainOfSteel]

Originally posted by Vanguard:
You might have been meaning that you're not wanting to abbreviate numerical data (population or world size for instance) into classification codes. I'd agree with that argument, we'd want to digest the more exact data into classification codes when necessary, rather than lose the exact data by storing a classification code.

Oh, I am also for this idea. I've been sad for a long time now that many points of exact world info are lost in the summarized code system.

 

[Vanguard]

I was worried more about the storage of redundant data than cpu-time for lookups.

You're right, in a small project it shouldn't really make much difference...

Of course, they always start out small, and always get much bigger than you wanted them to get.

Which is why I generally try to do it the "right" way to begin with. I'm a database nerd, so I try to do any database stuff the same way I'd do a large commercial project.

 

[RainOfSteel]

If the same data is going to exist in either table, the base or the lookup, then the redundency is the same. If you're talking quantity of data storage caused by the redundency, that's a question of degree to be dealt with at design time. Saying one way is right because it uses fewer bytes isn't taking the whole picture into account.

If your users won't use your database, or find it horrible but unavoidable to use, then it doesn't matter how it was designed.

Successful design involves the analysis and balancing of many factors, no one of which is ever paramount. (Not even the holy grails of reliability and portability.)

 

[robject]

Originally posted by Stuart Ferris:
I've already implemented something similar for the HE2 System Module beta. The data is an easy to read format and can be loaded into the Module for editing.

That's an excellent format, Stuart! If you don't mind, I think I'll steal your data and try to fit it into a YAML format. So here's my first pass.

</font><blockquote>code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">#/usr/bin/perl

use YAML;

my $data = Load(<<'...'); # File( 'serk.yaml' );
---
SYSTEM UWP:
Hex Location: 0304
Name: Serk
UWP: B89A866-C
Trade Codes: Wa
Population Multiplier: 7
Planetoid Belts: 2
Gas Giants: 3
Base Code: G
Allegiance: Vf
System Nature: Primary
Primary Star: M4 V

SYSTEM DETAILS:
Primary Habitable Zone: 0
Primary Maximum Orbits: 7
Primary Empty Orbits: 1
Primary Captured Planets: 0

PRIMARY SYSTEM:
0:
- Serk 0.2 B89A866-C Wa
- Shabush 7 YS00127-B
- Aglashlanlur 8 YS00265-B
- Ginamiiduug 40 YS00564-B Co
1:
- Aadnigazursha 0.4 Y000000-B
2:
- Esha 0.7 YB00000-B
- Uglen 30 YS00000-B
3:
- Alernishim 1.0 YC00000-B
- Uggargasmine 5 H100000-B
- Pashigamam 8 Y100000-C Ml
- Liishkhi 9 Y500000-B
- Lushigegushe 40 Y740000-C Re
4:
- Shilegishasha 1.6 Y000000-B
5:
- Gakakakhinka 2.8 YC00000-B
- Nanikirgale 1 YR00000-B
- Kugdumgaa 2 YR00000-B
- Arshimzigam 4 Y300000-B
- Ungagiginik 7 Y624000-B
- Arguur 11 Y663116-B
- Kuukidiikuki 30 Y543145-B
- Mikagamgir 35 Y100000-C Re
- Irshashuusade 40 Y300000-B
- Arla 300 Y450000-B
6:
- Empty 5.6
...

# try fetching out a world and its data.
my $esha = $data->{ 'PRIMARY SYSTEM' }->{2}->[0];

# now split it up
my ($name, $orbit, $uwp, @codes) = split( ' ', $esha );

# and print it out
print "Name: $name\n Orbit: $orbit\n UWP: $uwp\n Trade codes: @codes\n";</pre>[/QUOTE]

 

[robject]

Originally posted by kaladorn:
I think I like the H&E variant a bit better in terms of readability. Of course, I'm still not a fan of terseness at the expense of readability. Disk space is cheap these days, memory not much further off. The reasons for using arcane codings have (to my mind) essentially vanished. YMMV.

My mileage varies, depending on whether I'm in the city or on the highway. With databases, I like string explanations instead of government codes and everything cross-indexed to tables and population numbers written out. But with text files, I like it dense.

I like to grep and visually scan UWPs in text files. The problem is not disk space, but brain space: it's important to me to have small and recognizable patterns to eyeball. When I'm looking at all the stars, worlds, and moons in the Spinward Marches, I don't want to look at, for example, fully interpreted codes and XML. I want terse, and I want them to line up on the page, like Stuart's formatting.

And while I'm still big on databases, I think a serialized text format is more appropriate for Traveller UWP data in general. For big applications, like a Traveller MMORPG, a database might be a good idea.

As kaladorn said, YMMV.

 

[aramis]

I'm more a fan of an alignable delimiter.
EG:
</font><blockquote>code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">PRIMARY SYSTEM - M4 V

0304.1.0 (0.2) - Serk B89A866-C Wa
0304.1.1 (7) Shabush YS00127-B
0304.1.2 (8) Aglashlanlur YS00265-B
0304.1.3 (40) Ginamiiduug YS00564-B Co
0304.2.0 (0.4) Aadnigazursha Y000000-B
0304.3.0 (0.7) Esha YB00000-B
0304.3.1 (30) Uglen YS00000-B
0304.4.0 (1.0) Alernishim YC00000-B
0304.4.1 (5) Uggargasmine H100000-B
0304.4.2 (8) Pashigamam Y100000-C Ml
0304.4.3 (9) Liishkhi Y500000-B
0304.4.4 (40) Lushigegushe Y740000-C Re
0304.5.0 (1.6) Shilegishasha Y000000-B
0304.6.0 (2.8) Gakakakhinka YC00000-B
0304.6.1 (1) Nanikirgale YR00000-B
0304.6.2 (2) Kugdumgaa YR00000-B
0304.6.3 (4) Arshimzigam Y300000-B
0304.6.4 (7) Ungagiginik Y624000-B
0304.6.5 (11) Arguur Y663116-B
0304.6.6 (30) Kuukidiikuki Y543145-B
0304.6.7 (35) Mikagamgir Y100000-C Re
0304.6.8 (40) Irshashuusade Y300000-B
0304.6.9 (300) Arla Y450000-B </pre>[/QUOTE]Really, I'd tab-delimit, not list/count orbit numbers, etc. Drop empty orbits. And just drop the whole shebang into a space delimited text file... which I can fairly easily split out in Word, excell, OpenOffice or Appleworks into separate working fields, or even strip out with Python

 

[RainOfSteel]

I personally prefer "aligned" space-delimited layouts, myself.

This makes it far easier to transport the data across platforms.

Comma/Tab delimited will do in a pinch.

The main problem with starsystem data in the OTU UWP is, that it's mainworld-centric. The real center of the starsystem, in OTU terms, is the Hex it's in (Primary Key in a Galaxy layout would be Sector X,Y plus Hex Col,Row), and the star and companion stars in that Hex. Worlds (mainworld or non-mainworld) are details hanging off their stars, not the other way around. Or, at least, that's IMO.

Drat, work calls <pesky salary need thingamagigit />

 

[robject]

^ Aramis' preference works really well when you're importing into spreadsheets.

Not Very Important: I would be tempted to leave off the initial hex prefix. The system text seems to be more like a unitary record than an independent group of things; hence the header could contain the hex number.

Surely Python is one of the targets of YAML; however, I admit I found YAML a bit annoying when trying to format system data into it.

But the allure is still there. YAML already takes care of a readable-text serializer/deserializer, so I'm drawn to it.

Nevertheless, plaintext is very readable. Makes one want to write a special module just for serializing/deserializing extended Traveller data, doesn't it?

 

[robject]

Originally posted by RainOfSteel:
I personally prefer "aligned" space-delimited layouts, myself.

This makes it far easier to transport the data across platforms.

Comma/Tab delimited will do in a pinch.

The main problem with starsystem data in the OTU UWP is, that it's mainworld-centric.

Aligned is very helpful.

I didn't think about platform transport. It's possible that a structured format might not survive intact. I assume modern systems (and veteran systems like UNIX) don't have that problem. And in fact, since agile languages are meant to be portable, and YAML was designed primarily for them, I'd prefer a record structure. However, I have heard of problems using YAML with Windows, and since I'm on Windows...

And for mainworld centrism. My Solar System table still had a (perhaps non-3nf) foreign key to the mainworld, because when a ship jumped, by default it jumped at the mainworld.

But that makes me think that a System Record will have to tag the mainworld in an obvious way.

EDIT: Ah, by 'transport' you might also be hinting at a spreadsheet. That's an interesting problem, and I sort of feel the point is worth conceding.

 

[robject]

Well, what do folks think about this? The System data isn't YAML, but it's convertible to YAML. The more I look at our examples the more I like aligned, whitespace-delimited data. Commas are sort of ugly, although a good compromise would be to align the commas so that they look like column borders.

Hex 0304:
   Hex Name: Serk
   Primary System:
      Data:
         Name:             Serk
         Mainworld UWP:    B89A866-C
         Population:       700000000
         Trade Codes:      Wa          
         Planetoid Belts:  2
         Gas Giants:       3
         Base Code:        G
         Allegiance:       Vf
         Primary Star:     M4 V  
         Habitable Zone:   0
         Maximum Orbits:   7
         Empty Orbits:     1
         Captured Planets: 0
   
      System: |
         0  Serk               0.2   B89A866-C  Wa
            Shabush              7   YS00127-B              
            Aglashlanlur         8   YS00265-B              
            Ginamiiduug         40   YS00564-B  Co          
         1  Aadnigazursha      0.4   Y000000-B
         2  Esha               0.7   YB00000-B
            Uglen               30   YS00000-B              
         3  Alernishim         1.0   YC00000-B
            Uggargasmine         5   H100000-B              
            Pashigamam           8   Y100000-C  Ml          
            Liishkhi             9   Y500000-B              
            Lushigegushe        40   Y740000-C  Re          
         4  Shilegishasha      1.6   Y000000-B
         5  Gakakakhinka       2.8   YC00000-B             
            Nanikirgale          1   YR00000-B              
            Kugdumgaa            2   YR00000-B              
            Arshimzigam          4   Y300000-B              
            Ungagiginik          7   Y624000-B              
            Arguur              11   Y663116-B              
            Kuukidiikuki        30   Y543145-B              
            Mikagamgir          35   Y100000-C  Re          
            Irshashuusade       40   Y300000-B              
            Arla               300   Y450000-B              
         6  Empty              5.6 
   
   Secondary System:
      - etc etc

Hex 0306: etc etc

 

[kaladorn]

Very nice. I think I prefer this to comma delimited. In fact, if you use newlines/CRLF as line delimiters and either spaces or tabs as inter-token delimiters within the lines, you should be easily able to process and parse this. It is more compact than XML, human readable/editable, and looks good. And is probably easily convertible to XML, YAML, etc.

 

[RainOfSteel]

Originally posted by robject:

EDIT: Ah, by 'transport' you might also be hinting at a spreadsheet. That's an interesting problem, and I sort of feel the point is worth conceding.

Spreadsheets, databases, scripting programs (YAML, Tablesmith, etc.), you name it, especially if one is moving it to Linux/Unix, Mac/OS (a version of Unix), etc.

 

[RainOfSteel]

Originally posted by robject:
Well, what do folks think about this? The System data isn't YAML, but it's convertible to YAML. The more I look at our examples the more I like aligned, whitespace-delimited data. Commas are sort of ugly, although a good compromise would be to align the commas so that they look like column borders.

</font><blockquote>code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">Hex 0304:
Hex Name: Serk
Primary System:
Data:
Name: Serk
Mainworld UWP: B89A866-C
Population: 700000000
Trade Codes: Wa
Planetoid Belts: 2
Gas Giants: 3
Base Code: G
Allegiance: Vf
Primary Star: M4 V
Habitable Zone: 0
Maximum Orbits: 7
Empty Orbits: 1
Captured Planets: 0

System: |
0 Serk 0.2 B89A866-C Wa
Shabush 7 YS00127-B
Aglashlanlur 8 YS00265-B
Ginamiiduug 40 YS00564-B Co
1 Aadnigazursha 0.4 Y000000-B
2 Esha 0.7 YB00000-B
Uglen 30 YS00000-B
3 Alernishim 1.0 YC00000-B
Uggargasmine 5 H100000-B
Pashigamam 8 Y100000-C Ml
Liishkhi 9 Y500000-B
Lushigegushe 40 Y740000-C Re
4 Shilegishasha 1.6 Y000000-B
5 Gakakakhinka 2.8 YC00000-B
Nanikirgale 1 YR00000-B
Kugdumgaa 2 YR00000-B
Arshimzigam 4 Y300000-B
Ungagiginik 7 Y624000-B
Arguur 11 Y663116-B
Kuukidiikuki 30 Y543145-B
Mikagamgir 35 Y100000-C Re
Irshashuusade 40 Y300000-B
Arla 300 Y450000-B
6 Empty 5.6

Secondary System:
- etc etc

Hex 0306: etc etc</pre>

[/quote]I'm very sorry to say, but without extensive scripting (in whatever language; and no, I don't know YAML, I just haven't had time to poke my head into since it was originally mentioned on CotI many months ago), none of that can be directly imported into other applications.

The indentation kills it all.

 

[RainOfSteel]

Well, back on the original topic of human-friendliness:

Personally, I found all the above formats to be perfectly acceptable for human-readability.