Software Systems

[Daoloth]

So I'm already dusting off my b-grade software engineering skills to make another go at a Ref/Player aide application, and I came across one of the handful of recurring issues: Data Exchange. Now, I know this gets close to off-topic or something that maybe should be in the public forums, but: 1) chunks of this would have to be specific to T5; 2) I have bad experiences with some of the public Traveller communities (and various gamer forums in general,) so I am reluctant to bring this up elsewhere; 3) The readership here is the proper 'target audience'.

So, rather than starting to talk about each of our favorite apps, I want to talk about developing a data interchange standard to promote sharing between us all. Now, any code-monkey worth the name can hack out a translator or import module for a new data format in a couple of hours, it would still be nice not to have to.

So unless this thread gets locked (and no offense taken if it is,) here's what I'd like to hear about:

A set of properly formed XML-DTDs and documentation to cover all of the core data. Core data being defined as: character stats, gear stats, vehicles, starships, systems, anything else that pops up in the new rules that we want to share.

Why not just do this myself you ask? Well, I'd have to learn XML at a much lower level than I currently do and this isn't exactly a task for a beginner, also if this is to have any use beyond my own pet projects at some point the community would need to buy in.

 

[kilbrj]

So I'm already dusting off my b-grade software engineering skills to make another go at a Ref/Player aide application, and I came across one of the handful of recurring issues: Data Exchange. Now, I know this gets close to off-topic or something that maybe should be in the public forums, but: 1) chunks of this would have to be specific to T5; 2) I have bad experiences with some of the public Traveller communities (and various gamer forums in general,) so I am reluctant to bring this up elsewhere; 3) The readership here is the proper 'target audience'.

So, rather than starting to talk about each of our favorite apps, I want to talk about developing a data interchange standard to promote sharing between us all. Now, any code-monkey worth the name can hack out a translator or import module for a new data format in a couple of hours, it would still be nice not to have to.

So unless this thread gets locked (and no offense taken if it is,) here's what I'd like to hear about:

A set of properly formed XML-DTDs and documentation to cover all of the core data. Core data being defined as: character stats, gear stats, vehicles, starships, systems, anything else that pops up in the new rules that we want to share.

Why not just do this myself you ask? Well, I'd have to learn XML at a much lower level than I currently do and this isn't exactly a task for a beginner, also if this is to have any use beyond my own pet projects at some point the community would need to buy in.

I can do that relatively easily. I've already started some of that anyway since I've been building a php web application to do all of character creation and so I had to find a way to store history(since I wanted the characters to know how each year went after chargen was done) along with automatic checking for waivers, etc from what I had of the old chargen. After we get each section I'll take the core data around character, weapon, ship, etc and turn them into an XML statement for posting here.

 

[Daoloth]

Awesome. We're in a good place to set up a de-facto standard.

For systems, I was thinking of using the XML 'container' idea to handle stuff like detailed system generation etc... having required & optional stuff... you get the idea I'm sure.

 

[stpdfool]

If either of you are interested, I actually have an XML Schema file for holding "galaxy" information: that is, sectors, subsectors, etc.

I wrote it up based on the Mongoose Traveller playtest docs they used to have posted, so it may not be "complete". I also wrote about it when I was first learning XML, so I went with XML Schema (the "W3C Standard" to replace XML-DTD) instead of RELAX NG (what the world outside of the W3C is migrating to instead of XML-DTD).

Because it's in XML Schema, it's in a painful and long format, but hey, at least it can be used with a database (in theory). I could whip up a RELAX NG version in a jiff, though. Just drop me a line.

 

[Daoloth]

Well, since it's too premature to do anything specific, perhaps something more normative?

Here's what I want this to be able to do:

1) Fairly transparent. It should be possible to open the file up with a text editor and reverse-engineer something to import the data, or ideally have it be human readable.

2) Complete. If you can generate it in the rules, then you can use the file format.

3) Extensible. Basically the ability to add custom data to any object, to be handily discarded by software that doesn't know what it is. This might break verifiability.

Those reasonable goals?

 

[Clay]

If the data is defined with an XML Schema, it probably wouldn't be too hard to also define a standard software library that can read and write the data. In fact, it should be possible to generate that library or the core of it using the schema and XSL Transforms. I already do this for database schema defined in XML, so I don't see why it would be that hard to do the same for a file based schema.

If that library were linked against a cross-platform XML library like libxml2 (http://www.xmlsoft.org) and had import libraries for common languages like C, C++, Delphi and Visual Basic, it seems like we'd be in a pretty good place to start writing software.

 

[Daoloth]

My software engineering stuff is really crusty... the stuff out there now is so cool. But yeah, having a simple C library we could write wrappers against would be a worthy goal.

You mentioned using XML schema for RDBMSs... this would provide for a standard representation with such systems? Means you could write code against a MySQL or SQLite backend, say, and have it run against anyone's data, yes? That would allow for data-driven applications to be completely modular.

 

[robject]

If the data is defined with an XML Schema, it probably wouldn't be too hard to also define a standard software library that can read and write the data.

Absolutely, if we have a standard then we all can write libraries in our language of choice. But one thing at a time -- schema first, then API, then library? And the best guys to work on that are the ones who are writing apps that create it or use it.

 

[Daoloth]

You bet. The idea was to get all the people on the same page before we end up with a bazillion formats

 

[coliver988]

there was another thread about using XML for the back-end/data transport, and all sorts of arguments pro/con. As well as differing ideas about using it (attributes vs. elements, stuff like that - your traditional XML wars in the Traveller format!).

I'll keep out of it (I've been messing w/my own XML format for cargo codes which evolved into stuff for sectors, but I've stalled on that, and I saw some better schemas others were working on). But if you do get specs in order - post them - I'll probably update my little cargo program to use that instead of my own solution (and I started to also add support for SEC format files, but again, stalled out: there is just not enough time!)

 

[Daoloth]

Yeah, I saw that thread. While I'm not certain, I suspect that it might be too closely married to his particular app. And all we are doing now is brainstorming, so please chip in.

 

[coliver988]

what I had for the sector files (and starports were eventually to be added, as I was thinking about a starport program to sort of track info for that (how big, busy, etc)) was something like below. But there's lots of room for improvement - this was a first draft.

<?xml version="1.0" encoding="UTF-8"?>
<Sector Name="Spinward Marches">
    <Date>1101</Date>
    <Subsector Name="Regina">
        <System Name="Efate">
            <Location>1705</Location>
            <Political>Imp</Political>
            <Main>
                <UPP>646930</UPP>
                <Starport>A</Starport>
                <Tech>D</Tech>
            </Main>
        </System>
        <System Name="Alell">
            <Location>1706</Location>
            <Political>Imp</Political>
            <Permalink><![CDATA[http://www.travellermap.com/?x=-97.6015&y=74.5078125&scale=64&options=887]]></Permalink>
            <Main>
                <UPP>46789C</UPP>
                <Starport>B</Starport>
                <Tech>A</Tech>
            </Main>
            <Planets>
                <Planet Name="Mahnmut" Orbit="0">
                    <UPP>10013BC</UPP>
                    <Starport>G</Starport>
                    <Tech>9</Tech>
                    <Moons>
                        <Moon Name="Bink" Orbit="8">
                            <UPP>S00000</UPP>
                            <Starport>Y</Starport>
                            <Tech>0</Tech>
                        </Moon>
                    </Moons>
    
                </Planet>
                <Planet Name="Orphu" Orbit="1">
                    <UPP>12013A</UPP>
                    <Starport>F</Starport>
                    <Tech>A</Tech>
                </Planet>
                <Planet Name="Illium" Orbit="2">
                    <UPP>30021E</UPP>
                    <Starport>G</Starport>
                    <Tech>9</Tech>
                    <Moons>
                        <Moon Name="Skitt" Orbit="1">
                            <UPP>R00000</UPP>
                            <Starport>Y</Starport>
                            <Tech>0</Tech>
                        </Moon>
                        <Moon Name="Eon" Orbit="4">
                            <UPP>S00000</UPP>
                            <Starport>Y</Starport>
                            <Tech>0</Tech>
                        </Moon>
                    </Moons>
                </Planet>
                <Planet Name="Ionnus" Orbit="3">
                    <UPP>56012D</UPP>
                    <Starport>G</Starport>
                    <Tech>A</Tech>
                    <Remarks>Research station</Remarks>
                    <Moons>
                        <Moon Name="Halys" Orbit="2">
                            <UPP>200000</UPP>
                            <Starport>H</Starport>
                            <Tech>A</Tech>
                            <Remarks>Research station</Remarks>
                        </Moon>
                    </Moons>
    
                </Planet>

            </Planets>
        </System>

 

[kilbrj]

Start

Folks: Here's what I cranked out in about 2 minutes. It just starts at getting at what a character is(has the statistics), using XML Schema. I cranked it out using LiquidXML Studio simply because it's a fast and easy way to see how it looks from a data structure viewpoint. Feel free to add/modify it, comment on it etc.

James Kilbride

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Liquid XML Studio 1.0.8.0 (http://www.liquid-technologies.com) -->
<xs:schema elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:complexType name="Statistic">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
      <xs:element name="Value" type="xs:hexBinary" />
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="UPP">
    <xs:sequence>
      <xs:element name="C1" type="Statistic" />
      <xs:element name="C2" type="Statistic" />
      <xs:element name="C3" type="Statistic" />
      <xs:element name="C4" type="Statistic" />
      <xs:element name="C5" type="Statistic" />
      <xs:element name="C6" type="Statistic" />
      <xs:element name="CS" type="Statistic" />
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="TravelerCharacter">
    <xs:sequence>
      <xs:element name="FirstName" type="xs:string" />
      <xs:element name="LastName" type="xs:string" />
      <xs:element name="Stats" type="UPP" />
    </xs:sequence>
  </xs:complexType>
  <xs:element name="TravelerCharacter" />
</xs:schema>

 

[coliver988]

I like...

Probably need to add birth date (to calculate aging), home system/planet, perhaps optional things such as height, weight, etc. Room for equipment lists possibly.

All that depends on what this is going to be used in - but the real advantage of XML is the optional stuff: the schema has it but you don't need to include it if you don't want to in your data. Just include the required stuff.

I may have to dig up Stylus Studio (used for work, and it's been 2+ years). Also, for there are several free XML editors out there as well if people want to dabble. I've used XML Spy for some things, such as validating files.

And then some APIs to dig into resulting data - it would be relatively simple for me to generate .Net DLLs (pretty much ActiveX when compiled as a library) to be able to extract/update info from a specified schema.

 

[Daoloth]

There is a bunch of missing info, (skills for example) but this is manifestly the right idea. I know we're going to be getting rules to create proper racial profiles, so that needs to be there (it is on the char sheet.)

And yeah, the idea that extra data is simply ignored is wonderful... for instance if you want to include QREBS data (the gear-stat system mentioned in the rules) just add it in, and if the app doesn't want it, straight into the bit bucket.

 

[Daoloth]

Totally off topic...

But I really wanted to share this anecdote with you other code monkeys.

I have a Computer Information Systems textbook from 1957, a day so long gone that it was able to include a complete list of computer manufacturers as an appendix. In their discussion of hexadecimal encoding, the mapping supposedly in widest use was: 10 -> T, 11 -> E, 12 -> W, 13 -> R, 14 -> F, 15 -> V. Our current A-F mapping was completely absent from the discussion.

 

[Clay]

Tangential to this discussion, I have knocked out a world generator. It doesn't (yet) serialize the data to XML, but there's nothing stopping it from doing that. Right now it just doesn't contain enough data to make sense that way.

http://www.obrienscafe.com/node/2

On the topic of XML Spy, have a care when using that tool. There's a firm that I exchange a lot of data with, and quite a few very serious, show-stopping problems have been tied directly to the use of XML Spy on their part. I'm sure it's a great program for those who know what they are doing, but for the rest, think hard before using it.

 

[kilbrj]

There is a bunch of missing info, (skills for example) but this is manifestly the right idea. I know we're going to be getting rules to create proper racial profiles, so that needs to be there (it is on the char sheet.)

And yeah, the idea that extra data is simply ignored is wonderful... for instance if you want to include QREBS data (the gear-stat system mentioned in the rules) just add it in, and if the app doesn't want it, straight into the bit bucket.

It was not meant to be all inclusive. Since we don't have everything for skills I haven't included it but what I'd want to do is a 'history' section which lays out what career a character had when and what skills/stats were aquired during that and so on. I plan to add those in but wanted to shove this out the door first as a 'working draft' much like the pdf's are.

 

[nwnwnw]

what I had for the sector files (and starports were eventually to be added, as I was thinking about a starport program to sort of track info for that (how big, busy, etc)) was something like below. But there's lots of room for improvement - this was a first draft.

<?xml version="1.0" encoding="UTF-8"?>

There should be an encapsulating top level element here that
somehow designates the universe, milieu (sp?). Something like

<milieu date="1105-001" id="canon">

The id field need to be thought out of course.

<Sector Name="Spinward Marches">
    <Date>1101</Date>

date should be an attribute and attached to any element that
needs it, inheriting downward.

The sector element also needs some kind of locational information
itself to place it.

<sector name="Spinward Marches" x="-4" y="1">

date being inherited from the encapsulating milieu as it isn't specifically
different.

    <Subsector Name="Regina">
        <System Name="Efate">
            <Location>1705</Location>

As a matter of design style, I would make information that can't
conceivably have sub-elements be an attribute, rather than
an element. At least in the general case. Thus,

<system name="Efate" location="1705">

Also, if the standard is going to use sector wide location numbering
(and I think it should), the subsector container isn't necessary.
I would make it optional.

            <Political>Imp</Political>

Shouldn't "Political" be "allegiance" for consistency. At that,
we need some sort of canonical list of allegiance codes. The two
character codes seemed to be running out of space. Actually,
now that I think about it, there should be a series of "Imperial
Standards" which spell out these sort of things. Thus,
"Imperial Standard 2137 - Three Character Allegiance Codes (IS-2137)".
Q.V. ISO-3166-1 http://en.wikipedia.org/wiki/ISO_3166-1_alpha-3

            <Main>
                <UPP>646930</UPP>
                <Starport>A</Starport>
                <Tech>D</Tech>

Isn't the UPP technically all of the three above? I would just
make it an attribute, probably a single one. At the point we
have to pick apart the upp to get at the specific individual numbers,
picking apart the string for the starport code and tech level isn't
really any more work.

Also, since you don't name the main here (it being the same as the system name),
we need to put the star's name somewhere. I suggest that a
<star> element is needed.

            </Main>
        </System>
        <System Name="Alell">
            <Location>1706</Location>
            <Political>Imp</Political>
            <Permalink><![CDATA[http://www.travellermap.com/?x=-97.6015&y=74.5078125&scale=64&options=887]]></Permalink>

What is the permalink element intended to accomplish?

            <Main>
                <UPP>46789C</UPP>
                <Starport>B</Starport>
                <Tech>A</Tech>
            </Main>
            <Planets>
                <Planet Name="Mahnmut" Orbit="0">

Here I would make the orbit a sub-element, rather than an attribute
so that it can have more detailed information, e.g. exact value,
eccentricity, declination.

Similarly, the rotational period needs to go somewhere.

(Remainder of example trimmed since I had no further comments
on it.)

 

[Daoloth]

See, this is exactly what I wanted

One thing that putting a 'timeframe' into it would allow you to, for instance, watch the unfolding of the 5FW or the Imperial expansion as a kind of slideshow... which would be so awesome for us "history" fiends.