Standardized Data Format

[DickNervous]

So over in a Facebook group there is a discussion about building apps for Traveller and even integration via APIs. As I have been diving into web development and I learned more about XML it made me wonder why nobody has ever created a standard data format for exchanging information (such as character data, ship data, system info) between different applications using XML/XSD (or some other format that I don't know about).

NOTE: Please keep in mind that I am a NOVICE programmer and I can guarantee that there are tons of things I have no clue about, so be gentle if I am completely wrong about something)

I know some apps (like TravellerMap) allow you to get info in various formats (JSON, CSV, etc) via the API, but it seems to me that while CSV is great for large amounts of data (ie: sector data) that JSON requires a bit more programming know-how to use than XML would. But I am thinking more along the lines of character and ship data to be honest.

If there was a publicized XSD for character data then one app that generates characters could export them like that and people could use a variety of other apps to make character sheets. It would be pretty easy to read an XML file into Word or a PDF form to populate fields, and not much more difficult to use HTML/CSS to create a sheet from the XML.

The XML could be formatted in a way to be rule-system agnostic since tags can either be empty or simply not included if they are not applicable to a specific rule set. It would also be able to include a complete history (like the careers and a log), equipment lists, etc. And the same thing could be done for ships, though it might be a bit more complicated.

Is this something that has ever been discussed or talked about in the community? Is it even possible, or am I just off my rocker?

 

[whartung]

The only actual "standardized" (for assorted values of Standard) is the stellar data, which is spelled out via T5 on the travellermap website.

This is not a delimited format (ala XML or JSON), it's more CSV-ish, but the format can be specified in the file. Arguably, you can easily send a subset of data using this format, but by Standard, all of the fields are supposed to be there.

For everything else, outside of the UPP, USP, etc., I don't know of anything formal. That all gets ad hoc.

As a rule, people do not care for XML (I'm not one of them), so any new effort is likely to push toward a JSON format due to its ubiquity.

Proposals can be made all day long as to what a format should or should not be, but in the end, "code has momentum". Since there are a bevy of utilities, but no real source code, no real sharing, no real integration, there's no real de facto formats out there.

T5 Sector data is an exception due to it being documented and the fact that it's well supported by the 800lb Gorilla that is travellermap. (That whole momentum thing…)

 

[robject]

"Momentum is the driver of standards." -me

Anyway. JSON is a wonderful transfer format, mainly because it's (potentially) human-readable, rather straightforward, and (best part) has a bazillion already-made packages in many many languages which will do the import and export for you.

YAML is a much heavier format, but I think it's even more human-readable than JSON (and is a strict superset of JSON besides). I prefer YAML --- or, more typically, YAML "Tiny".

A description of what YAML "Tiny" is, is here.

an incomplete but correct and usable subset of the functionality, in as little code as possible.

[...] it provides support for only a limited subset of YAML. But the subset supported contains most of the features for the more common uses of YAML.

 

[tjoneslo]

I know some apps (like TravellerMap) allow you to get info in various formats (JSON, CSV, etc) via the API, but it seems to me that while CSV is great for large amounts of data (ie: sector data) that JSON requires a bit more programming know-how to use than XML would. But I am thinking more along the lines of character and ship data to be honest.

The preferred method for web applications and API (like Traveller Map) is JSON. All of the web programming languages (e.g. php, ruby, and the like) understand and use JSON. For the client side, well JSON stands for "JavaScript Object Format", it's supposed to be a serialized format for the Javascript objects.

XML as a data transfer format for web applications seems to be on decline. It's still used in many very heavy weight web applications. But all the cool kids use JSON these days.

If there was a publicized XSD for character data then one app that generates characters could export them like that and people could use a variety of other apps to make character sheets. It would be pretty easy to read an XML file into Word or a PDF form to populate fields, and not much more difficult to use HTML/CSS to create a sheet from the XML.

That said, if you would like a worked example of such a thing:

https://github.com/makhidkarun/cartography/blob/master/public_html/AstroML.xsd

This is an xsd file for the stellar data that I worked through

https://github.com/makhidkarun/cartography/blob/master/public_html/AstroML.css

And the corresponding CSS file to output it for a web page.

 

[kilemall]

What you need is more a data interface standard rather then forcing everyone to a language/enviornment, which will be ignored anyway as most programmers are going to use the tool that they are most comfortable with and/or fits the need of the app they are writing.

My first thought out of the box is to go .csv files. If everyone can work off of csv by having import/export APIs to their environments, and things like world data, character data, ship data, etc. are standardized as to which row carries what data in what order, you should be able to export/import with relative ease.

A sticky point of course are the differences between different Traveller versions for the above as you could get into nightmare scenarios of 2-10 character standard formats- might be a niche for 'converter' apps.

 

[DickNervous]

tjoneslo: Excellent! I will check those out!

And I guess I need to read up about JSON to understand it better and how it works. I agree that it would probably work better for web-based apps from what little I know of it. But I am also thinking of desktop apps (including Office documents, PDF forms, etc) that people may want to use.

One example that I thought of would be character sheets. One style could be appropriate for players, but too "big" for the referee to use. So a different style, where you could fit multiple characters (player of NPC) on a single sheet would be preferable. It would be nice to be able to populate those two different forms from the same data source.

At this point I am just trying to get people to think about it and brainstorm ideas, stirring up the pot as it were.

Thank you for the feedback!

 

[phorner]

I am using xml in the upcoming version 3 of the Travellers Aid Society Access Terminal. I am using the format for sector fleet information, configuration settings, and user profiles. I like the idea of standardizing the format for character and ship data. BTW, version 3 should be released before the end of the year.

 

[whartung]

CSV simply isn't rich enough, and we're not pushing around spreadsheets.

JSON is simple enough to be universally applicable, and has momentum in tooling.

JSON distinguishes from XML specifically by having a native array idiom, which XML lacks. Thus each XML format represents collections in their own way, but XML is richer in most every other category.

Both allow trivial extensibility (programs can simply ignore stuff they don't recognize as long as the meta structure isn't changed).

Programmers enjoy working with lists of maps of native JSON rather than trees of nodes of native XML. Simple XML obviously can be marshaled in to lists of maps, but natively it's not. Most anyone who actually works with XML in production does marshal the XML in to lists of maps (typically using first class classes and structures rather than maps).

JSON parsers are trivial to write. I appreciate that languages like QuickBasic aren't known for their dynamic data structures, and thus would struggle with something like JSON, or XML, or any other rich format that isn't a list of strings. That said, for mostly static information, QB is more than capable of parsing a JSON in to whatever structure it's comfortable with.

But, in this day and age, reducing the formats to such a lowest common denominator such as CSV really is a disservice to all of the modern development environments.

I'd advocate a JSON standard myself. I would argue against YAML, as JSON is readable enough, and (apparently) YAML is whitespace based (like python), thus more easily subject to corruption. There are numerous JSON and XML formatters available that will pretty-print "ugly" data if worst came to worst. I personally don't care for data or code that I can't run through an auto-formatter. Indention is a human need, not a machine need. That also means indention is a human burden. With deterministic grammar, a computer can remove that toil from my workload. I cut and paste too much code to want to have to worry about that.

 

[agorski]

My early Mac apps used XML because it was convenient at the time. Here's a sample format for a stored character:

<?xml version="1.0" encoding="UTF-8"?>
<character_file xmlns:xml="http://www.w3.org/XML/1998/namespace">
  <file_info schema="1" creation_date="5/6/11" modification_date="5/6/11"/>
  <character name="Wolf Blitzer" noble_title="Knight" str="7" dex="9" end="8" int="5" edu="8" soc="B" age="30" apparent_age="30" counter_ID="G12" college_grad="No" college_honors="No" cash="10000" retired="No" pension="No" pension_amount="0" TAS_member="No" psionics_trained="No">
    <worlds>
      <world name="Efate" type="Home" sector="Spinward Marches" subsector="Regina" hex_number="1705" UWP="A646930D"/>
      <world name="Regina" type="Discharge" sector="Spinward Marches" subsector="Regina" hex_number="1910" UWP="A788899C"/>
    </worlds>
    <career>
      <service name="Army" branch="Infantry" terms="3" officer="Yes" title="Second Lieutenant" rank_number="1" traveller_rank_number="1" academy_grad="No" academy_honors="No">
        <assignments>
          <assignment name="OCS" number="1"/>
          <assignment name="Protected Forces Training" number="1"/>
        </assignments>
        <awards>
          <award name="Combat Service Ribbon" number="2"/>
          <award name="Command Ribbon" number="5"/>
          <award name="MCUF" number="1"/>
        </awards>
      </service>
    </career>
    <skills>
      <skill name="Administration" level="1"/>
      <skill name="Advanced Combat Rifle" level="1"/>
      <skill name="Assault Rifle" level="1"/>
      <skill name="Automatic Pistol" level="1"/>
      <skill name="Carbine" level="1"/>
      <skill name="Gauss Rifle" level="1"/>
      <skill name="Grav Vehicle" level="1"/>
      <skill name="Revolver" level="1"/>
      <skill name="Rifle" level="1"/>
      <skill name="Vacc Suit" level="4"/>
      <skill name="Zero-G Combat" level="1"/>
    </skills>
    <possessions>
      <possession name="Automatic Pistol" number="1"/>
    </possessions>
    <creation_log text="Age=18 etc. etc."/>
  </character>
</character_file>

All my recent iOS apps use JSON, which I find easier to use and is better supported on the platform.

 

[robject]

My desktop shipyard reads and writes YAML, JSON, and XML. 'Cause when folks provide packages for processing in any data format, they're easy add-ons.

The XML is indeed just lists and attribute maps. Essentially all three are generated representations of internal data.