Serge NOEL
487 lines
24 KiB
Plaintext
487 lines
24 KiB
Plaintext
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta name="generator" content="HTML Tidy for HTML5 for Apple macOS version 5.8.0">
|
|
<title>JMRI: Xml Schema Usage</title>
|
|
<meta name="author" content="Bob Jacobsen">
|
|
<meta name="keywords" content="JMRI technical code xml schema usage">
|
|
<!--#include virtual="/help/en/parts/Style.shtml" -->
|
|
</head>
|
|
<body>
|
|
<!--#include virtual="/help/en/parts/Header.shtml" -->
|
|
|
|
<div id="mBody">
|
|
<!--#include virtual="Sidebar.shtml" -->
|
|
|
|
<div id="mainContent">
|
|
<h1>JMRI: XML Schema Usage</h1>
|
|
|
|
<p>JMRI uses XML for a number of purposes: to hold decoder definitions, for its <a href=
|
|
"XmlPersistance.shtml">persistance system</a> for configuration and panel information, and to
|
|
create parts of the website from other files. This page describes how we specify the content
|
|
of those files using <a href="https://www.w3schools.com/xml/xml_schema.asp">XML
|
|
schema</a>.</p>
|
|
|
|
<p>For examples (not a tutorial!) on the structure of our schema, see the <a href=
|
|
"XmlSchemaExamples.shtml">examples page</a>.</p>
|
|
|
|
<p>The current schema can be seen online in the <a href="https://www.jmri.org/xml/schema/">schema
|
|
directory</a>. The most commonly used one is the <a href=
|
|
"http://jmri.org/xml/schema/layout.xsd">layout.xsd schema for panel files</a>. See below on
|
|
how it's organized.</p>
|
|
|
|
<h2>Access to Schema Definitions</h2>
|
|
JMRI uses XML Schema to define the format of its files.
|
|
<p>Those XML Schema may need to be available to the program when it reads the files, as they
|
|
define the default values of missing attributes and other needed information.</p>
|
|
|
|
<p>In the JMRI distributions, these are stored in the /xml/schema/
|
|
directory. Note that they are not stored in each directory alongside the XML files. There are
|
|
just too many locations to keep such a set of schema definition files up to date. JMRI
|
|
itself, via the jmri.jmrit.XmlFile class, provides support for locating those files when the
|
|
XML parser needs them.</p>
|
|
|
|
<h2 id="modschema">Modifying JMRI Schema</h2>
|
|
This section talks about how to handle small changes to existing schema, for example added or
|
|
removing an attribute or element. For more extensive changes, including creating entirely new
|
|
types or new file formats, see the next section on "<a href="#devschema">Developing JMRI
|
|
Schema</a>".
|
|
<p>Every time you change what JMRI writes (and therefore reads) in an XML file, you must also
|
|
do the following:</p>
|
|
|
|
<ol>
|
|
<li>You need to change the code that does the reading and writing.</li>
|
|
|
|
<li>You need to change the schema file(s) so that the XML format can be properly
|
|
checked.</li>
|
|
|
|
<li>You have to provide new test XML files to make sure nothing has been broken, and in
|
|
some cases have to adjust old ones.</li>
|
|
</ol>
|
|
|
|
<p>Please don't skip the latter steps. They matter a lot to the long-term stability of JMRI
|
|
code.</p>
|
|
|
|
<p>If possible, it's best to make changes by addition, so that existing files can continue to
|
|
be read unchanged. JMRI strongly values backward compatibility, where any newer version of
|
|
JMRI can still load and use a file written by an older version.</p>
|
|
|
|
<p>If you can make a change that's just an addition, then the process is:</p>
|
|
|
|
<ol>
|
|
<li>Change your code</li>
|
|
|
|
<li>Add the new elements and attributes to the most recent version of the schema
|
|
file(s).</li>
|
|
|
|
<li>Check your updated schema files are valid with e.g.
|
|
<code class="block">
|
|
xmllint -noout -schema http://www.w3.org/2001/XMLSchema.xsd myChangedFile.xsd
|
|
</code>
|
|
|
|
<li>Run "ant headlesstest" to make sure that the older files already present in the tests
|
|
can still be processed. Fix anything that breaks. (You may discover at this point that you
|
|
didn't actually make a backward compatible change, in which case either improve that or see
|
|
the section on "<a href="#versioning">Schema Versioning</a>" below).
|
|
</li>
|
|
|
|
<li>Create a test file with the new content. Ideally, this won't require using the screen,
|
|
so it can actually be loaded and stored as part of headlesstest. In that case, put your
|
|
file in a "load" test subdirectory (see below). At a minimum, though, put your file in a
|
|
"valid" test subdirectory so that your schema changes will be tested. For more on this,
|
|
see <a href="#junitschematest">below</a>.
|
|
</li>
|
|
</ol>
|
|
|
|
<h3 id="versioning">Schema Versioning</h3>
|
|
<p>"Versioning" schema allows us to have different schema for older and newer files. This lets
|
|
newer versions of JMRI continue to check and read files that were written by older versions
|
|
of JMRI. This backward compatibility is an important JMRI feature which we don't want to
|
|
lose.</p>
|
|
<p>In practical terms, versioning consists of having multiple-but-related versions of the
|
|
schema definition files which are labeled by the first JMRI version that can read them.</p>
|
|
|
|
<p>When do you need to create a new version?</p>
|
|
|
|
<ul>
|
|
<li>You <em>do not</em> need to create a new version of the schema if you add or change
|
|
things such that existing files will continue to validate.
|
|
<p>In that case, just make your schema changes in the current schema document, and commit
|
|
them back to the JMRI code repository.</p>
|
|
</li>
|
|
|
|
<li>You <em>do</em> need to version a schema when you make a change to the schema such that
|
|
previous files will no longer validate with the current schema.
|
|
<p>In this case, the steps to version the schema are:</p>
|
|
|
|
<ol>
|
|
<li>Copy the current schema file to a new one with the number of the <em>next</em> JMRI
|
|
release. E.g. copy types/turnouts-2-9-6.xsd to types/turnouts-3.7.3.xsd if you're doing
|
|
this before JMRI 3.7.3 is released. Make your changes and commit that new version.</li>
|
|
|
|
<li>If this is a subfile, such as the types/turnouts-2-9-6.xsd, that is included in a
|
|
main schema such as layout-2-9-6.xsd, that main file also needs to be copied, have the
|
|
include changed, and commited. Now you've got a new layout-3-7-3.xsd schema for output
|
|
files to reference.</li>
|
|
|
|
<li>Then, change the Java code that writes the schema reference to the top of output
|
|
files to use the new filename. For example, layout (panel) files are written by
|
|
<div class="wide">
|
|
<code>src/jmri/configurexml/ConfigXmlManager.java</code>
|
|
</div>
|
|
Look for the
|
|
<div class="wide">
|
|
<code>static final public String schemaVersion = "-3-7-3"</code>
|
|
</div>
|
|
line and change it to your new version number suffix.</li>
|
|
|
|
<li>Run "ant headlesstest" and when tests fail, fix them. If a LoadAndStoreTest flags a
|
|
compare inLine/outLine error, update the schema version in the header of the reference
|
|
file. For example in
|
|
<div class="wide">
|
|
<code>java/test/jmri/configurexml/loadref/BlockAndSignalMastTest.xml</code>
|
|
</div>
|
|
point to the most recent layout.xsd version in line 3:
|
|
<div class="wide">
|
|
<code class="block"><layout-config
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:noNamespaceSchemaLocation="http://jmri.org/xml/schema/layout-4-7-2.xsd"></code>
|
|
</div>
|
|
</li>
|
|
|
|
<li>If there's an associated XML stylesheet(s), its name has to be changed in a
|
|
coordinated way. (You should also update the stylesheet to show your new XML content,
|
|
of course).</li>
|
|
</ol>
|
|
</li>
|
|
</ul>
|
|
In either case, it's important to include sufficient test files that the unit tests catch any
|
|
problems with the new and old schema. See the <a href="#test">test section below</a>.
|
|
<p>Note that the unlabeled schema is the primordial, oldest, now-obsolete schema. For example
|
|
layout.xsd is <em>older</em> than layout-2-9-6.xsd, and therefore not to be used for new
|
|
files anymore. Don't assume that layout.xsd is the default for new files!</p>
|
|
|
|
<h2 id="junitschematest">Checking JMRI Schema</h2>
|
|
<p>It's important that the JMRI schema definitions be kept semantically correct. If we let too
|
|
many problems build up, we'll eventually have a lot of back-fitting to do. The W3C online
|
|
<a href="https://www.w3schools.com/xml/xml_validator.asp">schema validation tool</a> is a
|
|
good tool for checking that changes to the JMRI schema are still technically correct. You
|
|
should check your changes with it before committing them to the repository. Unfortunately, it
|
|
doesn't seem to check compliance with nested schema elements, e.g. from DocBook (see below)
|
|
or JMRI schema, but it's still a very useful check.</p>
|
|
<p>Using the JMRI "Validate XML File" tool in the "Debug" menu to validate a .xml file
|
|
("instance file") that uses your new or updated schema is an important check of both. Use it
|
|
often during development. You can also use it from the command line via e.g.:</p>
|
|
<div class="wide">
|
|
<pre>
|
|
<code>
|
|
./runtest.csh apps/jmrit/XmlFileValidateRunner xml/decoders/0NMRA.xml
|
|
</code>
|
|
</pre>
|
|
</div>
|
|
<p>There is also a PowerShell script available, see the <a href="JUnit.shtml">JUnit</a> page
|
|
for more information.</p>
|
|
|
|
<p>For a quick file check, Linux and Mac OS X users can validate from the command line with
|
|
e.g.</p>
|
|
<code class="block">
|
|
cd xml
|
|
xmllint -schema schema/aspecttable.xsd -noout signals/sample-aspects.xml
|
|
</code>
|
|
<code>xmllint</code> can check schema files themselves if you specify
|
|
<code>-schema http://www.w3.org/2001/XMLSchema.xsd</code> in the command.
|
|
<p>Your schema docs should point to our standard stylesheet in their head matter:</p>
|
|
|
|
<pre>
|
|
<code>
|
|
<?xml-stylesheet href="schema2xhtml.xsl" type="text/xsl"?>
|
|
</code>
|
|
</pre>Stylesheets turn XML code, like this schema, into a human-readable form when the XML is
|
|
parsed and displayed by a browser. For an example of that, click on this link to the <a href=
|
|
"https://www.jmri.org/xml/schema/aspecttable.xsd">aspecttable.xsd</a> schema file. Our standard
|
|
stylesheet is pretty basic. It just shows basic structure. If anybody knows of a better stylesheet,
|
|
we can certainly switch to it.
|
|
<h3 id="test">JUnit testing</h3>
|
|
You should also add a <a href="JUnit.shtml">JUnit test</a> that checks a typical file. There
|
|
are three kinds of checks that can be done:
|
|
<ol>
|
|
<li>You should always have a class that validates a typical file against the schema.
|
|
<p>That's done by having a SchemaTest class in your test-tree package (see e.g. <a href=
|
|
"https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/SchemaTest.java">test/jmri/configurexml/SchemaTest.java</a>
|
|
for an example) that checks all the XML files kept there. If that's in place, just put a
|
|
copy of a (new) typical XML file in the existing "valid" subdirectory.</p>
|
|
|
|
<p>To more extensively check your schema, you can check that it fails XML files that you
|
|
think are not valid. There are lots of ways to be not valid, and you don't need to check
|
|
them all, but if there something specific you want to be sure of, put an example of that
|
|
in the "invalid" subdirectory. Those files are expected to fail for some specific reason.
|
|
You should document that reason via comments in the file itself so your colleagues can
|
|
figure it out later.</p>
|
|
|
|
<p>If there's no "valid" subdirectory, create one and add it to the end of the
|
|
SchemaTest class in that package. If there's no SchemaTest class, create one by
|
|
replicating an existing one, see link above.</p>
|
|
</li>
|
|
|
|
<li>If you're working on configurexml files (panel files), and your new code isn't involved
|
|
in the actual display of panels (e.g. can run as part of headlesstest), you should add a
|
|
test that a sample file can be loaded and stored back successfully. Beyond just checking
|
|
the schema, this also checks that all the load and store plumbing is working. (We run these
|
|
tests headless as part of Jenkins, so please don't add tests that pop screen windows;
|
|
they'll just cause errors).
|
|
<p>That's done by having a LoadAndStoreTest class in your test-tree package (see e.g.
|
|
<a href=
|
|
"https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/LoadAndStoreTest.java">
|
|
test/jmri/configurexml/LoadAndStoreTest.java</a> for an example) that checks all the XML
|
|
files kept there. If that's in place, just put a copy of a (new) typical XML file in the
|
|
existing "load" subdirectory.</p>
|
|
|
|
<p>If there's no "load" subdirectory, create one and add it to the end of the
|
|
LoadAndStoreTest class in that package. If there's no LoadAndStoreTest class, create one
|
|
by replicating an existing one, see link above.</p>
|
|
|
|
<p>When LoadAndStoreTest runs, it loads the files in the load directory one-by-one,
|
|
storing each back out to the "temp" directory within the local preferences directory, and
|
|
then compares the input and output files. Sometimes this load-and-store process results
|
|
in something that's in a different order, or contains more info (e.g. attributes missing
|
|
from the input file are written with default values in the output file). If the
|
|
comparison fails, but the output file is still OK when you manually inspect it, copy that
|
|
output file to the "loadref" directory (create it if needed) within your test package.
|
|
See <a href=
|
|
"https://github.com/JMRI/JMRI/tree/master/java/test/jmri/configurexml/loadref"
|
|
target="_blank">test/jmri/configurexml/loadref</a> for an example. LoadAndStoreTest will
|
|
compare to files it finds here, instead of to the original file in the "load"
|
|
subdirectory.</p>
|
|
|
|
<p>If your changes to the code cause this testing to fail with older versions of the
|
|
file, <em>do not change the older file version.</em> Instead, either put an updated
|
|
output reference in the "loadref" directory, version the schema to allow the older file
|
|
to continue to load, or improve your code. Backward compatibility is important!</p>
|
|
</li>
|
|
|
|
<li>You can also add a custom JUnit test that reads your sample file and makes sure that
|
|
the proper objects have been created, that they have the correct data and state, etc. This
|
|
could be anything from a "load and check that new beans exist in the new manager" to
|
|
something quite more extensive.</li>
|
|
</ol>
|
|
At a minimum, please do the schema checks. They're easy, and will save lots of trouble in the
|
|
future. If your new features don't involve displaying on the screen, adding load-and-store
|
|
checks is also valuable, and isn't so hard.
|
|
<p><em>Note: Do not remove or modify any existing XML file checks. Those keep old versions of
|
|
the files working!</em> If your new code and/or schema breaks the processing of the existing
|
|
files, you need to either fix your code or <a href="#versioning">version the schema</a> to
|
|
allow multiple formats to coexist.</p>
|
|
|
|
<h2 id="devschema">Developing JMRI Schema</h2>
|
|
For some examples of the XML schema structures described here, see the separate <a href=
|
|
"XmlSchemaExamples.shtml">XML Schema Examples page</a>/
|
|
<p>Our preferred organization for XML schema is based on the structure of the underlying
|
|
code: A particular *Xml class is the unit of reuse.</p>
|
|
|
|
<p>Lots of classes descend from jmri.configurexml.XmAdapter: (<a href=
|
|
"https://www.jmri.org/JavaDoc/doc/jmri/configurexml/XmlAdapter.html">see Javadoc</a>)</p>
|
|
|
|
<p>By convention, provide <xsd:annotation><xsd:appinfo> element containing the
|
|
class name that reads/writes the element:</p>
|
|
<code class="block">
|
|
<xs:annotation>
|
|
<xs:documentation>
|
|
Some human readable docs go here
|
|
</xs:documentation>
|
|
<xs:appinfo>
|
|
<jmri:usingclass configurexml="false">jmri.managers.DefaultSignalSystemManager</jmri:usingclass>
|
|
</xs:appinfo>
|
|
</xs:annotation>
|
|
</code>
|
|
<h3>The Venetian Blinds Pattern</h3>
|
|
We are moving toward structuring our XML using the "<a href=
|
|
"https://www.xfront.com/GlobalVersusLocal.html#ThirdDesignCharacteristics">Venetian Blinds
|
|
pattern</a>". In this style, the top level elements that are written by classes have types
|
|
defined for them. Any elements that are within those are defined anonymously, within those
|
|
elements. For an example of this, see the <a href=
|
|
"http://jmri.org/xml/schema/types/sensors.xsd">types/sensors.xsd</a> file, which defines a
|
|
type for the "sensors" element written for SensorManagers. Within that, there is included a
|
|
definition of a "Sensor" element, and a "comment" element within that.
|
|
<p>This limits the number of types, and keeps the schema files roughly aligned with the
|
|
classes that do the reading and writing.</p>
|
|
|
|
<p>There are a few items (elements and attribute groups) that extend across multiple types.
|
|
They are defined in the <a href=
|
|
"http://jmri.org/xml/schema/types/general.xsd">types/general.xsd</a> file.</p>
|
|
|
|
<p>More information on XML schema design patterns is available at the <a
|
|
href="https://www.oracle.com/technical-resources/articles/java/design-patterns.html">Oracle
|
|
Java site</a>.</p>
|
|
|
|
<h3>On Elements vs Attributes</h3>
|
|
When defining how to store new classes or updates to classes, the basic questions are:
|
|
<ul>
|
|
<li>Are we storing data? In that case, it should be stored in an element of its own.
|
|
Comments, speed values, user and system names are all examples of data that should be
|
|
separately stored.</li>
|
|
|
|
<li>Is this a modifier that provides information about the data in the element? In that
|
|
case, it's OK to store the modifier information in an attribute. Color of a label, whether
|
|
a turnout is inverted, which icon of the following list to load are examples of
|
|
modifiers.</li>
|
|
</ul>
|
|
JMRI XML originally leaned heavily to attributes due to limitations in the JDOM library.
|
|
These limitations are long gone, and we're now moving toward using elements in the proper
|
|
way.
|
|
<h3>Available Defined Types</h3>
|
|
The JMRI schema provides a large number of pre-defined data types. These (generally) check
|
|
their content, and will be maintained in the future as valid content changes, so it's best to
|
|
use these where possible instead of defining your own.
|
|
<p>A partial list of the pre-defined types:</p>
|
|
|
|
<dl>
|
|
<dt>systemNameType</dt>
|
|
|
|
<dd>System names, to eventually be tightened up to a real test of validity</dd>
|
|
|
|
<dt>userNameType</dt>
|
|
|
|
<dd>User names, not including the empty name</dd>
|
|
|
|
<dt>nullUserNameType</dt>
|
|
|
|
<dd>User names, with an empty value allowed</dd>
|
|
|
|
<dt>beanNameType</dt>
|
|
|
|
<dd>Either user or system name</dd>
|
|
|
|
<dt>turnoutStateType</dt>
|
|
|
|
<dd>closed, thrown</dd>
|
|
|
|
<dt>signalColorType</dt>
|
|
|
|
<dd>red, yellow, etc</dd>
|
|
|
|
<dt>trueFalseType</dt>
|
|
|
|
<dd>true, false</dd>
|
|
|
|
<dt>yesNoType</dt>
|
|
|
|
<dd>yes, no</dd>
|
|
|
|
<dt>yesNoMaybeType</dt>
|
|
|
|
<dd>yes, no, maybe</dd>
|
|
</dl>
|
|
For others, browse the <a href="http://jmri.org/xml/schema/types/general.xsd">general types
|
|
schema</a>.
|
|
|
|
|
|
<h2>Copyright, Author and Revision Information</h2>
|
|
<p>For various reasons, we are moving to DocBook format for Copyright, Author and Revision
|
|
information in our XML files (instance files).</p>
|
|
<p>Sample XML:</p>
|
|
<code class="block">
|
|
<copyright>
|
|
<year>2009</year>
|
|
<year>2010</year>
|
|
<holder>JMRI</holder>
|
|
</copyright>
|
|
<authorgroup>
|
|
<author>
|
|
<personname>
|
|
<firstname>Sample</firstname>
|
|
<surname>Name</surname>
|
|
</personname>
|
|
<email>name@com.domain</email>
|
|
</author>
|
|
</authorgroup>
|
|
<revhistory>
|
|
<revision>
|
|
<revnumber>1</revnumber>
|
|
<date>2009-12-28</date>
|
|
<dauthorinitials>initials</authorinitials>
|
|
</revision>
|
|
</revhistory>
|
|
</code>
|
|
|
|
<p>For an example, see the
|
|
<a href="/xml/decoders/0NMRA.xml">NMRA standard decoder definition</a>
|
|
in xml/decoders/0NMRA.xml.</p>
|
|
|
|
<p>The exact schema definition for this in
|
|
<a href="/xml/schema/docbook">schema/docbook</a></p>
|
|
|
|
|
|
<h2>External Standards and Future Work</h2>
|
|
The <a href="https://www.oasis-open.org/">OASIS collaboration</a> defines a number of schema
|
|
and schema elements that have become well-known standards. Were possible, we should use those
|
|
<a href="https://www.oasis-open.org/specs/index.php">standard elements</a> to improve
|
|
inter-operability. The first ones of interest are:
|
|
<ul>
|
|
<li>
|
|
<a href="https://docbook.org/">DocBook</a> defines elements for several concepts we use:
|
|
<div class="wide">
|
|
<ul>
|
|
<li>author (http://www.docbook.org/tdg/en/html/author.html)</li>
|
|
|
|
<li>address (http://www.docbook.org/tdg/en/html/address.html)</li>
|
|
|
|
<li>revision history (http://www.docbook.org/tdg/en/html/revhistory.html)</li>
|
|
</ul>
|
|
</div>
|
|
|
|
See
|
|
<div class="wide">
|
|
<ul>
|
|
<li>
|
|
<a href=
|
|
"https://docbook.org/specs/docbook-5.0-spec-cs-01.html">https://docbook.org/specs/docbook-5.0-spec-cs-01.html</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="https://docbook.org/xml/5.0/xsd/">https://docbook.org/xml/5.0/xsd/</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href=
|
|
"https://docbook.org/xml/5.0/xsd/docbook.xsd">https://docbook.org/xml/5.0/xsd/docbook.xsd</a>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
|
|
We have our own DocBook subset that we use, because the full DocBook 5.0 schema takes
|
|
a very long time to parse, and isn't fully consistent with versions of other software
|
|
that we use. We use the normal DocBook 5.0 namespace, so we can later easily convert to a
|
|
more complete schema transparently. Our smaller schema is located at
|
|
<div class="wide">
|
|
<a href=
|
|
"http://jmri.org/xml/schema/docbook/docbook.xsd">http://jmri.org/xml/schema/docbook/docbook.xsd</a>
|
|
</div>
|
|
(our usual schema location). It is <em>only</em> referenced from JMRI schema files, not
|
|
instance files, so that we can later convert with finite work.
|
|
</li>
|
|
|
|
<li>
|
|
<a href="http://ubl.xml.org/wiki/ubl-faq">UBL</a>, though aimed at
|
|
business transactions, defines elements to represent parties (companies, people),
|
|
devices, model numbers, etc.
|
|
</li>
|
|
|
|
<li>
|
|
<a href=
|
|
"https://en.wikipedia.org/wiki/OpenDocument">OpenDocument</a>
|
|
(OODF) defines a set of elements and structures for computations as part of its
|
|
spreadsheet module. (But they provide Relax-NG schema, not W3C XML Schema, so that
|
|
doesn't help so much)
|
|
</li>
|
|
</ul>
|
|
|
|
<p>Learning to use these will require some work, as we can't assume that computers using JMRI
|
|
have internet access, so can't just reference the entire schema as remote entities.</p>
|
|
|
|
<!--#include virtual="/help/en/parts/Footer.shtml" -->
|
|
</div> <!-- closes #mainContent-->
|
|
</div> <!-- closes #mBody-->
|
|
<script src="/js/help.js"></script>
|
|
</body>
|
|
</html>
|