learn xml script

Specifying XML Structures Using Schema OVERVIEW

2009-06-20T00:55:00.000-07:00

Specifying XML Structures Using Schema
OVERVIEW
We've already seen that XML documents can be described using Document Type Definitions, DTDs. DTDs originated with SGML and show those origins all too visibly. XML documents are far more complex and varied than their SGML cousins because XML is used in far more ways than SGML. This creates a problem. While DTDs are perfectly suitable for SGML, where they have been used successfully for many years, they are inappropriate for the newer technology of XML. DTDs cannot be processed by XML-only applications. Developers need to learn two relatively complex languages to use DTDs and they cannot be validated using XML validators. XML has more data types than can be expressed in DTDs, and is generally far richer. Basically DTDs cannot be used to express XML documents.

To remedy this situation, W3C has created a language called XML Schema which can be used to define XML structures. A number of different schema languages exist. In this chapter I will be writing specifically about XML Schema because it is a Recommendation of W3C. I'll be using the terms XML Schema and schema interchangeably – my choice being based purely upon which reads better in a given context. If I wanted to be precise all of the time I would use XML Schema when referring to the language and Recommendation, and schema when referring to a particular document that uses the language.

As I write this, far more tools exist to handle DTDs than XML Schema. This situation is changing rapidly since everyone sees the advantages of using schemas. DTDs are really a technical dead-end, although understanding them will remain important since so many exist. It's likely that when you are using older documents, they'll continue to be described using DTDs. New documents should always be described using XML Schema.[1]

The most important omission in the DTD is the idea of a data type. SGML documents tend to contain mostly plain text. Almost all data in an SGML application can be treated as strings of characters in definitions and applications. XML documents require a far richer set of data types, including strings of characters, numbers, both whole and decimal, and complex types such as dates and times. XML Schema introduces data types which, in turn, leads to more tightly defined XML structures which can be used with current database technologies or in conventional applications written in general-purpose programming languages. Other new, and useful, features in the XML Schema Recommendation include:

a simple pattern matching grammar which might be used, for example, to define the structure of an order code,

defined ordering of subelements so that document structure can be tightly controlled,

selection between different elements so that documents can share a schema without having identical structure.

DTDs are described using their own, unique, syntax. Using them means having to learn, and apply, two sets of syntactic rules in one application. While DTDs are not the most complex documents imaginable, it is vital that developers define them correctly. Equally as important, parsing and manipulating DTDs within applications requires special libraries. XML Schema documents can be handled much more easily because they are fully compliant XML documents in their own right. What does this mean in practice? The tools that you use to develop, parse and manipulate your XML can also be used for your schemas. Developers need learn only one set of rules for schema and document, and both could be created using the same pieces of editing software.

Using XML Schema requires an understanding of namespaces. Schema definitions always use namespaces, so much so that namespaces are one of the cornerstones of schema technology. I've mentioned namespaces before; now is the time to examine them in detail and learn how to use them.

[1]Although pragmatic realities such as organizational politics, historical preferences or the tools you have available may force you to use DTDs.

STRUCTURE XML

2009-06-20T00:42:00.001-07:00

STRUCTURE
The DTD is a series of declarations. Each declaration takes the form:

''

and contains one of four keywords. These are:

ELEMENT which defines a tag,

ATTRIBUTE which defines an attribute of an ELEMENT,

ENTITY which is used to define an ENTITY,

NOTATION which defines a data type.

The easiest way to understand the structure of a DTD is to look at a simplified one. Rather than create a novel structure, I'm going to use part of the DTD for the Business Letter. This is shown in Listing 3.1.

Listing 3.1: Partial DTD for the Business Letter

' ''
' ' ?, country?, code?)>'
''
''
''
''
''
''
' '
''
]>'

PARSING XML FILES

2009-06-20T00:40:00.001-07:00

Applications that manipulate XML need to be able to move through the data structure, finding elements, tags and content. Processing data to extract meaning from it is called parsing in computing. The same term is used to describe the processing of sentences in human languages to extract their meaning. The idea, in both cases, is the same. Few developers choose to write their own XML parsers. Although the rules of the grammar are relatively simple, writing fast and accurate parsers is a difficult task. Most people use a parser written by someone else. Many XML parsers are freely available; the choice of which you use tends to depend upon your system and the language that you are developing in. Two popular choices are MSXML from Microsoft, which can be programmed using C++ or Visual Basic, and Xerces from the Apache Foundation. Xerces comes in Java, C++ and Perl versions and can be used on many different operating systems. Both these parsers can be used directly from the command line or called from within applications.

Once you have installed MSXML on your system, it is automatically available within Internet Explorer. This means that you can, for instance, open XML files in Explorer and view them as tree structures.

As you read through this book, you'll find that XML parsers can do lots of interesting things with your XML. One of the most useful is to check if the XML you have written is correct, and if it adheres to the rules set out in the DTD or schema for that particular document.

2.4.1 Valid or well-formed?
XML documents may be either valid or well-formed. The two terms relate to differing levels of conformance with the XML Recommendation, the DTD, or schema, and the basic structure of the XML. All XML documents must be well-formed. Tags should be paired, elements should be properly nested, the document should have an XML declaration. Entities should be properly formed. Any application which can handle XML will be able to cope with a well-formed document. A valid document takes conformance rather further. To be valid, a DTD or schema should be identified for the XML data. The data must meet the rules set out in that document.

All XML parsers are able to check that a document is well-formed. For some such as MSXML, this is where their capabilities end. Other parsers such as Xerces are able to validate an XML document against a DTD. At the time of writing, Schema support in Xerces is in the alpha stage of development. That means it's far from ready for the big time – but it is being implemented. XML is a new technology, it's evolving rapidly and tool support does tend to lag slightly behind. In the near future, though, the tools will be available to use XML Schema as well as DTDs. It's at that stage that we'll start to see DTDs becoming less popular with developers.

Unparsed Character Data
Most of the content in an XML file will be handled by the parser. Generally elements and entities contain text that has some meaning. The content will not include characters such as < which have special meaning to the parser, and when it does contain them, those characters are usually entered as character entities. Sometimes a document will include large numbers of these characters. In such cases using entities may be impractical. The XML standard allows for this. Your document can include sections of CDATA, unparsed character data. All characters inside a CDATA section are assumed to be content, rather than markup. A section of CDATA is started with the string as shown in Listing 2.6. You'll meet CDATA again in the discussion of DTDs in Chapter 3.

Listing 2.6: CDATA Sections

Chris Bates
Mr. M. Mouse
Hi, how're ya doin'?

characters until the end of the section is reached
]]>

THE XML RULES

2009-06-20T00:29:00.001-07:00

THE XML RULES
Computer languages need to be formally defined in some way. Developers need to know what facilities are available in a language and that those facilities will work in the same way in all implementations. Languages are usually standardized by an international body such as the International Standards Organization, ISO, or the Institute of Electrical and Electronic Engineers, IEEE. For those languages that have defined standards, all compilers or interpreters must adhere to the standard: if a C++ compiler doesn't work according to the ANSI/ISO C++ standard then it really isn't a C++ compiler. Often these standards are minimum requirements which will be available in all products and on all platforms. Manufacturers of compilers are free to extend the language by adding their own proprietary features, although this does mean that the extended version will no longer be standard. Often large or powerful companies try to force their extensions into the standards. This can be extremely beneficial when it leads to improvements – too often standardized languages are developed by committees and become lowest common denominator languages. New extensions may only be available on one platform. If developers wish to write code on a Linux box but later compile and execute it on an Apple Macintosh, they can only do this if no extensions have been used. Problems like this tend to force people either to adhere rigidly to the standard or to work exclusively for a subset of all available platforms. When developing for heterogeneous systems such as the Web, adherence to the standard is clearly the preferred option.

XML requires a common set of rules. In fact, since any Web technology must work on every platform in a plethora of software applications, standardization is even more important than for programming languages. Perhaps surprisingly, XML, like HTML, isn't actually an international standard. It's a Recommendation of the World Wide Web Consortium (W3C). W3C Recommendations have much of the force of international standards but the process of creating them is far more flexible and far faster than standardization.

The current XML Recommendation is Version 1.0 (second edition). It can be viewed online at http://www.w3.org/TR/2000/REC-xml-20001006 or downloaded in a variety of formats. The second edition makes no major changes to the first edition of the Recommendation but does incorporate all of its errata. Most standards documents are necessarily complex. They don't make for an easy read, and the XML Recommendation is no exception. If you want to know just how much thought went into the design of XML, download a copy of the Recommendation and spend a few minutes leafing through it.

2.3.1 XML Tags
XML documents are composed of elements. An element has three parts: a start tag, an end tag and, usually, some content. Elements are arranged in a hierarchical structure, similar to a tree, which reflects both the logical structure of the data and its storage structure. A tag is a pair of angled brackets, <… >, containing the name of the element, and pairs of attributes and values. An end tag is denoted by a slash, /, placed before the text. Here are some XML elements:

The Lord Of The Rings
Helm's Deep
Professor J. R. R. Tolkien

XML elements must obey some simple rules:

An element must have both a start tag and an end tag unless it is an empty element.

Start tags and end tags must form a matched pair.

XML is case-sensitive so that name does not match nAme. You can, though, use both upper and lower-case letters inside your XML markup.

Tag names cannot include whitespace.

Here are those same elements with introduced errors:

The Lord Of The Rings
Helm's Deep
Professor J. R. R. Tolkien

2.3.1.1 Nesting Tags
Even very simple documents have some elements nested inside others. In fact, if your document is going to be XML it has to have a root element which contains the rest of the document. Tags must pair up inside XML so that they are closed in the reverse order to that in which they were opened.

The code in the left column of Table 2.2 is not valid XML since the ordering of the start and end tags has become confused. The correct version is shown on the right side of the same table.

Table 2.2: Nesting Elements Incorrect
Correct

Chris Bates
Mr. M. Mouse

Hi, how're ya doin'?

Chris Bates
Mr. M. Mouse

Hi, how're ya doin'?

2.3.1.2 Empty Tags
Sometimes an element that could contain text happens not to. There may be many reasons for this – the attributes of the element may contain all the necessary information, or the element may be required if the document is to be valid. These empty elements can be represented in two ways:

The Lord Of The Rings

The empty element can be included by placing an end tag immediately after the start tag. More simply, a tag containing the name of the element followed by a slash can be used.

2.3.1.3 Characters in XML
When the XML Recommendation talks about characters, it means characters from the Unicode and ISO 10646 character sets. Until relatively recently most computing applications used a relatively small set of characters, typically the 128 letters of the ASCII character set which could be represented using seven bits. The ASCII character set, defined in ISO/IEC 646, only allowed users to enter those letters typically found in the English language.

In a multilingual world this is clearly an impractical limitation which led to the development of many alternative character sets. Web applications typically use ISO 8859 which uses 8 bits for each character and which defines a number of alphabets. These include the standard Latin alphabet used as default by most Web browsers. Unicode goes further and uses two bytes to represent each character. This means that Unicode includes 65,536 different characters, insufficient for Chinese but suitable for most uses. ISO 20646 extends the Unicode idea by using four bytes for each character, giving approximately 2 billion possible characters. Unicode is implemented as the default encoding in Microsoft Windows and the Java programming language, among others. But it clearly needs extending to access those extra characters, and has been. Version 2.1 of Unicode includes some facilities that give access to the ISO 10646 character set.

Using ISO 10646 to represent ASCII data is highly inefficient – effectively three bytes of memory are wasted. Even though computer memory and storage are extremely cheap today, such inefficiency is expensive if an application is handling gigabytes of data. Therefore applications use encoding schemes to store data more efficiently. Applications that process XML must support two of these: UTF-8 and UTF-16. UTF-8, for instance, uses a single byte for ASCII data and two to six bytes for extended characters.

Note XML applications support extended character sets. These allow up to 2 billion different characters. When you develop using XML you can use any language and character set that you need to in your applications. You are not restricted to the English language or to the set of languages supported on a particular operating system.

It's worth noting that everything in an XML document that is not markup is considered to be character data. Markup[4] consists of:

start tag,

end tag,

empty tag,

entity reference,

character reference,

comments,

delimiters for CDATA sections,

document type declarations,

processing instructions,

XML declarations,

text declarations.

The final, important thing about characters is that some of them have special meaning or cannot be easily represented in your source text using a conventional keyboard. Most of the characters in ISO 10646 clearly fall into this category. Some mechanism is therefore required to permit the full range of characters to be included in documents. This is done through character references. To demonstrate the use of character references, I'll look at those characters that can have special meaning inside markup. Characters such as <, >, ', " are used as part of the markup of the document. If they're encountered by the parser inside an XML file, it assumes that they are control characters which have special meaning to it, and it then acts accordingly. The obvious example of this behavior is found in handling attributes. The following two examples would be illegal in XML:

In each case, the parser will assume that the content of the src attribute starts at the first apostrophe or set of quotation marks, and stops at the second. Attribute content following this point cannot be parsed since it is not valid XML.

What happens when the file should legitimately contain < as part of its character data? The appropriate character reference is entered instead.[5] Table 2.3 shows the references which must be entered in an XML document if you want a particular character. Here's the previous example reworked to be valid XML:

The Disadvantages of XML Searching

2009-06-17T01:07:00.001-07:00

The Disadvantages of XML Searching
Despite the optimistic view of many people in the XML community, the XML searching problem is complicated, from both technical and business perspectives. In some situations, XML-based contextual searching can be a major advantage; in others, it can be an unnecessary cost; in yet others, it can make the search engine's results worse. This section introduces some of the problems with the very idea of XML-based searching.

XML searching may be too complex for most users.

Documents on the Web can use deceptive markup to raise their ranking in a search.

XML documents are generally not interoperable in the same search environment, because of all the different, incompatible vocabularies.

6.2.1. Usability
Tim Bray, cofounder of Open Text, which ran an early Web search engine, and coauthor of the original XML specification [XML], wrote the following passage in a Web log (http://www.tbray.org/ongoing/When/200x/2003/06/17/SearchUsers):

Nobody Uses Advanced Search...

Every search engine has an "advanced search" screen, and nobody (quantitatively, less than 0.5% of users) ever goes there. This drove us nuts back at Open Text, because our engine was very structurally savvy and could do compound/boolean queries that look like what today we'd call XPath. But nobody used it.

What most people want is to have a nice simple field into which they will type on average 1.3 words and hit Enter, and have the result come back to them. So anyone who's building search needs to focus almost all their energy on doing an as-good-as-possible job given those 1.3 words and no other inputs.

This observation does not bode well for XML searching. If users are unwilling to use even relatively simple full-text techniques, such as Boolean or proximity searches, how much hope is there that they will be willing to formulate the complex queries that can take advantage of XML markup? Fortunately for the future of XML searching, Bray does go on to qualify that observation:

...Except the People who Do

Of course, the people who do use Advanced Search are your most fanatical users, the professional librarians, spooks, and private investigators. And the ones who will do what it takes to find out everything about research on the rare disease their child just got diagnosed with. These people tend to be loud-mouthed and aggressive and will get in your face if you don't have advanced search or it's not real good.

Presumably, these same kinds of people would be the ones using XML context in their searches. Others of Bray's "fanatical users" might be academics preparing papers, journalists researching news stories, and software agents collecting and amalgamating information for politicians and managers. This last example, in fact, may point to the real potential users of XML searching: not people but software. People other than governors and CEOs need to make decisions in their own lives, from changing jobs to buying new clothes, and software agents that find information for peoplesay, for price comparisoncould benefit greatly from the extra information provided by XML markup, assuming, of course, that vendors were willing to encode their pricing information in a standard format and accept the transparency that comes with that.

And that leads to another usability problem: XML searching requires people or software to know a lot about the structure of the documents they're searching. If all XML documents shared a single, global vocabulary, searching would be relatively straightforward, at least for power users: Every price would appear inside a price element, every bar code would appear inside a upc element, every person's name would appear inside a person element, and so on. This is unlikely ever to happen, for two reasons:

No single, accepted authority could impose a common vocabulary on all users.

XML documents can encode a potentially infinite variety of information, so a common vocabulary would always be incomplete.

Some XML-related specifications are designed to work around these problems, at least partlysee Section 6.3.3 for more informationbut in reality, if a large amount of XML markup did appear on the Web today, generalized XML searching would be almost useless, given the hundreds of incompatible XML-based vocabularies. The best people can hope for is specialized searching inside repositories or across Web collections in which all XML documents share a common type: Conceptually, this is the equivalent of a site search engine rather than a Web search engine, and it falls far short of a revolution in Web searching. Even then, searching will be more complex than the most difficult "advanced search" page currently available on full-text Web search engines. Either users will have to become experts in XML structure, or they will have to limit themselves to a few precooked searches, such as "Search for a person" or "Search for a part number," through Web sites that can construct an XML query for them.

In the end, XML searching may be useful for specific project applications. But usability issues alone make it seem unlikely that XML will ever cause the social revolution in Web searching that some supporters hoped for when the specification first appeared.

The Advantages of XML Searching

2009-06-17T01:03:00.000-07:00

The Advantages of XML Searching
XML markup makes searching smarter by adding contextual information and makes it possible to correlate information from more than one document. Any serious Internet user is familiar with searches that do not work: The words are too common or have too many different meanings for any search engine to return useful results, or perhaps the information is spread among several pages. Solving these problems was one of the initial goals of XML's creators.

6.1.1. Context
Much of the time, full-text search engines do a good job, but they sometimes fall flat. Consider, for example, the difference between Bush the U.S. president and bush the shrub, or Washington the U.S. state and Washington the U.S. city. If you were trying to find information on bush pilots flying out of Washington State, you might try the search "bush pilot washington." In late 2003, Google's first ten results were as follows:

A site selling a book about a Canadian bush pilot (no connection to Washington)

Two newspaper stories about the U.S. Navy naming an aircraft carrier after President Bush Sr.

A 2002 USA Today story about a small plane violating airspace near President Bush Jr.

A 2000 Washington Post story about President Bush Jr.'s service in the Texas Air National Guard

A 2001 Pravda story critical of President Bush after the midair collision between a Chinese fighter jet and a U.S. surveillance plane

The Amazon.com page for a biography of an Alaskan bush pilot, published by University of Washington Press

Two pages from FlyRod & Reel magazine: one stating that it has no listings for Washington and another listing angling retailers in the state

A 2003 news story from the Washington Times about a U.S. Navy pilot being held by the Iraqi government

I choose Google for this example precisely because it is a very good full-text search engine: It infers the relevance of information on a Web page not only from the text on the page but also from the other pages that link to it, the pages that link to those pages, and so on. With this difficult example, the first slightly relevant result is the twenty-third, which mentions a bush pilot who did fly once in Washington State; after that, the matches revert mainly to politics.

An experienced search-engine user could work around the problem by adding more words to the query. For example, bush pilots in Washington State have to deal with a lot of mountains, and the query string "bush pilot washington mountains" returns fewer political hits. Even better, a search string that contains specific aircraft types used in bush plane flying, such as "bush pilot washington cessna 180" returns almost all relevant matches. Most of the population is not that adept with search-engine query strings, however, and would likely give up on the whole thing; furthermore, these more specific queries would miss pages that do not happen to mention mountains or Cessna 180 airplanes.

Although homographs, such as Bush and bush can make full-text searches difficult, an even trickier problem comes when the search results depend more on context than on the individual words. For example, consider trying to find Web pages that discuss the history of the word sex, without hitting thousands of pornography sites. Unless the decades-old dream of full machine artificial intelligence finally shakes off its dust and comes true, these are searches that will continue to flummox traditional full-text search engines.

As long as artificial intelligence remains a distant dream, we need to concentrate on getting plain old human intelligence into our XML documents, and that is precisely what XML-based markup languages do. The following News Industry Text Format [NITF] fragment shows how news providers can tag articles to avoid confusing search engines.

Today in Seattle,
Washington, President
Bush opened a new museum.

The markup represents added human intelligence from the news reporter or editor: Washington represents a state, not a city, and Bush represents a person, not a shrub. The XML document contains not only the news story itself but also what the author knew about the news story. As this information survives all the way to the final document, search engines require no special artificial intelligence to use it.

Similarly, several markup languages, including DocBook [DOCBOOK] and the Text Encoding Initiative [TEI] define markup for talking about the history of words. Following is a DocBook example:

The word sex has gradually come
to mean not only gender, but the physical act of procreation, and,
eventually, all physically-intimate acts.

The wordasword element makes it clear that this paragraph discusses the word sex rather than the act: A search engine could easily pick out this contextual information to return exactly the pages the user wanted.

XML markup is information that a document's creator knew but could not put in the main text. Because this information is available, search tools could potentially use it to return far more accurate results.

6.1.2. Correlation
In addition to basic context, XML markup also makes it possible to correlate information, matching instances of the same thing described in different ways and converting among different representations. To start with a simple example, consider only a few of the many ways documents might refer to British Prime Minister Tony Blair:

Tony Blair

Prime Minister Blair

the British Prime Minister

the prime minister

the P.M.

Mr. Blair

Blair

Given this variety, searching for information about Prime Minister Blair is difficult, and the work is made even worse by the fact that many of these phrases can apply to other people: for example, in a different context, "the prime minister" could refer to the prime minister of Australia, Canada, India, or many other countries.

How is it possible to define a Web in which people can easily search for information about Prime Minister Blair no matter how he is described? One possibility is always to normalize the name when it appears; unfortunately, if people are forced always to write "British Prime Minister Tony Blair," text will become awkward and unnatural, and even that might not be sufficient if another person named "Tony Blair" became British prime minister in the future.[1]

[1] This risk is not far fetched: Consider the phrase "U.S. President George Bush."

Using XML markup, however, it is a simple matter to attach a unique identifier to every location that mentions Prime Minister Blair. As long as the identifier is well known, search engines can look for it rather than for the text it contains, as shown by the following markup fragments:

Tony Blair

Prime Minister Blair

the British Prime Minister

the Prime Minister

the P.M.

Mr. Blair

Blair

If in the future, another person shared the same name and title, that person would have a different identifier, so search engines would not return false hits.

Most things in the worldpeople, concepts, historical periods, and so ondo not yet have standard, universally accepted identifiers, so this is more than a markup problem. However, many identification schemes do exist, such as stock market symbols; publication identifiers, such as ISBNs; social security numbers, phone numbers, postal codes, and country, language, and currency codes. The following example shows the use of a stock ticker symbol for identifying a company:

Today, Microsoft
announced a new software strategy.

Even limiting searching to current widely accepted identifiers, XML markup can make it significantly easier to correlate information described in various ways.

Now, consider a more difficult problem than simple identification: a search for world government spending programs that cost more than USD 1 billion. This kind of a search is far beyond the capabilities of current full-text search engines, but XML markup can add hints to help future search engines do the work. The following example uses News Industry Text Format [NITF] markup once again:

Today, Congress approved an additional 3
billion in education spending.

The money element makes it clear that this article is referring to "3 billion" in currency, and the unit attribute indicates that the currency is U.S. dollars, using a code from the ISO 4217 standard for identifying world currencies [ISO-4217]. A search engine would still use full-text searching algorithms to determine that the article dealt with government spending, but then the tagging would help it determine the amount. Even more interestingly, the money element adds enough intelligence that the search engine could return correct results for pages using entirely different currencies; at the time of writing, the money in the next example is less than USD 1 billion, so it should not return a hit:

The Canadian federal government committed an additional unit="CAD">1.1 billion in health-care spending.

Automatic currency conversion during searchingbased on intelligent tagging like thiscould be especially useful for financial institutions and others mining large international document repositories for information. Many other types of conversion and substitution are also possible with markup, including dates and times, language conversion and recognition of synonyms, subsets, and supersets. (For example, a search for information about New England should return pages that mention Vermont.)

Obviously, a lot of infrastructure is required before search engines can work like this. But it does provide an intriguing view of a future that markup might help to enable, where people and programs can search for and find the precise information they need, relying on intelligence encoded in XML markup.

Network Resources

2009-06-16T18:24:00.001-07:00

Network Resources
People have concerns about how XML networking will perform once it is in widespread use, but XML networking brings big performance advantages in one area: Because it can contain arbitrarily complex structure, an XML document can batch up information and reduce the number of network transactions required. For example, a hypothetical accounting server with an XML networking interface might allow a client to request information about multiple accounts with a single XML document sent over HTTP, as in Listing 5-1.

Listing 5-1. XML Batch Request

The server could respond with all the information also in a single XML document, as in Listing 5-2.

Listing 5-2. XML Batch Response

account="assets.current.accounts-receivable">144298.00
account="assets.current.petty-cash">2119.16
account="liabilities.current.accounts-payable">89376.78
account="income.professional-services">2033945.39

In a non-XML system, the same information could require many request/response exchanges, and the extra latency would create major slowdowns for an application. More advanced distributed-computing protocols have mechanisms for batching information on the flycalled marshalingbut they are complex to implement and have proved less than impressive in the field. Perhaps XML's simple approach will turn out to be more robust and effective.

Advantages of XML Networking

2009-06-16T18:23:00.000-07:00

Advantages of XML Networking
XML and HTTP are a great combination: XML brings full internationalization, transparency, and extensibility; HTTP brings compatibility with the Web infrastructure and a high level of hardware and software support. Combining XML and HTTP makes it possible to have stateless, lightweight, decentralized, and distributed computing, the same way that HTML and HTTP made it possible to build a stateless, lightweight, decentralized, and distributed document base.

5.1.1. Internationalization
Many of the basic Internet protocols date back to the days when the Internet and its predecessor were a phenomenon primarily of the English-speaking parts of North America. International support for non-English characters and different writing directions is often either unavailable or tacked on.

XML, on the other hand, was designed from the ground up for full internationalization (I18N). Because XML not only allows but also requires full Unicode support [UNICODE], an XML document can appear in any combination of languages. For example, a single document might have an Urdu element name, a Japanese attribute name, and an Arabic attribute value.

5.1.2. Transparency
The advantage of transparency applies as much to XML networking as to other areas of XML use. Because XML is plaintext, it is easy to test and debug applications that use it. This advantage becomes important for networking because bits on a wire can be particularly difficult to analyze. For a detailed discussion, see Section 7.1.2.

5.1.3. Extensibility
Extensibility has long been an important part of Internet and pre-Internet networking protocols and formats. For example, implementers often add new features by using a distinctive reserved prefix, such as "x-".[3] Typically, applications that do not understand the extensions simply ignore them.

[3] This usage will be familiar to many readers from the MIME (Multimedia Internet Mail Extensions) types, such as "video/x-msdownload," and e-mail headers [RFC 822], such as "X-Mailer."

This kind of extensibility is much simpler with high-level text-based protocols, such as HyperText Transfer Protocol [HTTP], than with low-level binary protocols using fixed-length fields, such as the Transmission Control Protocol [TCP] or the Internet Protocol [IP]. Even there, though, the protocols are designed to allow a certain amount of extensibility: Both the IP and TCP have room for new option types to be defined in the future.

The Internet culture of easy extensibility can be traced back in a large part to John Postel's 1981 Robustness Principle, included in RFC 793 [TCP], Section 2.10: "TCP implementations will follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others." People quickly extended this principle to other areas of Internet work and often emphasized the second part"be liberal in what you accept from others"over the first. In the 1990s, the idea of extensibility jumped from Internet protocols to Web formats: The Hypertext Markup Language (HTML) required Web browsers to ignore any markup they did not understand rather than only specially flagged markup. This approach led to major headaches for Web designers and became ammunition for both sides during the browser wars of the 1990s, but it also allowed experimentation and innovation, moving HTML from a simple specification for sharing research papers in 1990 to the rich visual medium that caught the world's attention and enabled global online communication by 1995.

Ironically, despite the fact that the X in XML stands for extensible, XML itself (Section 1.2) deliberately violates Postel's robustness principle in its definition of the term fatal error:

After encountering a fatal error, the processor MAY continue processing the data to search for further errors and MAY report such errors to the application. In order to support correction of errors, the processor MAY make unprocessed data from the document (with intermingled character data and markup) available to the application. Once a fatal error is detected, however, the processor MUST NOT continue normal processing (i.e., it MUST NOT continue to pass character data and information about the document's logical structure to the application in the normal way).

In other words, an XML parser is not allowed to try to recover from, say, an omitted end tag and continue normally: It has to be conservative rather than liberal in what it accepts. In fact, XML is extensible, but its extensibility comes at a higher level.

The design of XML makes it easy to add new element and attribute types to any format, as in HTML, but the XML Namespaces specification [NAMESPACES] goes further by making it possible to avoid naming collisions between extensions from different sources. The ability to add new fields and complex structures to protocols without breaking existing software is one of XML networking's greatest strengths, even if it does come to Postel's principle by an indirect route.

5.1.4. Compatibility
XML networking provides compatibility in two ways.

XML itself ensures that information is compatible with any application, operating system, or hardware: There are no byte-order issues, line-end problems, or any of the other small snags that can create big bugs and portability headaches.

HTTP, the most popular transport in XML networking, ensures that XML-based network protocols are compatible with existing Web hardware and software, including firewalls, which typically allow HTTP traffic through with no extra configuration required.

Only the end pointsthe computers sending and receiving XML networking messagesrequire any special software. Everything elsefirewalls, routers, switches, caches, multiplexers, and all the other bells and whistles of the Internetwill simply work. (Note that many people consider the firewall compatibility to be a security flaw, as discussed in Section 5.2.2).

Common Data Styles

2009-06-15T23:36:00.000-07:00

Common Data Styles
What does XML data look like? Three popular ways of modeling machine-readable information are

The tabular style, familiar from spreadsheets

The graph style, familiar from relational databases and the World Wide Web

The hierarchical style, familiar from computer file systems

XML can use any of the three styles, but it is optimized for the hierarchical (or tree) style. Database specialists are particularly fond of the graph style, as it simplifies database import and export and is most suited for fully normalized data. Developers of desktop applications often prefer the tabular style, especially for initialization files, as it is easy to set up; some use simple lists, which are equivalent to a single-column table. This section looks at the strengths and weaknesses of all three styles.

4.3.1. The Tabular Style
At their best, tables are a space-efficient way of representing structured information for machines or for humans. A column in a table represents a labeled fieldone piece of information about a thingwhereas each row represents all the fields for the same thing, as in Table 4-1.

Table 4-1. Simple Data Table Employee
ID
Title
Unit
Specialization
Years

Janet Mulville
e000234
Senior Consultant
Database
Data modeling
9

Ahmed Said
e000345
Project Manager
Systems
System integration
11

Julie Fujikawa
e009122
Intern
Systems
System integration
1

The arrival of the consumer spreadsheet with VisiCalc in 1979 gave end users their first chance to work with structured information directly, and its tabular format proved easy to understand and work with. Twenty-five years later, the spreadsheet is still at the center of many business applications; in fact, much business software is nothing more than customizations built on top of Microsoft's Excel spreadsheet.

The tabular style is especially effective when every data object has roughly the same kinds and quantities of information. But this style becomes awkward quickly when different objects have different kinds of information or when information can repeat itself. In Table 4-1, it could be that the more senior employees have more than one specialization; for example, Janet Mulville might also have 5 years of experience with C++ programming and 3 years of experience with application server design. How would that fit into this table? The initial solution is to start repeating the Specialization and Years columns, as in Table 4-2.

Table 4-2. Messy Data Table Employee
ID
Title
Unit
Spec-1
Years-1
Spec-2
Years-2
Spec-3
Years-3

Janet Mulville
e000234
Senior Consultant
Database
Data modeling
9
C++
6
Appl. servers
3

Ahmed Said
e000345
Project Manager
Systems
System integration
11

Julie Fujikawa
e009122
Intern
Systems
System integration
1

Because the structure is a table, Ahmed Said and Julie Fujikawa have blank fields for additional specializations hat they do not have. If a new employee arrives with five specializations, the table will add columns for all the rows, so again, all other users will have more unnecessary blank fields. When Julie is temporarily assigned to the Database unit for 50 percent of her time, an additional Unit column will be needed as well, and so on, until it becomes extremely difficult for a person or a machine to make much sense of the information.

XML has techniques to make tabular information a little more readable and efficient, however. As a starting point, consider Listing 4-1, which is a direct rendition into XML of the information in Table 4-2.

Listing 4-1. Raw Table in XML

Employee Information

Employee
ID
Title
Unit
Spec-1
Years-1
Spec-2
Years-2
Spec-3
Years-3

Janet Mulville
e000234
Senior Consultant
Database
Data modeling
9
C++
6
Appl. servers
3

Ahmed Said
e000345
Project Manager
Systems
System integration
11

Julie Fujikawa
e009122
Intern
Systems
System integration
1

XML can improve on that representation in several ways while still staying in the spirit of the tabular approach. First, because XML already labels every element, a header row labeling the columns is not needed; instead, the column labels can appear as element names. Then, once each entry is labeled, the blank ones need not be included.[2] The result is the much more readable and space-efficient XML table in Listing 4-2, although it is still far from ideal XML markup.

[2] In fact, the blank entries in Listing 4-1 could have been left out anyway, as they are all trailing; however, any nonblank entries at the ends of the rows would have had to be included.

Listing 4-2. Readable Table in XML

Janet Mulville
e000234
Senior Consultant
Database

Data modeling

C++

Appl. servers

Ahmed Said
e000345
Project Manager
Systems

System integration

Julie Fujikawa
e009122
Intern
Systems

System integration

The Disadvantages of XML Data

2009-06-15T23:34:00.002-07:00

The Disadvantages of XML Data
Besides XML's advantages for data interchange, there are nonetheless some disadvantages. The first thing new users notice is that you cannot simply load arbitrary XML into an application or database the way that you can load a spreadsheet. That level of interoperability requires higher-level specifications, and those have been slow to gain acceptance. Without those specifications, XML processing still, unfortunately, requires a lot of custom coding and a not insignificant amount of processing time.

4.2.1. Interoperability
XML alone is not sufficient for sharing data: higher-level specifications are required to define what information should appear in the XML and how it should be structured. It is difficult to write a general-purpose, off-the-shelf tool to import XML into a database, but people can provide off-the-shelf tools to import NewsML [NEWSML], Extensible Business Reporting Language [XBRL], or many other such formats.

Higher-level XML data formats, however, are difficult and expensive to create, and there are too many of them. These problems might seem to contradict each other. In fact, people propose many higher-level XML formats but implement few of them, as discussed in Section 1.4.4: The proliferation of proposals causes confusion without providing solutions that people can build around.

Because there are rarely middle layers between low-level XML and high-level data formats, most XML data specifications are stovepipes, built from low-level XML on up. Although most of them share common concepts, such as identifiers, references, entities, attributes, and relationships, they all implement them differently, destroying the chance to use shared codeand shared user knowledgeto lower costs. As a result, there are few useful off-the-shelf components for dealing with data and no significant economies of scale or network effects.

XML networking specifications (see Chapter 5) are helping this problem somewhat by providing general-purpose data formats for the networking payloads. These data formats may not be optimal for exporting large amounts of information from databases or publishing data on the Web, but they will at least provide a starting point.

4.2.2. Abstraction
Abstraction can also be a problem. If one organization exports its data to XML, a second organization probably will be unable to read that data into its system immediately but will have to write transformation and import scripts to change the XML to a format that its system can use. That is the major problem with XML for data sharing: Its abstraction gives it many advantages, but it also means more work. You cannot simply save or load any arbitrary XML data the way that you can save or load a spreadsheet file.

4.2.3. Resources
XML has a reputation, partly deserved and partly undeserved, as a resource hog. XML is plaintext, and XML data is usually in an abstract, logical format, so reading XML into an application often requires two steps: parsing the XML from plaintext and transforming the parsed XML so that the application, such as a database, can use it.

Parsing and transformation take time, as does generating XML for output. Chapter 8 discusses how to avoid some of the worst inefficiencies in XML processing, but in an extremely high-speed environment, the time and processor use of XML might still be unacceptable.

The Advantages of XML Data

2009-06-15T23:34:00.001-07:00

The Advantages of XML Data
In many of the other chapters in this book, the lists of the advantages and disadvantages of XML are close to even; for sharing data, however, XML is a much more obvious choice. Of course, there are disadvantages that will rule out XML for certain projects, but in most cases, XML's advantages far outweigh the disadvantages, because there really is no general, portable alternative to XML for exchanging data with complex structure.

Behind the scenes, many companies are already using XML for internal data sharing, but despite its value, XML data itself is no longer a hot topic. Although the vendors continue to add support for XML in databases and other applications, the spotlight has moved on to XML networking (Chapter 5) and, especially, Web Services. However, even XML networking is still, at heart, using XML for data: Nearly all the networking specifications are simply combinations of XML data with a network transport layer, such as HyperText Transfer Protocol [HTTP].

XML brings clear and persuasive advantages for data exchange in five areas:

Platform and storage independence

Self-documentation

Reusability

Verification

Archiving and auditing

4.1.1. Platform and Storage Independence
In the past, save files for PC-based databases, such as FoxPro, sometimes became a de facto interchange format, just as Microsoft Excel spreadsheet files are a de facto standard for exchanging simple tabular data. Spreadsheets and other data applications also support some more portable formats, such as comma-delimited text or the sequels to the old Data Interchange Format (DIF). None of the portable alternatives, however, is particularly good at exchanging complex structured data.

XML can model arbitrarily complex data structures and is not tied to a specific vendor or product. With XML and Unicode, there are no byte-order, line-end, or character-encoding problems moving from machine to machine or even from country to country or culture to culture. All major and many minor programming and scripting languages have good support for XML, as do most databases.

XML's independence, however, goes far beyond simply avoiding platform incompatibilities. As discussed in Section 4.3, XML can represent data in its logical format rather than its physical storage format. Whereas a relational database might divide information into a series of separate, cross-linked tables, using attributes for order, XML can pull everything into a single, ordered, logical document.

XML's abstraction encourages loosely coupled data interchange. It does not matter whether the data provider and the data consumer arrange their databases differently, with different tables linked in different ways, because that physical level is not fossilized in XML as it would be in a series of comma-delimited table dumps. This fact, perhaps, matters more than all the other portability issues.

4.1.2. Self-Documentation
For humans, although not for computers, XML documents are at least partly self-describing, as discussed at length in Section 7.1.2. It might seem inconsistent, at first, to suggest that human readability is an advantage, when this chapter has been emphasizing that XML data is intended for machines. However, in practice, people still have to implement the programs or scripts to work with XML. Because XML documents normally have human-readable names for elements and attributes, developers will find it easier to produce, process, transform, and, most important, debug XML input and output. Human-readable names also make it possible to automate the creation of forms and other human interfaces for data entry.

4.1.3. Reusability
This advantage is related to storage independence (see Section 4.1.1). Well-designed XML data is almost always loosely coupled; it does not slavishly follow the structure of a spreadsheet, a set of database tables, or any other source but rather reorders the information into an abstract, logical presentation. Therefore, XML data is not limited to a single purpose. Data exported from a database can be read into another database, of course, but can also be edited, displayed, analyzed for statistical patterns, transformed, searched and queried, or even published to a Web page or print. The original data provider does not have to be able to anticipate all the uses people might have for XML data. Once the information is in XML format, XML tools and utilities will simply be able to work with it any way the recipient desires.

4.1.4. Verification
Developers often spend a lot of their time developing and testing code for data validation. With XML data, it is possible to do basic structural validation and, in some cases, data-type validation (see Section 4.5.3), using schema languages with off-the-shelf software tools and libraries. Writing an XML schema or DTD is a specialized skill, but it can sometimes be much easier and more robust than writing custom code to perform the same structural checks.

XML data allows verification in other ways as well. Because XML is plaintext, people can easily examine it for problems that automated tests might have missed, the same way that people can examine text-based Internet protocols, such as HTTP, SMTP, or Post Office Protocol v. 3 (POP3), manually for verification and debugging. And, like any text file, XML is relatively simple to sign digitally.

4.1.5. Archiving and Auditing
XML data consists of files, not bits on the wirelike a networking protocolor application-specific binary datalike a spreadsheet file. Because XML is not tied to an application that may no longer exist in a few years, XML data will still be useful in an archive 50 or 100 years from now; because it can be saved to disk, every transaction can be archived and audited, something that is difficult with protocols.

These points are especially important for companies or other organizations that face complex legal and reporting requirements. A saved XML transaction, like a saved e-mail, provides an ongoing record of what was happening in a company at a specific time. In case of a security breach, people can go back and review not only logs but also past transactions to track down the problem.

XML Content Management

2009-06-15T23:26:00.003-07:00

Content Management
This chapter has already emphasized that the biggest real benefits for XML publishing come with large, highly structured technical publications, such as regulatory documents, dictionaries, legislation, or maintenance manuals for complex equipment, such as aircraft or weapons systems. Such documents typically have many authors and a formal and often complex editorial review process. These living documents are under continual revision for their entire lives. With a dictionary, for example, lexicographers write individual entries, which will then pass through several stages of editing and approval before being added to the dictionary proper. As corrections and new slipsusage examplescome in, lexicographers will revise an entry and start the entire review process again. At some major projects, the lifespan of the documents is longer than that of the authors: For the Oxford English Dictionary, for example, this process has continued nonstop for almost a century and a half. There's never any concept of a finished book; each edition is only a snapshot of a never-ending work in process. Exactly the same process applies to the aircraft maintenance manual for a large airliner, for example, except that the snapshots are published every few months rather than every few decades.

Obviously, this kind of publishing was possible before computers, much less XML, but it was labor- and paper-file-intensive: precisely the kinds of operations that can benefit from some kind of content-management system. Many computer programmers are already familiar with source-code-management systems, such as CVS or Visual SourceSafe. Content-management systems for documentation are very similar: They allow authors to check objects, such as documents or pictures, into or out of a central repository, which tracks revisions and often also allows searching, indexing, and even final document assembly. A documentation content-management system may also have a workflow component attached to it, so that it can both track the status of each object through the editorial process and manage the process by sending files to the people who need to approve them.

None of these features is unique to XML, but they are especially likely to be required in a large, multiauthor XML documentation project. Some systems go further: Instead of managing each XML document as a single object, the same as a picture or word-processing file, they allow users to check out part of an XML documentsay, the third task in the second chapterwhile other users work on other parts of the same XML document. Typically, the system parses the XML document and converts it into a series of entries in a specialized database, then reconstructs it as XML when needed.

For a large project with multiple authors, a content-management system is often a requirement: The only question is whether to use a traditional system or a special XML-aware one. It is best to approach this problem backward and start with the non-XML solution. Assume that a company is creating a large amount of technical documentation in XML, using a team of authors and editors. At any time, authors will have lists of items assigned to them for writing or revision, and editors will have lists of items assigned to them for editing and approval. The technical documentation itself consists mostly of independent tasks. Listing 3-9 shows a fragment of a simplified DTD.

Listing 3-9. Top Level of a DTD

Most of the authoring items moving through the workflow chain are tasks with their associated graphics; the editors will assign tasks to individual authors, who will then pass the tasks back up to the editors for approval. When a task requires revision, the cycle repeats itself.

Would an XML-aware CMS bring much benefit to a project like this? No. Authors do not need to check out a single step or title or an entire chapter, only tasks. The CMS can handle each task as a separate XML document without needing to know anything about the task's internal markup structure.

To require an XML-aware content-management system, there would have to be no standard unit for authoring, editing, or workflow; the author would need to be able to check out and lock anything from a single list item to the entire document. The author of this book has not yet seen such a requirement in the real world. Every big multiauthor project has standard units of work, whether they are tasks, dictionary entries, or newspaper articles.

Content-management systems also typically offer searching, indexing, and packaging. A full-text search is sometimes sufficient, but because XML markup can provide much more detailed content, a good case can sometimes be made for making the search XML-aware. For more on this point, see Chapter 6.

Script Client-Side XML

2009-06-15T23:26:00.001-07:00

Client-Side XML
Single-source publishing allows people to write documents in XML and then convert them to multiple formats, such as PDF or PostScript for print and HTML for the Web. However, the World Wide Web Consortium [W3C] did not design XML simply to be a source for other formats; many people intended XML itself to be a delivery format for the Web, replacing HTML. Beyond XML is a family of W3C specifications, such as XML Linking Language [XLINK], XML Pointer Language [XPOINTER], cascading style sheets [CSS], and XSL Transformations [XSLT], all designed to help browsers display XML documents directly.

In many ways, this work has succeeded. Both Microsoft's Internet Explorer and the variants of Mozilla contain extensive XML support: They can display a raw XML document without a style sheet, or they can use style sheets to make an XML document in a browser window indistinguishable from HTML for the casual user. Behind the scenes, however, client-side scripts can perform sophisticated tricks based on the XML source, providing a richer browsing experience for users. Although the XML support in the two browsers has incompatibilities, and some specifications, such as XLink, are barely supported, if at all, there is also a surprising amount of compatibility, especially in comparison with the bitter browser wars of the late 1990s.

Both the standards and the software are in place for delivering XML directly to users without going through a middle format, such as HTML or PDF, and most users not only have the software installed on their system but use it daily for viewing HTML pages. Nevertheless, XML on the Web is almost nonexistent, as HTML is good enough for almost everything that anyone wants to do on the Web, and the extra benefits of delivering XML directly do not make up for the costs of new training, new authoring tools, and incompatibility with the minority of users who still have old, pre-XML browsers installed.

Although client-side XML has failed to take off the way that HTML did in the mid-1990s, the tool availability is still a benefit for XML documentation projects. Instead of purchasing and installing special XML viewers, authors can preview formatted versions of XML documents directly in their familiar Web browsers. On intranets and other areas where browser versions are more uniform, client-side XML viewing is a useful ability, even if it has failed to become a social phenomenon on the Web.

3.4.2. Reuse
Documentation always fits awkwardly into technology projects. Technical writers complain about programmers who make changes at the last minute, forcing the writers to redo most of the documentation; programmers, in turn, complain about writers who seem unable to write most of their documentation until just before a release deadline.

In fact, programmers have long created and used systems to help them write and maintain documentation aimed at other programmers. In the 1980s, Donald Knuth promoted literate programming, whereby the source code for a program, such as TeX, was embedded inside its own documentation and extracted automatically for compilation; anyone editing the documentation would edit the code at the same time, and vice versa, ensuring that the documentation and the code remained synchronized. In the mid-1990s, as the Java programming language increased in popularity, the opposite approach became common: Programmers embedded documentation in the source code as specially formatted comments and extracted it automatically for publication. (Earlier programming languages, such as Emacs LISP, had already used this approach on a smaller scale.) The JavaDoc system proved extremely effective for generating programmer's API documentation and has been much imitated for other programming languages.

JavaDoc and literate programming work for programmer's documentation because the documentation nearly always follows the structure of the source code. When the programmer deletes a class or a method, the documentation disappears with it; when adding or modifying a class, the programmer simply needs to modify the documentation that is right there on the screen with it.

Unfortunately, things are not so easy for most technical writers. Normally, their documentation is designed for users rather than for programmers, so it is based on tasks or concepts rather than source-code structure. As a result, there is no natural connection between the changes a programmer makes to the source and the changes a technical writer has to make to the documentation. A single user task, such as creating a new account, might touch code from dozens of source code modules managed by different programmers; a single source code module might affect dozens of different task descriptions. Even a trivially small change to the source code can have an exponential impact on the documentation.

Consider a simple code module that displays a dialog box containing a message and two buttons labeled Accept and Cancel. The quality-assurance specialist sends a note to the programmer, saying that, for consistency, the first button should be labelled OK rather than Accept; the programmer takes 5 minutes to change one line of code, test, and commit, and the documentation specialist then announces that it will take 2 weeks to revise the tutorial and manual. What happened?

First, dozens of different parts of the code might invoke that dialog box, and each may be used by dozens of different tasks. Suppose that a manual has text like the following:

Select crop from the File drop-down menu.

A confirmation dialog will appear. Select Accept to continue.

For each instance, the writer will have to change Accept to OK, and then the writeror editor or quality-assurance specialistwill have to recheck all the documentation against the software. Even worse, the manual may contain screenshots of the dialog in different contexts, all of which will have to be recaptured and recropped. If a small change like this can cause so much trouble, it is not difficult to understand how a more fundamental change could throw technical documentation into chaos.

This kind of problem was common in computer programming as well until the structured programming movement, beginning in the 1970s, and the object-oriented programming movement, beginning in the 1980s, helped programmers get better at writing reusable code. Programmers have learned to encapsulate reusable code in a single place, such as a function or an object or even a library, rather than duplicating the same code over and over again in their programs; database designers do the same thing when they normalize their database. In fact, document writers have been able to do this for centuries before computers existed, simply by embedding a reference in a text, such as "(see Job 8:810)."

Modern technical documents could use the same include-by-reference approach as modern computer programs, in which case their documents might look like this:

To create a new document, take the following steps:

(See p.145)

(See p.251)

(See p.18)

(See p.44)

(See p.182)

It would be easy to write documents this way, especially with word processors that can track references and automatically fill in page numbers, but it would not be easy to read such a document. Following cross-references and keeping track of previous locations are a lot more difficult for humans than for a computer, and people reading documentation like this will quickly get frustrated or simply lost. Documentsat least in the final form that readers seeare necessarily highly redundant, or what a database specialist would call denormalized: They have to repeat information.

This is where another of XML's big promises comes in. XML, like its predecessor, SGML, is designed to allow writers to reuse text the same way that programmers reuse code: A single change in the XML source document can automatically propagate itself throughout the output formatted document at the other end of the XML publishing system.

XML has several mechanisms for allowing reusable text, among which the simplest is the internal text entity. In the internal DTD subset, an author includes a declaration like the following:

Then, in the main text, the author enters a reference to that entity:

A confirmation dialog will appear. Select
&accept-button-name; to continue.

Although this approach is dead simple for a single author creating XML in a text editor, it can cause problems in a large, multiuser environment, in which regular authors should not be able to modify the DTD, where the entity declarations appear. In those cases, system designers come up with more elaborate methods for reusable text. For example, in a maintenance manual, the following caution might appear many times:

Caution: Use calibrated torque wrench. Overtorquing may cause the bolt to shear.

In a big project, an author might create this caution once, possibly as an independent XML document like the following:

Use calibrated torque wrench. Overtorquing may
cause the bolt to shear.

An author who needs to include this caution will include it by reference, often through a custom-designed dialog box added to the editing system, as follows:

The same technique should work for shared steps in tasks, boilerplate legal text, and anything else that gets repeated throughout a document.

This approach looks like the ideal bridge across the discontinuity between the way coders code and the way writers write, but experience in field use has been disappointing. First, in technical writing, repeated text tends to be similar but not identical: A part name in the middle of a caution or the transition text at the beginning of a paragraph will change, depending on its context. Even when the text does not change, managing and locating small, reused chunks of text is mind-numbing work. Most authors would probably prefer to simply to retype when necessary, rather than spend several minutes each time searching through a repository of reusable steps to see whether one is appropriate. Even if authors were willing to use such a system, the savings would not be as great as forecasted. For example, if the same step were used in 200 places and the step were changed, authors or editors would still have to check all 200 places in the text to make sure that the change was appropriate in context, and checking the 200 places will usually take as long as retyping them.

It may be that new tools and new ways of running projects make reusable information more common in the future; for now, however, XML is not a universal solution to the documentation discontinuity.

3.4.3. Idioms
Although reusable text is a bit of a chimera, single-source publishing is a real benefit that can come from an XML publishing system. However, single-source publishing also has its limitations, and it is important to understand them before starting on a major XML publishing project.

Single-source publishing is an exciting idea that also happens to be easy to explain to nonspecialists. You create a single XML source document, then use scripts or templates to transform it automatically into different publication media, such as print, Braille, an automated voice telephone system, or the Web.

Much of the time, people want to publish from XML source documents only to print and the Web. For publishing to print, the typical data formats are TeX, PDF, RTF, PostScript, and MIF; for publishing to the Web, the typical data formats are HTML and XHTML but sometimes also PDF or Flash. XML specialists learn quickly that they need to write separate transformation style sheets for print and the Web, even if the core content is identical. The obvious problem is that they are transforming to different primitives: HTML deals with abstractions, such as paragraphs and lists, whereas print formats tend to deal with concrete layout elements, such as blocks, fonts, and spacing. This difference is not simply a design problem that could be fixed. Print formats are fundamentally page-based, whereas HTML is fundamentally screen-based. Each has pros and cons.

Page-based formats allow a designer to take advantage of all the available space, by including multiple texts and graphics in different parts of a single page, with fine control over the placement and size of each item. However, page-based formats are also brittle: A document needs to be optimized for a specific size and aspect and will not move easily across different display devices. (Try viewing a U.S. letter- or A4-sized PDF document on a handheld computer.)

Screen-based formats are inherently more flexible and, when properly used, will work for many different display sizes. However, that flexibility comes at the cost of surrendering control over the finer points of layout. (Try placing sidebars and graphics precisely in HTML.)

The real problem, then, is that a Web page is not simply a print document online but rather a fundamentally different kind of thing. That is why Web pages need their own style sheets. That's not such a big problem, however: Writing two, or even ten, style sheets accounts for very little overhead when you will be using them hundreds or thousands of times to transform XML for publication.

Single-source publishing works well for both print and the Web, as long as you are publishing the right kind of thing. A technical manual for a software program, or a novel, can easily pass from a single XML source document through a couple of different style sheets to print and Web versions, all without human intervention. You do end up, usually, with Web pages that scroll a lot (say, one page for each chapter). That will not be a problem if the user has decided to read a book or manual online, but it is not what you normally expect to find in a Web page. Web pages are typically short, dynamic, and interactive, not long and static.

There is no reason that a person cannot design an XML document type that takes dynamic content into account, so that the HTML rendition can contain animations, applets, forms, and so on, but doing so requires that you place new constraints on your XML document type in advance: You cannot publish just any document and have it look good both online and in print. This is not a medium problem but an idiom problem. You can print a Web page on paper or put a novel online, but neither fits naturally there, because Web pages and novels are fundamentally different kinds of things. Many other idioms cause trouble for single-source publishing. Consider, for example, slides, online help, and Unix man pages: Each of these follows a fundamentally different set of constraints and carries a different set of reader expectations, and it gets more and more difficult to generate all of them from a single XML source document. In the end, XML cannot deliver on all its promises for single-source publishing; it can allow you to publish to multiple formats from a single master document, but publishing to multiple idioms is much more difficult.

Is universal single-source publishing a hopeless case, then? Not quite. It turns out that, although all these idioms have drastically different top-level structures, they share much in common at the lower levels. Many of them, for example, have paragraphs of text with special-purpose phrases contained in them, and many use lists and tables. For the lower-level content, XML truly can deliver on at least some of the promise of single-source publishing: the same table might appear on a slide, a man page, a Web page, and in a printed book, all in significantly different contexts, appropriate to each idiom.

Furthermore, people are often willing to accept deficiencies for the sake of saving money or having easier access to information. For example, a state legislature might accept simple, unimaginative, automatically generated Web pages for its online legislation in exchange for a cost savings of $100,000 a year. The Web site for a popular magazine, on the other hand, will not likely be willing to make the same trade. Designing and implementing an XML-based publishing system is largely a matter of managing expectations. Raising false hopes about single-source publishing will lead to disappointment and hostility later on, but if both the implementers and the users understand and agree to the tradeoffs in advance, XML publishing can work and save money.

Formatting XML

2009-06-15T23:24:00.000-07:00

Formatting and Production
Once the higher-level issues are resolved and the authoring system is installed, it is time to turn to the nitty-gritty details of formatting. In many cases, there will be no problems at all; XML, together with transformation and formatting software, does a good job of handling the typical, routine tasks of formatting, particularly if the XML master document contains a single text flow continuing over several pages, such as a technical manual.

Unfortunately, things are sometimes not so simple. This section examines some of the physical aspects of printed documentation that can cause problems for XML publishing. Sometimes, these problems will not surface until late in a project, when there is not enough time or money left to fix them properly; learning to anticipate them can make a big difference to an XML publishing project's chance of success.

3.3.1. Change Markup
Technical publications often include various kinds of change information to make it easy for users to find differences between versions, and encoding this kind of information in XML markup probably represents the single biggest difficultly in XML publishing. The final change information in a printed text can take many forms:

Vertical bars in the margin beside changed text

Separate textual descriptions of changes made and the reasons for them

Different font combinations to show text removed and added

Differently formatted section headings to show sections that contain changes

Even finding the differences between two versions of the same XML document in the first place can be a problem, although more open-source and commercial software is becoming available. Some XML differencing algorithms scale badly with large documents, so it is worth load testing your intended differencing software early in any project. Assuming that you do have some mechanismeven manual identification by the authorin place for locating changes in an XML document, this section examines some of the problems with inserting the change markup into XML documents for publication.

3.3.1.1 Markup Issues
Change markup in XML documents causes publishing difficulties on several levels. Most basically, changes do not tend to fit neatly into XML markup trees. Consider the following:

[...] There are 203 authorized service depots in Southeast
Asia.

The authorized service depots all provide [...]

A change in the company's technical-support structure could cause the content to change, as follows:

[...] There are 55 service partners in the Asia-Pacific
region.