Driven to drink by code and coding

We were recently driven to drink trying to get an application to talk XML to the embedded Web server in HomeSeer, the home automation application we discussed some weeks ago.

We were recently driven to drink trying to get an application to talk XML to the embedded Web server in HomeSeer, the home automation application we discussed some weeks ago.

The application, written using Xcelsius from Infommersion, was supposed to send an XML datastream to HomeSeer server via a POST request. The payload of an HTTP POST request follows the header and looks something like this:

POST /test.asp HTTP/1.1 www.somehost.com

Host:

User-Agent: Mozilla/4.0

Content-Length: 27

Content-Type: application/x-www-form-urlencoded

userid=bob&password=fish

The payload is everything that comes after the “Content-Type” line. The format of the POST payload should be one or more “variable=value” pairs separated by ampersands.

Using Active Server Pages (ASP) code, the browser’s POST request is stored in a data structure called the “Form collection.” But with HomeSeer there’s a limited number of access methods to get to this data, so we used the following:

mydata = Request.Form(1)

From the POST request detailed above, this code would set mydata equal to “bob” (quotes not included). So, here’s an XML datastream that we’ll send to a Web server:

   A A A

   B B B

   D

   E

This is the format of a two-column, two-row table encoded in XML. When the data is accessed through the Form collection using the ASP code above, the contents of mydata will be:

"SomeStuff">AAABBB

CCCD

EF

What happens is the built-in parsing of POST requests by the Web server assumes the character string “

But the XML variable name was “Some Stuff” not “SomeStuff”: The white space is AWOL. And not only the white space between the XML tags disappeared but also the white space in the tag’s data.

This happened because it is the responsibility of the sender to “URL encode” all characters per RFC 1738, the “Uniform Resource Locators (URL)” specification, before sending them to the server and here, the white space isn’t encoded!

In the case of un-encoded white space, the Web server simply ignores it. If other un-encoded characters such as ampersands and equals signs are present, they will cause the server to make further assumptions and complicate matters even more.

As it turns out, that was our problem. Consider the following:

$&+,/:;=?@ “‘#%{}|^~’

If this was URL encoded to RFC 1738, the character sequence sent to the server should be:

%24&%2B%2C/:;=?@%20%22'%3C%3E#%25%7B%7D%7C%5C%5E%7E%60

Xcelsius generated the following:

$&+,/:;=?@ "'<>#%{}|^~'

This is not URL encoding, but HTML encoding. And the result of the application POSTing XML data without converting it to the URL encoded format that the Web server expected cost us about 6 hours before we figured out what was going on. It turns out that the cause was not actually Xcelsius. It was Macromedia’s implementation of XML POSTing in Flash’s Actionscript language. The Xcelsius engine is written in Actionscript so it inherits the snafu.

We found out subsequently that Colin Moock’s book on Flash’s underlying language, Actionscript: The Definitive Guide, says: “The server application receiving the posted XML string must be able to access the raw POST data of the HTTP request directly and should not attempt to parse it as normal name/value pairs. . . . In ASP raw POST data may be accessed via the Request.BinaryRead method. . . .”

We needed a stiff drink. What can we pour for you at gearhead@gibbs.com?