Game over for OpenDocument?

Is it game over for OpenDocument? Probably. We’ve been expecting Massachusetts ITD to publicly revise its open formats mandate to include Office Open XML (OOXML) ever since Louis Gutierrez resigned as CIO in early October 2006. That was as clear a signal that ODF had failed in Massachusetts as needed by anyone in the know.

The only surprise is that it took new ITD CIO Bethann Pepoli so long to make the announcement that OOXML would be officially recognized as an “open” XML file format going forward. How she determined OOXML is open when the Microsoft Open Specification Promise allows no other developer to implement it is a mystery, but it seems a safe bet that the decision was based on political and practical considerations rather than sound legal advice about OOXML’s qualification as an open standard. See definitions ostensibly used by Massachusetts ITD to define openness in its ETRM:

Open Standard – Specifications for systems that are publicly available and are developed by an open community and affirmed by a standards body. Hypertext Markup Language (HTML) is an example of an open standard. Open standards imply that multiple vendors can compete directly based on the features and performance of their products. It also implies that the existing information technology solution is portable and that it can be removed and replaced with that of another vendor with minimal effort and without major interruption. (Enterprise Open Standards Policy)

Open Format – The Commonwealth defines open formats as specifications for data file formats that are based on an underlying open standard, developed by an open community, affirmed and maintained by a standards body and are fully documented and publicly available. (ETRM, Information Domain)

According to a Network World story, “Pepoli said the state needs to pick up the pace in adopting XML-based formats and we think now that ‘to have both formats will make it easier.'”

But Pepoli said adoption of Open XML is not a done deal.”Someone could submit a comment and we could make a review of ETRM and make changes,” she said. Those changes could include eliminating Open XML from ETRM in the final draft.

But we’re not holding our breath, given the heavily politicized situation in Massachusetts and the technical barriers to ITD adoption of ODF. If you’re interested in submitting comments, the address is standards@state.ma.us. The deadline is July 20, 2007.

Yet a further mystery is why Massachusetts chose to add OOXML to its repertoire when not even Microsoft Office can generate files in that format. OOXML is a one-way, import-only format for MS Office, a crippled subset of Microsoft’s own XML formats (MOOXML). See the comment by Marbux on betanews.com. Were a Massachusetts agency to specify in a software procurement tender that candidate applications must read and write the Ecma OOXML formats, Microsoft Office itself would be ineligible. Presumably, Massachusetts ITD has not yet awoken to this fact.

Standards attorney Andy Updegrove understandably hopes that public pressure on the Massachusetts legislature can reverse the proposal to add OOXML to ITD’s ETRM. But how can this not be the death knell for ODF? The failure of ODF in Massachusetts will result in worldwide recognition that it is impossible to implement ODF at the enterprise level where business processes are already bound to Microsoft Office.

This is exactly what happened to ODF mandate legislation in California. The agency CIOs in California uniformly rejected both ODF legislation and Sun’s hapless effort to set up an ODF Pilot Study based on what had happened in Massachusetts. If Massachusetts could not implement ODF, then they saw no reason for them to try. And four other states thus far have shot down similar legislation, heavily backed by Sun and IBM, that attempted to force “rip out and replace” implementation of ODF.

And it does come down to implementation.

Most people think the implementation of ODF is as easy as downloading OpenOffice.org and converting your legacy documents to ODF as they are used. Simply fix the artifacts of conversion in process, and never look back. OOo is free. So what’s not to like?

The problem is that requirements differ at the enterprise level, which are heavily invested in fully automated business processes. In such situations there is no opportunity for manual inspection of documents for conversion artifacts. Flawless integration of applications is mandatory, and the existing inventory of enterprise documents is stored in Microsoft binary formats. The quality (fidelity) of document format conversions must be very, very high. Does it matter? Would you want your doctor to perform surgery on you while making decisions from an automatically generated history of laboratory testing results that contained errors due to data conversion corruption?

And from a legal standpoint, laws around the world effectively mandate 100% fidelity in migrating binary documents to XML. See e.g., E-SIGN Act, 15 U.S.C. 7001(d)(1)(B) : (electronically preserved records must “accurately reflect[] the information set forth in the contract or other record” and be “in a form that is capable of being accurately reproduced for later reference, whether by transmission, printing, or otherwise”); Sarbanes-Oxley Act, 15 U.S.C. 7261(b) (financial information must “not contain an untrue statement of a material fact”). In fact, it is rare to find a records retention law that does not require stored records be accurately preserved.

From a broader view, the world has 15-plus years of building business processes with fully integrated applications and other client/server integration, all atop the MS Office application suite. These business processes are hard-wired to MS Office, the familiar monopoly vendor lock-in product.

So the barrier for ODF applications and ODF is twofold. Any implementation of ODF must overcome both the barrier of converting Microsoft’s legacy binary documents to ODF, then be able to integrate with the MS Office-bound business processes, without corruption of data. This entails virtually flawless round-trip interoperability, without loss of data, what has been referred to on the OASIS OpenDocument TC as the “high-fidelity migration and business process use cases.”

The cost and disruption of a rip-out-and-replace migration to ODF at the enterprise level is impossibly high. The City of Munich, Germany, for example, spent more than $3,500 per seat to migrate from Microsoft Office to OpenOffice.org, with most of the expense attributed to file conversion expenses and the costs of replacing business process scripts. See also Finland Ministry of Justice, Migrating a Ministry to OpenOffice.org. (“A complete migration to OpenOffice.org was not considered a practical option due to the Microsoft technology based application integrations in the document handling of the Finnish Government.”)

Yet rip out and replace is the only solution ODF vendors offer governments with high-fidelity integration and migration requirements. But as was said by one of IBM’s experts in service-oriented architecture: “Because most companies have a significant investment in their legacy infrastructure, management is typically not open to ripping out and replacing legacy systems, regardless of the level of shortcomings evident in the infrastructure. Rewriting or significantly modifying large portions of a legacy environment is neither practical nor realistically accomplishable in a reasonable time frame.”

It’s too bad that IBM’s office productivity software executives never got that memo. IBM and Sun Microsystems remain the staunchest advocates of rip-out-and-replace migration to ODF and of a continued interoperability barrier between ODF applications and Microsoft Office. They apparently believe that excellent support for ODF in MS Office would prolong the life of that product. They seem oblivious to the fact that high fidelity interoperability is key to ODF applications’ infiltration of Microsoft-bound business processes.

Is there a way around this impossibly high double barrier? Yes, but it involves the use of an internal ODF plug-in for MS Office.

This should not come as a surprise to the file format cognoscenti because the internal plug-in route is exactly how Microsoft migrates existing documents and business processes to their own OOXML formats, MOOXML. Internal ODF plug-ins are simply clones of the MOOXML plug-in, installed through the MS Office 2007 Compatibility Pack in earlier versions of MS Office.

Massachusetts decided to go the plug-in route after conducting a year-long ODF Pilot Study. Before the Study was even complete, Massachusetts ITD realized the impossibility of the dual barriers. They issued an unprecedented Request for Information concerning the possibility of a ODF plug-in for MS Office. They wanted a plug-in that would be able to internally convert documents so transparently and with such high fidelity that there would be no disruption to existing business processes. That would allow time for a phased migration to other ODF applications as business process scripts were adapted or replaced.

But this also meant that the ODF internal plug-ins for MS Office had to match the performance and quality of the Microsoft internal OOXML plug-ins. This is not an impossible task. Where ODF failed in Massachusetts is not with the internal ODF plug-ins, but with the big ODF application vendors’ refusal to support da Vinci, which was the only internal MS Office plug-in whose developers were willing to go the ODF Community route former Massachusetts CIO Louis Gutierrez desired.

The da Vinci plug-in could be released within a few weeks if the only goal was to add virtually perfect native ODF support to MS Word. But that is insufficient to establish interoperability with other ODF applications such as OpenOffice.org. That is because Sun Microsystems, which absolutely controls the OpenDocument standard development process, has programmed OpenOffice.org to destroy all but two of what section 1.5 of the ODF specification refers to as “foreign elements and attributes” and is busily making sure that the new RDF metadata features in ODF v. 1.2 will not be dependable for interoperability purposes. See, for example, this thread on the ODF Metadata Subcommittee mailing list.

So now we have Massachusetts and Denmark recognizing both OOXML and ODF as open XML file formats. More governments will follow. The problem with that recognition is that the internal Microsoft OOXML plug-in is the only cost effective (free) and non-disruptive way for existing MS Office-bound workgroups and workflows to move to XML.

The truth is, the big ODF application vendors left governments with no other choice but to go with OOXML as the only way to migrate existing systems to XML. They hoped to capitalize on ill will against Microsoft and legislation forcing rip-out-and-replace migrations. But as the Massachusetts situation, the state legislation situation, and the situation in Denmark shows, government IT establishments are beginning to rebel against the foolhardy and expensive rip-out-and-replace strategy.

The bottom line: The best remaining hope for Open Document Exchange Formats now lies with the European Community. At a conference on Feb.28 to March 1, 2007, IT representatives of 21 European Community national governments laid out their demands for industry. It was in effect an announcement that neither ODF nor OOXML were acceptable, that they expect Microsoft and the big ODF vendors to collaborate on development of a single harmonized set of formats, with ODF providing the common features. The summing up at the plenary session says much about the likely end game in the office document file format war:

* There is a general dissatisfaction with the perspective of having competing standards.

* One format for one purpose: Administrations should be able to standardize (internally) on a minimal set of formats.

* No incomplete implementations, no proprietary extensions.

* Products should support all relevant standards and standards used should be supported by multiple products.

* Conformance testing and document validation possibilities are needed -> in order to facilitate mapping/conversion.

*  Handle the legacy/safeguard accessibility.

Europe is now openly threatening to use the government procurement power to force convergence of ODF and MOOXML. The question is whether the big vendors will bow to what the market requires.

But what is perhaps most remarkable about the situation is how thoroughly standardization organizations like Ecma, OASIS, and ISO have failed to protect software users’ interests from big vendors’ efforts to maintain separate office productivity software markets, divided by incompatible file formats. It’s time for the Free and Open Source Software community to develop its own office document standard, along the lines of what Europe has demanded.

A new set of formats, perhaps based on a wedding of XHTML , CSS 3.0, and RDF, or perhaps an interoperable enhancement of ODF, is in order. This is clearly what the EU IDABC has in mind with their recently announced Open Document Exchange Format proposal — ODEF. The amazing thing is that the internal plug-ins can produce ODF, XHTML , and ODEF. The internal plug-in architecture is how those not aligned with the big vendors can neutralize big vendor control over both applications and standards.

The big vendors had their chance and blew it. Now it’s time for the demand side of the equation to pick up the pieces and go for the universal interoperability demanded by a real world convergence of information systems needing to perfect the high fidelity exchange of portable XML documents.

Sad, isn’t it?

Edwards and Martin are respectively, president and director of legal affairs for the nonprofit OpenDocument Foundation, which is affiliated with the developers of the da Vinci plug-in discussed in this article.