The Open Hyperdocument System

An Open System for Collaborative Knowledge Work 1

Principal Investigator: Douglas C. Engelbart
Co-Investigators: Lee A. Iverson, Eugene Eric Kim, Patrick D. Lincoln
Submitted to NASA Ames: July 13, 2000 1A

Introduction 2

Solving today's most urgent and complex problems, from designing a spacecraft to managing huge bodies of information, already exceeds the capabilities of any single person. Further, the complexity and urgency of many issues facing NASA are rapidly accelerating. Thus, these issues will soon far exceed the capabilities of current organizational systems. Matters are made worse by the increased speed with which key decision makers shift responsibilities, leaving behind insufficient information for their collaborators to seamlessly continue. We propose an open evolutionary approach to increasing the collective ability of groups of people to address complex urgent problems, through the construction of an Open Hyperdocument System. 2A

The most important aspect of any collaboration is the sharing and management of knowledge. There are two opportunities for enhancing knowledge management in an enterprise such as NASA. First is improving operations. At a minimum, this entails managing the highly iterative and dynamic flow of specifications and design information between an organization and its many suppliers and clients. 2B

Second is the enhancement of NASA's main mission, which is scientific exploration and discovery. Innovations in these areas require the ability to manage enormous amounts of data, make it available to others, and generate knowledge while maintaining the associations with that data. 2C

Fundamentally, the solution to these problems is the same: a system that manages the flow of information, the addition of new information, and the access to and sharing of knowledge. We propose to build a standards-based, collaborative knowledge management tool called the Open Hyperdocument System (OHS) that will support deep hyperlinking between documents. We plan on embracing open standards and building the Open Hyperdocument System as open source to encourage its widespread deployment and coevolution. 2D

Project Description 3

The Open Hyperdocument System will be an integrated, collaborative system for storing, editing, viewing, and discussing documents. It will enable sophisticated linking between all types of documents. The architecture will be extensible, allowing support for additional document formats and customizable views. 3A

This system will have four important technical features: 3B

Associating concepts is a natural and powerful method of organizing information. Hyperlinking provides a way to associate different concepts, and to easily navigate through recorded knowledge. Of course, hyperlinking is nothing new. The first hypertext system was implemented in the 1960s by Doug Engelbart, this project's principal investigator. Since then, there have been different incarnations of hypertext systems. 3C

Curiously, the most successful of these systems -- the World Wide Web -- is probably the least sophisticated. Web hyperlinks are unidirectional, and the ability to cite documents at a granular level is limited. These restrictions prevent users from annotating documents and traversing links in more than one direction. 3D

The Open Hyperdocument System will provide hyperlinking capabilities that enable sophisticated and desirable features such as annotations and granular addressability. These capabilities will be built on top of existing and evolving open standards, such as the W3C's XLink and XPointer specifications. 3E

Standards such as XLink and XPointer apply only to documents constructed in eXtensible Markup Language (XML). However, the vast majority of documents are not constructed in XML, but in a wide variety of formats, such as plain text, Microsoft Word, and PDF. For a knowledge management system to be truly useful, it must extend the capability of legacy document formats to support hyperlinking and addressability. 3F

By automatically translating documents into an intermediate XML format without modifying the original document, the Open Hyperdocument System will support linking to and from legacy document formats. For example, editors will be able to annotate an article by linking their comments to Microsoft Word documents submitted by the author. Product managers will be able to link software requirements documents to related e-mail discussion threads, use cases, class diagrams, and even source code. 3G

An extremely flexible and usable user interface is a crucial component to the system. Not only will it improve users' abilities to access information, it will help compel people to record their knowledge, one of the most significant problems in knowledge management. 3H

The Open Hyperdocument System will support customizable views for documents. For example, when viewing an article, you can have the system show the entire document or only the first sentence of each paragraph. Comments and responses to the article can be viewed as links to separate documents, or they can be displayed within the article itself. Views can be generated for different types of devices, from large CRT monitors to braille machines for the sight impaired. In addition, the Open Hyperdocument System offers an integrated editing and viewing system. 3I

To support these viewing and hyperlinking features, the Open Hyperdocument System will be built on top of an extensible application engine, which will also allow the system to be customized in other ways. For example, the system could be easily modified to link certain types of data automatically, or to generate machine-understandable knowledge representations of documents. 3J

Plan of Action 4

Our primary goals are both to build a highly usable system and to create an open platform that will rapidly evolve. To achieve these objectives, we plan on bootstrapping our development effort. Central to this strategy is deploying this system as open source. On the surface, this means that the source code will be open and freely available. However, we believe that this approach will have even more far-reaching benefits, including the fostering of a large community of users and software developers using and improving the system. 4A

To facilitate this bootstrapping process, we have decided initially to target the problem of collaborative software development. While many of the features that we implement first will be generic enough to apply to many types of collaborative knowledge work, by creating a compelling tool for software developers, we can encourage open source programmers to contribute to our project. 4B

How does the Open Hyperdocument System address the challenges of collaboratively designing and writing software? Countless studies have emphasized the importance of the documentation process for the successful completion and maintenance of software. However, most engineers spend little effort on gathering requirements, writing design documents, and annotating source code, preferring instead to focus on the programming. Accessing knowledge that exists only in the head of an engineer can be, at best, a chore. Even when this knowledge is recorded, it is often out of sync with the current state of the project. 4C

While the documentation process in most software development projects leaves a lot to be desired, a tremendous amount of recorded knowledge is produced anyway. The process of designing, writing, and building software results in many important documents, such as architecture diagrams and the source code itself. In addition, engineers often discuss their project over e-mail and other electronic forums. However, this information is rarely well organized, largely because of incompatible document formats, poor tool integration, and poor user interfaces. 4D

Initial versions of the Open Hyperdocument System will address these issues by providing templates that simplify the documentation process and by allowing deep linking between related documents, such as requirements, e-mail discussions, and source code. Customizable views for these documents will give programmers ways to focus selectively on the information most relevant to their immediate tasks. 4E

We have divided the project into smaller pieces that we plan on implementing quickly and releasing aggressively. Our initial targets for the first year are 4F

Upon completion of these goals, we will have designed and built an overall framework for the Open Hyperdocument System that can be used for collaborative knowledge management in all fields. While we continue to extend the system to support multiple views and other types of legacy document formats, we will begin to focus more of our efforts on improving the user interface, so that the system is well integrated and easy to use. 4G

To maximize the potential for the evolution of this project, we plan on fully supporting existing open standards while introducing new ones. For example, one likely contribution will be a specification for a Distributed DOM (DDOM), which will allow programmers to manipulate XML documents distributed over different machines. 4H

Initially, our system will be HTTP-based, although that may change as we continue to develop the system. While the application infrastructure will most likely be written in Java, we will work hard to promote language independence and interoperability as much as possible. 4I

References 5

Engelbart, Douglas C. "Augmenting Human Intellect: A Conceptual Framework." AFOSR-3223 under Contract AF 49(638)-1024, SRI Project 3578 for Air Force Office of Scientific Research, Stanford Research Institute, Menlo Park, CA. October 1962. 5A

Engelbart, Douglas C. "Collaboration Support Provisions in AUGMENT." OAC '84 Digest (Proceedings of the 1984, AFIPS Office Automation Conference, Los Angeles, CA, February 20-22). 5B

The W3C DOM Working Group. "Document Object Model (DOM)." The World Wide Web Consortium 5C

The W3C XML Working Group. "XML Pointer, XML Base and XML Linking." The World Wide Web Consortium 5D