Creole Wiki Markup, WikiOhana, and More Ward Wisdom

Several folks asked me on Saturday what I thought about the conference, and I kept saying, “It’s been great, except I haven’t had any interesting conversations about Wikis.” That changed in an unexpectedly generative way on Saturday afternoon, resulting in a new little cooperative effort we’re calling WikiOhana.    (KZ1)

At Thursday night’s conference party, I got reacquainted with Chuck Smith, whom I had met at Wikimania last year. Chuck is the founder of Esperanto Wikipedia and was the one who first introduced Brion Vibber — now the lead developer of Mediawiki — to the Wikipedia community. At the conference last year, Chuck met Christoph Sauer, a German Wiki researcher, and the two of them started working together.    (KZ2)

At the party, Chuck told me that he and Christoph had been working on a proposed WikiMarkup standard, and I loudly grimaced in response. At WikiSym last October, I made it clear to several people that I thought trying to standardize WikiMarkup was a noble waste of time.    (KZ3)

First, people are really attached to their own markup. Previous discussions about standardization usually started and ended with, “Great idea. Let’s just standardize on mine.”    (KZ4)

Second, WYSIWYG in theory obviates the value of a standard markup from a user’s perspective. It seemed more valuable to work out a standard interchange language, which WYSIWYG widgets could use to interact with different Wiki engines and which would make data migration between different Wiki engines easier. It also seemed like it would be easier to come to agreement on an interchange language. In fact, folks have already worked out a good proposal for an XHTML interchange language on the Interwiki mailing list (now migrated to the wiki-standards list) and on CommunityWiki.    (KZ5)

What made all this talk worse for me was that I thought that people were trying to go about this the wrong way. We were not going to come to agreement by group authoring a spec, then getting everyone to agree on it. The only way this was going to work was if a few Wikis came to some agreement and actually implemented the changes. This wouldn’t necessarily result in a standard, but it could act as a catalyst that could lead to a standard. Sunir Shah has been advocating this approach for a long time, to no avail. I would have organized such an effort myself, except that I didn’t think it was important enough to spend any time on it.    (KZ6)

Chuck and Christoph changed my mind. Chuck explained that he and Christoph had extensively analyzed existing Wiki markup, and they had taken pains to come up with a small subset of things that would be as noncontroversial as possible. (There had been a similar effort by others to do such an analysis before, but it hadn’t gotten very far.) They were going to present the results at WikiSym in a few weeks, but they had prepared a poster for Wikimania. Still clucking my disapproval, I promised Chuck that I would check out his poster.    (KZ7)

On Saturday, I finally made some time to check out the poster. I studied it and found myself thinking, “Hmm. This isn’t bad.” Chuck came over later, and we started going over it in depth. I still have some issues with it (discussed below), but I told him, “You know what, this is almost good enough for us to adopt in PurpleWiki. But you really should find one or two other Wikis who might say the same.” We kicked a few ideas around, and then I said, “Let’s go see what Ward thinks.”    (KZ8)

So we rounded up Ward Cunningham, and he took a look and started asking questions. He seemed to be okay with the answers, because he said that if someone were to write a Perl function that converted from his markup to the proposed markup and vice versa, he would consider incorporating it into his Wiki.    (KZ9)

Creole Markup    (KZA)

Well, you know me. Low-hanging fruit, a spare hour before the next party. The long and the short of it is, half of the converter now exists. It’s called Creole, and it’s written in Perl. Currently, it consists of a brain-dead simple script that converts Ward’s markup into Creole Markup (Chuck and Christoph’s proposal) and some tests.    (KZB)

The name was Ward’s suggestion. Rather than call it “pidgin,” which is what it actually is (a non-native language cobbled together from two or three other languages), we chose to optimistically call it “creole” (a native language that emerges from a pidgin), which is what we hope it will become.    (KZC)

I have three issues with the current proposal. First, I don’t like the exclamation point syntax for headers, not because of the character itself, but because of the numbering:    (KZD)

!!!Heading 1 !!Heading 2 !Heading 3    (KZE)

As you can see, you need to remember that a top-level header has three exclamation points, not one, and you are only capable of having three levels of headers. The advantage of the equals syntax:    (KZF)

= Heading 1 = == Heading 2 == === Heading 3 ===    (KZP)

is that the heading level is represented by the indentation.    (KZQ)

Second, there needs to be a proposal for preformatted blocks.    (KZR)

Third, the spec isn’t rigorous enough. You have to answer questions like, “Will italics work across multiple lines or blocks?” The reason I didn’t write a Creole-to-Ward converter was that these questions were not yet answered.    (KZS)

Most of the time developing the Creole converter was actually spent reverse engineering Ward’s markup. Ward pointed me to a Literate Programming version of his source code — implemented in a Wiki, of course — which helped a lot. As it turns out, Ward’s been thinking a bit about how to write a good markup parser recently. Parsers are often on my mind as well, as we’ve been talking about rewriting the PurpleWiki parser forever. Plus, the topic came up at Hacking Days among the Mediawiki developers before the conference. We’ll probably noodle on this problem a bit both before and during WikiSym. Fortunately, Ward knows a lot more about writing parsers than I, as anyone who’s seen the PurpleWiki code can attest.    (KZT)

WikiOhana    (KZJ)

When we were discussing names, Christoph proposed “ohana,” which means “family” in Hawaiian. It has a wonderful connotation — respecting our individuality while emphasizing the bonds that keep us together in a positive, welcoming fashion. It’s also consistent with Ward’s original naming convention, as “WikiWiki” means “quick” in Hawaiian.    (KZU)

We decided to call our little cooperative effort, WikiOhana, of which Creole Markup is a first project. We hope the family grows over time.    (KZV)

More Ward Wisdom    (KZK)

While retelling an experience he had writing a parser, Ward said, “That’s when I learned how to write beautiful code: Get it working, then spend the same amount of time making it better.”    (KZW)

Announcing the Hyperscope Project

One of the new projects I have the joy of being involved with this year is Doug Engelbart‘s Hyperscope. Doug recently got a bit of NSF funding to build an Open Source prototype of his vision for a Hyperscope, and Doug asked me to lead the project.    (KAT)

You can read more about the project on the project blog. In short, we’re replicating the original hypertext system’s (Augment) browsing (jumping) and viewing capabilities in Firefox.    (KAU)

Why is this significant? Because even though most of the world knows that Doug Engelbart built the first hypertext system in the 1960s, very few people have ever seen and experienced the system first-hand. And boy, is it a doozy. Doug continues to use the system every day, and it has capabilities that no other system has today.    (KAV)

We want the world to see these features, and we want the world to have the opportunity to learn from them and to integrate them into their own systems. We also want the world to realize that these are not features for features’ sake, but are part of a much larger vision of how society can (and must) get collectively smarter.    (KAW)

This project — and Doug’s larger vision — is about improving society’s ability to collaborate. And that’s why I’m involved. We’re not building code for code’s sake (although the code will kick ass). We are kicking off a larger community conversation about how we can improve collaboration, and how we can and should improve our tools to augment our capabilities.    (KAX)

Join The Community    (KAY)

We’ve got a fantastic core team in place. Brad Neuberg is our lead developer. I’ve written about Brad’s coworking efforts, but I haven’t written about how he’s a kick-butt developer who’s doing wild things with AJAX. More importantly, he’s also a deep thinker, which is an absolute requirement to be successful in this position, a leader in the Open Source community, and a lifetime Engelbart fan (not a requirement, but it didn’t hurt).    (KAZ)

Also joining us is Jonathan Cheyer. Jon’s a long-time member of the Collaboration Collaboratory, and he’s also the tech lead for the Computer History Museum‘s NLS/Augment Restoration Project. I’m pretty sure he’s the most proficient Augment user under 40, and I’m quite certain he knows more about Augment’s internals than anyone else under 40.    (KB0)

What makes the project even more fun is that we’ve been working with folks from Doug’s original lab. His daughter, Christina Engelbart, is program manager and is also sharing her insights into how the system was used as well as her knowledge of Doug’s larger vision. Jeff Rulifson and Charles Irby, the first software leads in Doug’s lab, have shared a lot of their knowledge with us, as have Harvey Lehtman and Raylene Pak. These gatherings have not only been extremely educational — stories galore — they have been tons of fun, and we want to encourage other ex-ARC folks to touch base with us and be part of this new community.    (KB1)

Some additional acknowledgements: Mark Finnern is one of Doug’s biggest supporters, and he has given Doug a valuable forum at Future Salon. He also hooked us up with Blake Ross and Joe Hewitt of Firefox fame, two very cool guys who spent some time with Doug and whom we hope will continually engage with this community. A big shout out also to Philip Gust, also of the NLS/Augment Restoration Project, Dorai Thodla, who experimented with an early prototype of the Hyperscope in Java, and Dave Thomas and his OpenAugment team.    (KB2)

Please join us! Subscribe to our mailing lists and blog (and weekly podcasts), and participate on our Wiki. I’ll mostly blog about the project on the project blog, but I’ll also occasionally discuss stuff here. Brad is also blogging about the project. We’ll also be at SuperHappyDevHouse this Saturday doing an Augment jam session.    (KB3)

Blue Oxen Associates    (KB4)

I’m involved with this project for personal and professional reasons. As I said earlier, this is not just about building a piece of code. It’s about engaging with the community at large and building a movement that falls squarely within the mission of my company. The Hyperscope project will help ground larger conversations about how we can and should improve tool interoperability and usability. More importantly, it will ground larger conversations about process and the bigger picture.    (KB5)

At a very concrete level, the Hyperscope falls well within our goal to bring these deeper ideas into existing tools. Not surprisingly, the Hyperscope should integrate very easily with PurpleWiki (as well as other Wikis), providing a new, slick and useful browsing capability. As the project progresses, I’m looking forward to evolving our tools, as well as seeing other folks evolve theirs.    (KB6)

On a personal level, Doug is why I’m in this business. I’ve never been happier with my work than I’ve been in the past three and a half years at Blue Oxen Associates. I bring people doing meaningful work together, I help them collaborate better, and I do it in an open way that hopefully has a much larger societal impact. It’s intellectually satisfying and emotionally fulfilling. And I wouldn’t be doing any of this had it not been for Doug, who’s been a mentor, a friend, and a cheerleader.    (KB7)

In a way, starting Blue Oxen Associates was my gift to him — a commitment on my part to carry out his larger vision. But working on the Hyperscope is a much more concrete way of returning the favor, and I look forward to doing this for him and for the community at large.    (KB8)

Purple v0.9

After RecentChangesCamp earlier this month, I drove up to Seattle to visit Chris Dent and others. Chris and I spent a day talking shop and life, and also taking care of a few things that we’ve been discussing for a while. The biggie was extracting the Purple Number generator from PurpleWiki into its own library and creating a RESTish front-end to it. The result — Purple v0.9, available on CPAN.    (K9N)

Not only will Purple make it easier for folks to incorporate Purple Number generation in their own software, it will also enable us to start properly experimenting with distributed Purple Numbers. Right now, I can transclude content between this blog and my Wiki. With Purple, I’ll be able to transclude from Chris’s blog and any other sites using these Purple Numbers (including all of Blue Oxen Associates‘s collaboratories.    (K9O)

It was very cool to get this done, and more is to come. Chris has already started to incorporate it into PurpleWiki, and Paul Visscher has already started to incorporate it into perplog. You can read Chris’s commentary and follow more on the Church Of Purple‘s progress at the Purple collaboratory.    (K9P)

The Pete Paradox

Peter Kaminski has a great maxim, which those of us who know and love him commonly call The Pete Rule:    (K8N)

“Time together in person is too important to spend working.”  T    (K8O)

I think it’s a great rule, but there’s a corollary, which I’ll call The Pete Paradox:    (K8P)

Time together in person is the best time to work.    (K8Q)

I actually had a specific experience with Peter last October that led me to think about this paradox.    (K8R)

At RecentChangesCamp earlier this month, I faced this paradox first-hand. On the one hand, I didn’t want to waste the opportunity to really get to know some of the excellent folks in the Wiki community who aren’t local to the Bay Area. On the other hand, I had some specific things I wanted to work on — namely SisterSites with Ward Cunningham and other Wiki developers and a YADIS implementation with the good folks at JanRain. Working on these projects meant breaking away from the larger group, popping open the laptop, and focusing on the work at hand.    (K8S)

The paradox resolved itself to my satisfaction. I got both projects done. We implemented SisterSites in PurpleWiki and a bunch of other Wikis, and Brian Ellin and I wrote a YADIS library in Ruby. I met several interesting and cool people for the first time, including Brian, and, I got to know folks like Bayle Shanks a lot better. On the second night, after the conference party, I decided to go to the main ballroom to check my email, even though I was exhausted. The ballroom was empty at first, but about half an hour later, Bayle showed up. We had met at WikiSym, but this was the first time we had a chance to sit down and really talk, and we stayed up past 3am. That was excellent. I had already known that Bayle had done some cool stuff, but it was great to dig deeper into his ideas as well as to talk about non-Wiki stuff.    (K8T)

I even got to discuss The Pete Paradox with Pete in person. (How’s that for meta?!) And here’s how I finally resolved it in my head. Both The Pete Rule and The Pete Paradox are about maximizing engagement. When you are working closely together, you are engaging in a way that is not only more productive, but more meaningful. Face-to-face time spent working can be a waste, but face-to-face time spent truly working together usually isn’t.    (K8U)

Purple Numbers and WIKIWYG

For a while, it was looking like I was going to break another personal blogging record last month, then things got so busy I had zero time to blog whatsoever. That means I’m in catch up mode again, so as usual, I’ll post in reverse chronological order (which in the blogosphere is really reverse reverse chronological order).    (JYN)

Yesterday, I spent the afternoon at Socialtext, where they were having an all-hands meeting. They graciously invited me to participate in the Open Space segment of their gathering, which meant quality time with Chris Dent and a rare opportunity to evangelize the Church Of Purple together. Of course, Chris has been spreading the Purple religion at Socialtext for a while now, so it wasn’t as much about evangelism as it was about next steps.    (JYO)

As I’ve mentioned many times before, browser-based WYSIWYG editors are an exciting development because they allow us to make Purple Numbers transparent in the authoring process. Right now, when you edit a PurpleWiki page, you see the node ID tags (e.g. {nid 123}). This is impossible to get around with the default browser text-editing widget. However, with a WYSIWYG editor, you can hide the Purple Numbers while still maintaining their associations with a node behind the scenes.    (JYP)

That’s the theory, anyway. In particular, I’ve been excited about WIKIWYG ever since Ross Mayfield showed me an early prototype last August. I had a personal bias, since Chris Dent and Matt Liggett helped write it, as did Casey West and the inimitable Brian Ingerson, whom I finally met last weekend at Tag Camp.    (JYQ)

Yesterday afternoon, we discussed Purple Numbers and WIKIWYG, and it was good. Then in the evening, Ingy and I spent a few hours trying to get WIKIWYG integrated into PurpleWiki.    (JYR)

We didn’t quite make it. Our biggest roadblock was a bug we discovered in Mozilla’s design mode that we can’t do much about. (My days of statically typed languages are well behind me.) But, we got something somewhat working, and I learned a heckuvalot. You can play with our semi-working demo.    (JYS)

WIKIWYG seems well-architected and is easy to customize. For folks with relatively standard Wiki editing requirements, I highly encourage you to play with it. PurpleWiki has some special formatting funkiness (mainly due to the Purple Numbers), but we were able to get around this fairly easily. (This was also true thanks to PurpleWiki‘s model of parsing to an intermediate data structure, then using view drivers to serialize. I wish more Wiki engines did this. I know Magnus Manske is thinking about doing this for Mediawiki, and I think Janne Jalkanen is already doing it with JSPWiki.)    (JYT)

The Mozilla bug annoyed me, because it’s a show-stopper in some ways, and there’s not much I can do about it. I didn’t realize it, but all of the JavaScript WYSIWYG widgets actually switch to the browser’s “design mode” in order to handle WYSIWYG editing. As with many HTML editors, design mode does not handle structure cleanly, and you end up getting weird artifacts such as spurious break tags. Our problem was that we serialize node ID information as id attributes in the HTML tags. However, Firefox does not maintain those attributes correctly when you move content around.    (JYU)

I’ll report the bug (if folks have suggestions as to the best way to bring this to the right people’s attention, let me know), but it also puts the kibosh on my hopes for WIKIWYG and Purple Numbers. Even if the bug is fixed in the next version of Firefox, we’re still prey to all the folks using older versions as well as Internet Explorer or Safari, which have their own problems with design mode.    (JYV)

Chris and I discussed one workaround that I’m still pondering: render the Purple Number and have users be responsible for maintaining the association with the nodes. That’s the status quo, except users are doing it in WikiText rather than in WYSIWYG. Doing it in WYSIWYG certainly lowers the bar, and it’s probably the next best thing for us to do.    (JYW)