Creole Wiki Markup, WikiOhana, and More Ward Wisdom

Posted on Published by Eugene Eric Kim

Several folks asked me on Saturday what I thought about the conference, and I kept saying, “It’s been great, except I haven’t had any interesting conversations about Wikis.” That changed in an unexpectedly generative way on Saturday afternoon, resulting in a new little cooperative effort we’re calling WikiOhana.    (KZ1)

At Thursday night’s conference party, I got reacquainted with Chuck Smith, whom I had met at Wikimania last year. Chuck is the founder of Esperanto Wikipedia and was the one who first introduced Brion Vibber — now the lead developer of Mediawiki — to the Wikipedia community. At the conference last year, Chuck met Christoph Sauer, a German Wiki researcher, and the two of them started working together.    (KZ2)

At the party, Chuck told me that he and Christoph had been working on a proposed WikiMarkup standard, and I loudly grimaced in response. At WikiSym last October, I made it clear to several people that I thought trying to standardize WikiMarkup was a noble waste of time.    (KZ3)

First, people are really attached to their own markup. Previous discussions about standardization usually started and ended with, “Great idea. Let’s just standardize on mine.”    (KZ4)

Second, WYSIWYG in theory obviates the value of a standard markup from a user’s perspective. It seemed more valuable to work out a standard interchange language, which WYSIWYG widgets could use to interact with different Wiki engines and which would make data migration between different Wiki engines easier. It also seemed like it would be easier to come to agreement on an interchange language. In fact, folks have already worked out a good proposal for an XHTML interchange language on the Interwiki mailing list (now migrated to the wiki-standards list) and on CommunityWiki.    (KZ5)

What made all this talk worse for me was that I thought that people were trying to go about this the wrong way. We were not going to come to agreement by group authoring a spec, then getting everyone to agree on it. The only way this was going to work was if a few Wikis came to some agreement and actually implemented the changes. This wouldn’t necessarily result in a standard, but it could act as a catalyst that could lead to a standard. Sunir Shah has been advocating this approach for a long time, to no avail. I would have organized such an effort myself, except that I didn’t think it was important enough to spend any time on it.    (KZ6)

Chuck and Christoph changed my mind. Chuck explained that he and Christoph had extensively analyzed existing Wiki markup, and they had taken pains to come up with a small subset of things that would be as noncontroversial as possible. (There had been a similar effort by others to do such an analysis before, but it hadn’t gotten very far.) They were going to present the results at WikiSym in a few weeks, but they had prepared a poster for Wikimania. Still clucking my disapproval, I promised Chuck that I would check out his poster.    (KZ7)

On Saturday, I finally made some time to check out the poster. I studied it and found myself thinking, “Hmm. This isn’t bad.” Chuck came over later, and we started going over it in depth. I still have some issues with it (discussed below), but I told him, “You know what, this is almost good enough for us to adopt in PurpleWiki. But you really should find one or two other Wikis who might say the same.” We kicked a few ideas around, and then I said, “Let’s go see what Ward thinks.”    (KZ8)

So we rounded up Ward Cunningham, and he took a look and started asking questions. He seemed to be okay with the answers, because he said that if someone were to write a Perl function that converted from his markup to the proposed markup and vice versa, he would consider incorporating it into his Wiki.    (KZ9)

Creole Markup    (KZA)

Well, you know me. Low-hanging fruit, a spare hour before the next party. The long and the short of it is, half of the converter now exists. It’s called Creole, and it’s written in Perl. Currently, it consists of a brain-dead simple script that converts Ward’s markup into Creole Markup (Chuck and Christoph’s proposal) and some tests.    (KZB)

The name was Ward’s suggestion. Rather than call it “pidgin,” which is what it actually is (a non-native language cobbled together from two or three other languages), we chose to optimistically call it “creole” (a native language that emerges from a pidgin), which is what we hope it will become.    (KZC)

I have three issues with the current proposal. First, I don’t like the exclamation point syntax for headers, not because of the character itself, but because of the numbering:    (KZD)

!!!Heading 1 !!Heading 2 !Heading 3    (KZE)

As you can see, you need to remember that a top-level header has three exclamation points, not one, and you are only capable of having three levels of headers. The advantage of the equals syntax:    (KZF)

= Heading 1 = == Heading 2 == === Heading 3 ===    (KZP)

is that the heading level is represented by the indentation.    (KZQ)

Second, there needs to be a proposal for preformatted blocks.    (KZR)

Third, the spec isn’t rigorous enough. You have to answer questions like, “Will italics work across multiple lines or blocks?” The reason I didn’t write a Creole-to-Ward converter was that these questions were not yet answered.    (KZS)

Most of the time developing the Creole converter was actually spent reverse engineering Ward’s markup. Ward pointed me to a Literate Programming version of his source code — implemented in a Wiki, of course — which helped a lot. As it turns out, Ward’s been thinking a bit about how to write a good markup parser recently. Parsers are often on my mind as well, as we’ve been talking about rewriting the PurpleWiki parser forever. Plus, the topic came up at Hacking Days among the Mediawiki developers before the conference. We’ll probably noodle on this problem a bit both before and during WikiSym. Fortunately, Ward knows a lot more about writing parsers than I, as anyone who’s seen the PurpleWiki code can attest.    (KZT)

WikiOhana    (KZJ)

When we were discussing names, Christoph proposed “ohana,” which means “family” in Hawaiian. It has a wonderful connotation — respecting our individuality while emphasizing the bonds that keep us together in a positive, welcoming fashion. It’s also consistent with Ward’s original naming convention, as “WikiWiki” means “quick” in Hawaiian.    (KZU)

We decided to call our little cooperative effort, WikiOhana, of which Creole Markup is a first project. We hope the family grows over time.    (KZV)

More Ward Wisdom    (KZK)

While retelling an experience he had writing a parser, Ward said, “That’s when I learned how to write beautiful code: Get it working, then spend the same amount of time making it better.”    (KZW)

Fighting WikiSpam: Eaton and Shared Blacklists

Posted on Published by Eugene Eric Kim

WikiSym 2005 was awesome. Massive props to Dirk Riehle and the program committee for throwing an outstanding event and drawing tons of great, great people. With Wikimania last August and WikiSym this past week, the Wiki community is really starting to gel. And it’s about time. Can you believe Wikis are 10 years old?    (JXD)

Now the bad news: I walked away with some action items. How do I get myself into these messes?!    (JXE)

The first action item can be traced back to an ad hoc meeting that happened at Wikimania regarding WikiSpam. On August 6, a group of Wiki developers — me (PurpleWiki), Alex Schroeder (OddMuse), Brion Vibber (Mediawiki), Thomas Waldmann (MoinMoin), Sven Dowideit (TWiki), Janne Jalkanen (JSPWiki) — along with John Breslin and Jochen Topf, got together to discuss ways we could collaborate on fighting WikiSpam. Our goal was to identify the simplest possible first step and not to get mired in process discussions.    (JXF)

Since all of us were already maintaining URL blacklists, we decided to merge them and host it as a Sourceforge project. We agreed on a standard format (which I’ll document and post soon), and we agreed to send our respective lists to Alex, who already has scripts to slice, dice, and merge.    (JXG)

One of my action items then was to create the Sourceforge project. I did that immediately, but for some reason, the project was rejected. Thus began a month-long go-around with Sourceforge support where I tried to discover why they had rejected the proposal. In the end, the project was approved, and I never got an answer as to why it was rejected in the first place. At that point, I was mired in other work, and so I never followed up.    (JXH)

WikiSym was the kick in the butt I needed to follow-up. On Sunday, Sunir Shah hosted an antispam workshop, which about 40 people attended. First, Sunir reviewed techniques (many of which are listed at MeatBall:WikiSpam). Then we broke out.    (JXI)

In my breakout, I described what we had agreed on at Wikimania. Then Peter Kaminski described a very cute idea he had for making it easy to fight WikiSpam. In a nutshell, Peter suggested we write a simple drop-in replacement CGI wrapper that would filter a POST payload for spam and call the real CGI script — be it a Wiki, a blog, or anything else — if the payload were spam-free. Such a wrapper would enable users to install spam-protection for any CGI script without having to write a single line of code and without having to do any complex configuration. It wouldn’t require any special access to your web server, since it would just be a CGI script. And you could easily add other spam-fighting measures, such as throttling and IP blacklists.    (JXJ)

I thought it was a brilliant idea. So Peter and I sat down afterwards and whipped it up. Took about an hour. It’s called Eaton, it works, and it’s Public Domain. Peter Kaminski has already blogged about it, and there’s some important commentary there from Jay Allen, the creator of MT-Blacklist.    (JXK)

It’s a proof of concept, and it won’t scale. It can and should be improved, and I’d encourage folks to do so. Nevertheless, it’s pretty cool. Bravo to Peter for a very clever idea.    (JXL)

By the way, the first person to figure out the origins of the name “Eaton” wins a cookie.    (JXM)

Patterns at WikiMania 2005

Posted on Published by Eugene Eric Kim

When I first met Christine Peterson (now a Blue Oxen Associates advisor), she told me the story about coining “Open Source.” The sign of a good name, she explained, is when people naturally start using it on their own. At a meeting of developers and evangelists in early 1998, rather than argue strongly in favor of the term, she introduced it subtly. Although the response wasn’t enthusiastic at the first, everyone in the room found themselves using the term, and by the end of the meeting, they all agreed to evangelize it.    (JMS)

Similarly, my strategy for introducing and identifying patterns of high-performance collaboration is to subversively introduce patterns into various communities and then to listen. If people naturally use a pattern in conversation, the name is probably good and the pattern itself is probably real and repeating. As people become familiar with the concept, they are more likely to identify and name other patterns. Over time, the language shifts your thinking, giving you a cognitive framework for thinking about, talking about, and improving collaboration and collaborative tools. Moreover, the process itself is iterative and collaborative, which is both the right way to develop Pattern Languages and also another application of collaborative patterns.    (JMT)

I’ve been giving some variation of a stock talk on patterns for over a year now, including last week at Wikimania 2005. It usually consists of a quick introduction, a few examples, and an interactive portion where I tease out patterns from the audience. The audience banter is always the best part. It’s always different, and it’s provided me with entertaining anecdotes, new patterns, and better pattern names.    (JMU)

Last week, I mentioned four patterns that Wikis facilitate: Permission To Participate, Shared Display, Visible Pulse, and Working Draft. Tim Starling followed my talk with an overview of Mediawiki development, and when he mentioned their IRC channel, he said, “This is our community’s Visible Pulse.” I love it when the process works!    (JMV)

The discussion teased out other patterns, especially Celebration and Initiation. (Linda Rising and Mary Lynn Manns mentions both of these in their book. They have a better name for the latter, but I don’t remember it off-hand.) One person told a great story about both. His team met for the first time in Australia, and before embarking on their project, they brewed beer that they planned on drinking after they finished their project.    (JMW)

Other patterns observed and not observed at the conference and within the community:    (JMX)

  • The Celebration at the end of the conference was great, but the Initiation wasn’t particularly remarkable. This isn’t unusual for conferences, where Initiation tends to be in the form of a keynote.    (JMY)
  • One pattern closely related to Initiation is Introductions. At conferences, this generally comes in the form of name badges, which we had at Wikimania. I think there’s a huge opportunity for further facilitating this pattern at conferences, which is something Blue Oxen Associates is working on. Hacking Days also would have been significantly more effective if we simply went around the room each day and quickly introduced ourselves. It’s one of those patterns that sound obvious when you hear about it, but is often forgotten when actually designing an event.    (JMZ)
  • Conferences usually do Water Cooler well, and Wikimania was no exception. We had long lunches, a party on the last night, an IRC channel, and some organized activities in Frankfurt am Main for stragglers following the conference. In particular, the organizers did two things really well. The Haus der Jugend was an excellent choice of venue, because it was an intimate space where social interaction was practically unavoidable. Most of the participants stayed at the hostel, which meant that there was always an interesting conversation to be had by simply going downstairs and hanging out in one of the common rooms. Plus, most of us who stayed there also had roommates, which is not typical for the conferences I attend. The restaurants and bars — touristy though they were — were in very close proximity, which made it easy to grab a beer and talk. (Food is a closely related pattern.) Finally, Hacking Days served as an unintentional Water Cooler, at least for those of us who were not Mediawiki developers. Coming early is something the organizers should encourage more widely next time.    (JN0)
  • I’m very biased towards highly interactive event design, and it seemed like it would have been especially appropriate for a Wiki conference. That said, the panel format worked better than at most conferences, and I think the Wiki culture of Permission To Participate had a lot to do with that. The audience interacted easily with the presenters at all of the talks I attended, and most of the speakers did a nice job of incorporating feedback into their presentations. One thing they did not do well was actively promote and integrate the conference Wiki as a place to take notes and have additional discussions. Again, this seemed surprising for a Wiki conference — yet another indication that making patterns explicit is a good thing.    (JN1)
  • Another favorite — also found in Linda and Mary Lynn’s book — is Spotlight On Others. This runs rampant throughout the Wikipedia community, which is wonderful. Tim cited me when mentioning Visible Pulse, and by the next morning, Sunir Shah had incorporated Permission To Participate into his talk on conflict resolution. I found this time and again in people’s talks. One reason it occurred so often during the conference is that the organizers used a Wiki to develop the program transparently. People watched other people’s talks develop and incorporated their content even before the conference began.    (JN2)
  • A critical pattern, especially for inter-organizational collaboration, is Neutral Space. Wikipedia would not have been successful if it did not have an open content license (which facilitates Neutral Space) and, to some extent, if it were a for-profit company. At the board panel on the last day, several people brought up the question of online ads. On the one hand, ads have the potential of bringing in a tremendous amount of revenue, which could be put to good use for the community. On the other hand, it breaks the Neutral Space pattern that has served Wikipedia so well. The community was more or less split on this issue. My vote: No ads!    (JN3)

One last pattern that I both observed and missed was Users Talk To Developers, a pattern I first described in, “An Introduction to Open Source Communities.” Previously, I criticized the Mediawiki developers for not practicing it enough. With the whole conference finally behind me, I want to both soften and and strengthen my statement.    (JN4)

Many of the Mediawiki developers came to the project as Wikipedia contributors. Brion Vibber, one of the leaders of the project, probably never would have joined had it not been for the Esperanto Wikipedia, of all things. After having more time to interact and observe the developers, I think that on average, community interaction is more prevalent among the Mediawiki developers than it is with many other projects.    (JN5)

That said, it’s still not nearly what it can and should be. During the sessions on politics and developing countries, several panelists complained that the tools had a way to go to meet their needs, and yet, none of the developers were attending their sessions. Hossein Derakhshan noted that techies are generally not interested in issues outside of their sphere.    (JN6)

Not all the blame falls on developers, however. As great as it would have been to see more developers abandoning the technical sessions in favor of the more social ones, it would have been fantastic to see more Wikipedia contributors attend some of the technical sesssions. Both communities need to learn and respect each other’s language if they truly want to engage collaboratively. Bridges are critical to make this work. Note that this applies not only to Mediawiki, but to all Open Source projects.    (JN7)

WikiMania Hackfest Day 4

Posted on Published by Eugene Eric Kim

Bits and tids:    (JM7)

  • I didn’t plan my Hacking Days schedule very well. I missed most of the first day, when the Mediawiki developers apparently made progress on a new metadata design. Days 2 and 3, from which I based most of my criticism, focused on servers and reliability, an area to which I really couldn’t contribute, not because I’m ignorant, but because I’m powerless. This morning, they discussed Single Sign-On and usability, two areas that I do know something about. Sadly, I missed these sessions, because I was too busy spouting on and on about how we really can save the world. Owen Davis, Fen Labalme, Kaliya Hamlin, and the rest of the gang will undoubtedly kick my butt when they read this. In my defense, I managed to talk a bit about Identity Commons later in the day. I also plugged the FLOSS Usability Sprint, and met Zeno Gantner, who’s done some usability studies on Mediawiki.    (JM8)
  • I was one of the featured participants for the afternoon “Wiki developers informal discussion,” along with Ward Cunningham, Sven Dowideit, Christophe Ducamp, and Brion Vibber. Domas Mituzas, Wikimedia Foundation‘s head of operations, asked Ward, “Why Camel Case?” I won’t go into the explanation here — I have a long interview with Ward, to be published eventually, that explains this in detail — but you should know that hating Camel Case is a running joke among this community. I laughed along with everyone else, but when Sven mentioned his desire to remove Camel Case from TWiki, I felt compelled to pipe up. I gave a balanced defense, describing Camel Case’s advantages over free links, but also acknowledging the appropriateness of free links in Wikipedia. Then I got a very amusing introduction to Erik Moeller, one of Mediawiki‘s core contributors and the Wikimedia Foundation‘s chief research officer. Erik had a strongly worded response. It got a bit heated, but never overly so, and I closed by saying that we were in violent agreement. We laughed about it over dinner, but then we got serious again. We also talked about Purple Numbers. I’ve explained many times why I may seem like a poor evangelist, but I think Erik was one of the few people who appreciated my perspective. He was clearly not a big fan of Purple Numbers — as it turns out, he was somewhat familiar with my work — but after hearing my explanation, he responded, “Only intelligent people are going to understand what you just said.” Fair enough. Fortunately, regular folks don’t need to get Granular Addressability for Granular Addressability to become ubiquitous.    (JM9)
  • A group of us broke out into a small group to discuss a Wiki Interchange Format, knowing full well that this is an issue that’s been discussed many times before (Wiki:WikiInterchangeFormat, MeatBall:WikiInterchangeFormat). Nevertheless, I think our discussion was not only constructive, it has a high chance of succeeding. See my summary.    (JMA)
  • Magnus Manske, the original creator of Mediawiki, participated in our Wiki Interchange Format discussion. He also mentioned a clever idea: a “shopping cart” where people could aggregate and possibly export Wiki pages they were interested in.    (JMB)
  • Sven Dowideit demonstrated the prototype WYSIWYG editor for TWiki, based on Kupu. He also showed a WikiText editor with real-time preview, which was pretty slick. Also, Ross Mayfield showed me a prototype editor for KWiki in response to my previous post. Very good to see these things.    (JMC)
  • So many people have come to this gathering to learn from others with different experiences. Granted, all of these experiences center around Wikipedia, but I’m still envious. My neverending quest is for folks interested in collaboration to look beyond their own narrow domains for deeper insights.    (JMD)

WikiMania Hacking Day 2

Posted on Published by Eugene Eric Kim

Tidbits from the day:    (JLA)

  • There’s now a conference blog.    (JLB)
  • The main order of business was Wikipedia‘s servers. As expected, they deal with some wild challenges. One amusing exchange occurred between Brion Vibber, one of Mediawiki‘s core developers, and a guy from MySQL. Brion asked if MySQL will ever support four-byte characters. The MySQL guy just sort of looked at him, stunned, said no, then asked why they needed it. Apparently, they do. There really are all types of folks using Wikipedia.    (JLC)
  • We digressed for a bit this morning to talk about WikiSpam. I’m going to try and coordinate a deeper discussions with some Wiki developers here to collaborate on a shared blacklist.    (JLD)
  • Speaking of “all types of folks,” here’s yet another reason why I’m loving this conference. Sixteen of us went out to dinner tonight, representing 11 countries: the Netherlands, France, Germany, Venezuela, Chile, Brazil, China, UK, South Africa, Australia, and the U.S. By the way, this morning I mentioned Delphine Menard‘s trilingualism. Try fluency in six languages. I was surrounded by multilingual folks today; another fellow was fluent in something like nine languages. It was both fun and humbling.    (JLE)
  • The conference is expecting over 300 attendees and lots of press, most of whom will start converging on this sleepy hostel tomorrow. Wikipedia is a very big deal in Germany.    (JLF)