Introducing Syncato

OK, it took a little longer then I expected, but I think Syncato is finally usable. In case you missed it, Syncato is the new Weblog system that runs this site now. It is written in Python and uses Sleepycat Berkeley DB XML for its database. The system is heavily based around XPath manipulation of XML data. All presentation is handled through XSL-T and I built some extension modules that tie Berkeley DB XML into libxslt.

This project started out as an experiment to see if I could do anything useful with Berkeley DB XML and since I was looking to replace MovableType anyway I decided I'd just role my own Weblog system. Overall I'm quite happy with Berkeley DB XML and the system has turned out to be quite promising. It's still missing some features like comments and trackback support, but the other major weblog features are supported. The interesting thing about the way I built this is that adding features like Categories and RSS feeds required no Python coding at all. They're simply XPath queries that are passed through an XSL-T stylesheet.

I decided to build the system with a REST style Web Service interface. This means that all actions against the system are done via the basic HTTP methods of GET, POST, PUT and DELETE. Building the system this way has allowed me to build on the basic architecture of the web rather then having to roll everything into an XML-RPC or SOAP style. For instance I realized that once I had PUT working, to update content in the database, I could use any PUT enabled HTML editor to edit the content. I've been using the editor that's part of Omniweb for this purpose. For instance to edit the previous post you would just access this URL and then use the view source option in Omniweb to edit it. Authentication is handled via standard HTTP authentication mechanisms.

Now the really interesting feature of this system is that it's really an XML database Web Service. I exposed an XPath query facility through the URL so that the database can be queried via HTTP GET. As you already saw you can get the XML source for a post via a URL, but you can also query the database in a similar manner. This means the database is pretty open for read access, write is protected however. Here's a few examples, keep in mind most of the results are XML not HTML.

• A list of all pages I've ever linked to as XML http://www.xmldatabases.org/WK/blog/item//a
• All items posted on August 14 as XML http://www.xmldatabases.org/WK/blog/item[starts-with(pubDate/text(), '2003-08-14')] as HTML http://www.xmldatabases.org/WK/blog/item[starts-with(pubDate/text(), '2003-08-14')]?=front-page
• The titles of all posts made on August 14 as XML http://www.xmldatabases.org/WK/blog/item[starts-with(pubDate/text(), '2003-08-14')]/title
• The title of all posts made on August 14 as text. http://www.xmldatabases.org/WK/blog/item[starts-with(pubDate/text(), '2003-08-14')]/title/text() When the query results are just text it does not return an XML document.
• Of course we could also just use the query that I use to build the category pages, here's the query for the Python category. As XML http://www.xmldatabases.org/WK/blog/item%5Bcategory='Python'%5D and the HTML that you usually see http://www.xmldatabases.org/WK/blog/item%5Bcategory='Python'%5D?t=category&a=Python. You can also get it in the same style as the frontpage, http://www.xmldatabases.org/WK/blog/item%5Bcategory='Python'%5D?t=front-page&a=Python. Category RSS and such is also possible.

The possibilities of this are endless, especially as you add more meaningful markup to your posts. This is something that I will be doing more of as I go forward. This also means some interesting possibilities exist for adding new functions to the system. To this point I've just been focusing on the basics to reproduce the functions of my old system, in the future there should be some more interesting stuff to come.

The system was originally written as a simple CGI, but that proved to be way too slow on the 6 year old machine (PII 266Mhz) this site runs on. The front page took up to 14 seconds to render. Which is clearly way too slow. It was obvious the XSL-T process was a large percentage of this so that was my first target. I added caching of the result which cuts the time down to about 6 seconds. From there I moved the system to run under Webware to cut out the startup overhead on each request and this combined with the caching results in most requests being served in less then a tenth of a second. This is good enough for now. It was trivial to move to Webware since I had been figuring it would be necessary from the time I started. Cache misses are of course expensive since the machine is still slow, but I'll have to live with that until I move to a faster machine.

Overall I'm really happy with the way things came together. It wasn't problem free though. In particular I had to abandon the idea of using Textile to format the weblog posts. The flat text model of Textile is too incompatible with the processing done by XSL-T when you also want to be able to process XML contained within the content of the post. This was disappointing as I did originally have it working without further processing of the XML. There's still hope I may be able to figure it out.

Posted by Kimbro Staken