Sesame 2.0-alpha4 released

We are very pleased to announce the fourth (and hopefully final) alpha-release of Sesame 2. Sesame 2.0-alpha4 is a release that fixes a large number of issues with the previous alpha releases. It features major improvements in the Repository and SAIL APIs and a completely redesigned query engine for SeRQL and SPARQL. Additionally this release finally has a HTTPSail (thanks to Leo Sauermann, Gunnar Grimnes and Sebastian Weber) which enables transparent client-server communication with Sesame repositories over HTTP.

Sesame 2.0-alpha4 can be found in openRDF.org's download section.

A lot things have changed, most importantly:

• Sesame is now available under a BSD-style license, previous releases are still available under LGPL.
• The Repository and Sail APIs have been revised to use JDBC-style connections.
• The SeRQL query engine has been been replaced with an all new generic query engine that is modelled after the relational algebra for RDBMS's.
• Sesame now has partial support for SPARQL. Ask and describe queries are not yet supported, as well as a number of the SPARQL built-in functions.
• Sesame now has an HTTPSail, which allows clients to transparently communicate with a server over HTTP.
• The semantics of SeRQL set operations have changed: the results of the operands are now combined based on the variable bindings they produce. This means that you have to take care that the operands use the same variables names (or use aliases).
• The REST-style HTTP protocol has been improved. The new protocol is NOT backwards compatible with older versions.
• The Sesame code has been completely modularized into logical units, which makes it easier to reuse parts of Sesame in other projects.
• The Ant-based build system has been replaced with a Maven-based build system.
• The source code has been moved from SourceForge's CVS server to Aduna's SVN server: https://src.aduna-software.org/svn/org.openrdf/

Known issues

• This release contains a HUGE amount of changes to the code and, as a result, is expected to contain a considerable number of bugs.
• The new query engine currently lacks a query optimizer. Because of this you may see inferior query evaluation performance compared to earlier releases.

See the ChangeLog for details on the changes and improvements.

Sesame goes BSD

Although we haven't made an official announcement about this yet I thought I'd mention it: after a lot of discussion, both internally at Aduna and with various co-developers and users of Sesame outside Aduna, we have decided to change the open source license under which Sesame will be distributed.

Previously, Sesame has always been distributed under the GNU LGPL license. Although we have always interpreted this license as a very free license that allows reuse of Sesame as a library in any kind of client application, some users were wary of the consequences of including LPGL in our code. I don't particularly want to discuss here whether or not they are right of being wary, but it's a fact that it prevented some people from using Sesame.

So we have decided to move Sesame to a less restrictive license, namely a BSD license (that's about as unrestrictive as you can get I guess, bar public domain). The current Sesame 2 code base has been adapted (headers, license text) and the next releases of Sesame 2 (including the alpha-4 release that will be released this week) will all be available under BSD. Existing releases (including all Sesame 1 releases) will stay available under LGPL only for now.

The main difference between the licenses is that LGPL has a reciprocity-clause which BSD does not: basically, if you change something in Sesame and you distribute that changed version, the LGPL requires you to release the changed version as LPGL and so contribute the change back to the community. BSD does not require this.

I'm personally very happy with this decision. A license that is as free as possible allows as many people as possible to use and co-develop on Sesame. A broad user base for Sesame is in our best interest of course, both as developers (more bug-reports and hopefully more fixes) and as a company (more uptake means more exposure). Of course it also neatly ties in with Aduna's strategy of releasing all Aduna software as open source.

I'm not particularly worried about people taking our stuff and improving it without giving back: the history of many other BSD-licensed projects indicates that when developers get excited over a piece of software, they are typically happy to contribute, regardless of what the license requires. So now our main goal is to get as many developers as possible excited about Sesame 2 :)

openacademia: semantics-based publication management

openacademia is an initiative to collect, share, publish and manage bibliographical information, the Semantic Web-way.

Information about scientific publications is often maintained by individual researchers. Reference management software such as EndNote and Bibtex help researchers to maintain a personal collection of bibliographic references. (These are typically references to one's own publications and also those that have been read and cited by the researcher in his own work.)

Most researchers and research groups also have the tedious task of maintaining a web page about publications for interested colleagues from other institutes. Typically, personal reference management and the maintenance of web pages is a separate effort: the author of a new publication adds the reference to his own collection, updates his webpage and possible that of his research group. From then on it is waiting for other researchers to discover the newly added publication.

The openacademia system removes the unnecessary duplication of effort involved in maintaining personal references and webpages. It also solves the problem of creating joined publication lists for webpages at the group or institutional level. At the same time it gives a new way of instantly notifying interested peers of new works instead of waiting for them to visit the webpage of the reseacher or the institute.

openacademia is a distributed system on its own. A public openacademia website is available on the Web for general use, i.e. anyone can submit his own publications to this service. openacademia can also be installed at research groups locally in order to collect and manage the shared publication metadata of the group.

The most immediate service of openacademia is the possibility to generate an HTML representation of one's personal collection of publications and publish it on the Web. This requires filling out a single form on the openacademia website, which generates the code (one line of JavaScript!) that needs to be inserted into the body of the homepage. The code inserts the publication list in the page dynamically and thus there is no need to update the page separately if the underlying collection changes. The appearance of the publication list can be customized by choosing from a variety of stylesheets.

For example, on my own homepage, the list of publications is generated this way.

More interestingly, you can also generate an RSS feed from the collection. Adding such an RSS feed to a homepage allows visitors to subscribe to the publication list using any RSS news reader. Whenever a new publication is added, the subscribers of the feed will be notified of this change through their reader (information push).

The RSS feeds of openacademia are RDF-based and can also be consumed by any RDF aware software such as Piggy Bank.

There is also an AJAX based interface for browsing and searching the publication collection. This interface offers a number of visualizations. For example, the important keywords in the titles of publication matching the current query can be viewed as a tagcloud , where the size of the tags shows the importance of the keyword. It is also possible to browse the co-authorship networks of researchers using the same interactive applet used by Flink.

Another interactive visualization shows publication along a timeline that can be scrolled using the mouse (see below). This time-based visualization uses the Timeline widget developed by the SIMILE project.

Keywords or tags can be added to publications using the features of Bibtex or EndNote. The system also extracts keywords automatically from titles of publications. Lastly, openacademia connects to blog search engines in order to import blog comments about publications.

openacademia is built on Sesame and the Elmo API. For more information about openacademia, please see the documentation.

Using context/named graphs for provenance and time

Seth Ladd blogs about the inclusion of provenance and time labeling in RDF, partly in response to John Barstow's earlier wish list for RDF-Lite. Seth makes an excellent point about how the time aspect can be easily modeled in RDF already, by creating an object for the event occurrence.

However, there are definitely cases where the solution illustrated by Seth is awkward. Consider for example an electronic archive which contains photos of buildings. The photos are annotated with the street address, but for some few buildings the name of the street has changed over the years (or in rare cases the building itself was moved). We can capture this by thinking in Nouns, for example something like this:

:addressOccurrence1 :ofBuilding :building1 ;
               :street "Church Street 24" ;
               :startYear "1939" ;
               :endYear "1948" .

:addressOccurrence2 :ofBuilding :building1 ;
               :street "New Church Street 17" ;
               :startYear "1948" ;
               :endYear "2006" .

Although we do accurately capture the time-aspect of the notion of address, we end up with a very un-intuitive model here. Most of the time in querying this archive we will simply want to consider the address of a particular building without caring about dates, it is only in some corner cases that the temporal aspect is important. However, the chosen modeling makes little sense if we disregard the temporal aspect, making it an ill fit (think of the kind of SPARQL query you'll need to get all current addresses of all buildings. It's possible but not pretty).

Anyway, whether the solution works well in all cases or not (and I'm sure there are other modeling solutions possible for the example I just gave), a tacit assumption in general seems to be that in order to incorporate both provenance and time in the RDF model, we need to extend from triples (subject, predicate, object) to not just quads but quints (subject, predicate, object, source, time) or something like that.

IMHO this is not necessary and a simple extension to quads allows us to do everything we want already. The trick is to not think of the fourth parameter as the provenance identifier but merely as a group identifier, which in its own right can then be used as the subject of many statements about the group (such as its source, and possible start and end times, or how Jeen feels about these statements, etc. etc.).

In fact, this is almost exactly how named graphs in SPARQL work: by offering the means to identify groups of statements by an identifier without assigning any semantics to that identifier we can then use that identifier to encode all sorts of properties about groups of statements, including provenance and time properties. What would still be useful as an extension would be standardized vocabulary for denoting provenance (e.g. rdf:source) and time labeling though. For the latter, something like the OWL Time Ontology might be useful.

Pitfalls in Benchmarking Triple Stores

Arjohn and I have recently been running some scalability tests on Sesame's various backends, and we found a number of interesting little tidbits.

It turns out that Sesame's (and probably every triple store's) performance is highly dependent on not just things like processor speed, amount of RAM, etc., but also very much depends on the structure of the RDF that you are using to test the system.

There are a number of aspects of test data that may influence results. One is the number of new URIs introduced per triple. When you upload a data set that contains three new URIs in (almost) every single statement, upload performance is going to decrease. This is a problem that you will most likely not encounter in most real-life datasets (since they typically make a number of assertions in a limited vocabulary about a limited set of objects, they tend to share a lot of URIs between statements). However, if you decide to test scalability with an artificially generated dataset, and you are not careful, you can get badly skewed results.

Another big one we came across has to do with namespaces. Sesame internally splits each incoming URI into a namespace and local name. It determines this split according to a simple algorithm, roughly: find the first non-NCName character (typically a hash, slash or column) starting from the end of the URI. Works great. Except, the Uniprot dataset contains URNs of this form:

• urn:lsid:uniprot.org:embl-cds:AAO89435:1
• urn:lsid:uniprot.org:embl-cds:AAF63733:1
• urn:lsid:uniprot.org:embl-cds:CAA40583:1

When we tried to add the Uniprot dataset to a Sesame store, Sesame dutifully split each URI into a namespace and a local name. Unfortunately, the split algorithm produces three distinct namespaces for the above three URNs (because it splits at the last column char), resulting in, basically, a new namespace for every identifier in the dataset (instead of doing a more intelligent split, in which we split at the second column from the back instead of the first, and end up with a single, shared, namespace for all these URNs). Because we never assumed the namespace set to become that big and simply cache every namespace in memory (to improve lookup speed), we ran into big performance problems after adding little more than 5 million statements. Conclusion: Sesame sucks!

On the other hand, we also ran some tests with the Lehigh University benchmark data set. This (generated) dataset does not have this particular URI-splitting problem, so we happily added 70 million triples to a native store, without hitting a ceiling. Conclusion: Sesame rules!

I am quite certain that we will fix this particular problem in Sesame quite soon (we'll work on the split algorithm and/or do something smart with caching). I'm equally certain that other "weird" problems in real-life data will crop up at some point that will trip up Sesame, or Jena, or Kowari, or Redland or any of the other stores.

What I'm trying to say here is probably obvious but I thought it bears preaching^H^H^H^H^H^H^H^H^Hrepeating (since scalability is such a hot issue these days): Don't Trust Benchmarks. Or at least, distrust them just a little. Always keep in mind that even though a particular tool may shine on a particular dataset, it does not automatically mean that it can duplicate that shining performance on any dataset.

Sesame 2.0-alpha-2 released

We've just released a second alpha version of Sesame 2. Major new things include:

• Java 5.
  All access APIs now extensively use Java 5 features such as typed collections. This not only makes Sesame's own code more robust, but also makes client code faster, easier, and clearer.
• Native Storage Backend.
  Sesame's B-Tree based binary on-disk persistence backend (a.k.a. the native store) has been ported to Sesame 2. So now we have fast, scalable and reliable persistence, and of course it has context support as well.
• Pull-based Query.
  The Sesame query model has been redesigned to allow pull-based querying: through lazy iterators, the client can now actively control when individual query results are returned (instead of having to passively consume whatever the server sends it).

More detailed information on changes, features, documentation etc. to be found at the Sesame website.