Pitfalls in Benchmarking Triple Stores

Arjohn and I have recently been running some scalability tests on Sesame's various backends, and we found a number of interesting little tidbits.

It turns out that Sesame's (and probably every triple store's) performance is highly dependent on not just things like processor speed, amount of RAM, etc., but also very much depends on the structure of the RDF that you are using to test the system.

There are a number of aspects of test data that may influence results. One is the number of new URIs introduced per triple. When you upload a data set that contains three new URIs in (almost) every single statement, upload performance is going to decrease. This is a problem that you will most likely not encounter in most real-life datasets (since they typically make a number of assertions in a limited vocabulary about a limited set of objects, they tend to share a lot of URIs between statements). However, if you decide to test scalability with an artificially generated dataset, and you are not careful, you can get badly skewed results.

Another big one we came across has to do with namespaces. Sesame internally splits each incoming URI into a namespace and local name. It determines this split according to a simple algorithm, roughly: find the first non-NCName character (typically a hash, slash or column) starting from the end of the URI. Works great. Except, the Uniprot dataset contains URNs of this form:

• urn:lsid:uniprot.org:embl-cds:AAO89435:1
• urn:lsid:uniprot.org:embl-cds:AAF63733:1
• urn:lsid:uniprot.org:embl-cds:CAA40583:1

When we tried to add the Uniprot dataset to a Sesame store, Sesame dutifully split each URI into a namespace and a local name. Unfortunately, the split algorithm produces three distinct namespaces for the above three URNs (because it splits at the last column char), resulting in, basically, a new namespace for every identifier in the dataset (instead of doing a more intelligent split, in which we split at the second column from the back instead of the first, and end up with a single, shared, namespace for all these URNs). Because we never assumed the namespace set to become that big and simply cache every namespace in memory (to improve lookup speed), we ran into big performance problems after adding little more than 5 million statements. Conclusion: Sesame sucks!

On the other hand, we also ran some tests with the Lehigh University benchmark data set. This (generated) dataset does not have this particular URI-splitting problem, so we happily added 70 million triples to a native store, without hitting a ceiling. Conclusion: Sesame rules!

I am quite certain that we will fix this particular problem in Sesame quite soon (we'll work on the split algorithm and/or do something smart with caching). I'm equally certain that other "weird" problems in real-life data will crop up at some point that will trip up Sesame, or Jena, or Kowari, or Redland or any of the other stores.

What I'm trying to say here is probably obvious but I thought it bears preaching^H^H^H^H^H^H^H^H^Hrepeating (since scalability is such a hot issue these days): Don't Trust Benchmarks. Or at least, distrust them just a little. Always keep in mind that even though a particular tool may shine on a particular dataset, it does not automatically mean that it can duplicate that shining performance on any dataset.