Credits

Facebook offers a kind of generic currency – facebook credits. This is still in closed beta, but according to Mark Zuckerberg, you can already get in touch with them to get signed up. Again, this makes developing a payment-enabled web application much easier. And it probably will make a huge lot of money for facebook.

Open Graph API

In short, there’s a new API called »Open Graph«. Apparently, this API is much more simple that the old facebook API which is why I believe this will be a very powerful system. It will also save me from doing a long research which of the .NET facebook API wrappers is best – they are all obsolete as of now. Facebook promises that you won’t need to change your API calls in the future though. In the past, there have been quite a number of revisions of the API, often with breaking changes. The new API promises to be a lot more simple, but I have quite a set of questions in the back of my head as of now.

This simple API is also the web-wide return of the IFRAME. And here is my like button, with the evil color scheme (which looks just the like the normal?):

Note that this won’t show anything useful if you don’t have a facebook cookie set, which is quite annoying I believe.

If you want to have the your own web-wide “Like” button, here you go:

<iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fwww.example.com"></iframe>

That’s it. Fin. The official cooperation partners has the like button yet. You can find the new facebook like button documentation on their platform, which will also generate above code for you.

Due to the fact that this is an IFRAME, the hosting site of the IFRAME really doesn’t get any information about you, which might be interesting in case you don’t really trust the site. However, facebook will know you visited that site – kind of a problem for those pages you don’t really trust.

Most importantly, however, the Open Graph is a very strong move towards the semantic web. Instead of storing lots of free-form information, facebook’s open graph will allow to store semantically enriched information. You can see the new facebook like button also on pages such as imdb.com and if you “like” a movie there, it will pop up in your facebook profile – under the movies section! This is not just a gimmick.

This will be one of the most fundamental developments we have seen in the Internet. It’s easy to say that because Tim Berners-Lee predicted it. Facebook brought together some really important players and they will massively grow that semantic graph. Therefore, facebook’s content and ad-targeting will soon outperform anything we’ve ever seen on the net. Amazon suggestions? Google AdWords? Forget it!

To use a not-so-good example (see comments): “Golf”. If you google for “Golf”, Google has no idea what you mean. The sport? The car? Or one of the other uses of the word “Golf”? That’s why Google is keen on storing some additional information, especially your search history. If you have searched “stableford” before, you’re likely to be looking for the sport “Golf”. But the thing is: Facebook will be able to know even if you haven’t explicitly searched anything before, in case you (or your friends) are fans of the sport on facebook (or perhaps of a Volkswagen fanclub…).

What’s left in that puzzle, of course, is user behaviour: Do users really want this? Will they actually “like” everything around the world’s webpages? I guess so – after all, you’re not giving away anything it seems. You can interact with others, make a (small) statement and above all, be kept up-to-date through web hooks. So there is something to gain and little to lose for end-users – apart form getting a lot more transparent.

I’m very interested to see the full ontology they developed for Open Graph. So far, the examples shown are quite simple:

<meta property="og:type" content="band" />
<meta property="og:title" content="Green Day" />
<meta property="og:genre" content="Punk" />
<meta property="og:city" content="Berkeley" />

Open Graph API Call Examples

These examples have been presented by Head of Platform Products Bret Taylor at f8 – you can check them out just by clicking, of course:
http://graph.facebook.com/btaylor
http://graph.facebook.com/btaylor/groups
http://graph.facebook.com/btaylor/likes
http://graph.facebook.com/btaylor/videos

For example, when you call the first one, you’ll receive some basic information about Bret Taylor:

{
   "id": "220439",
   "name": "Bret Taylor",
   "first_name": "Bret",
   "last_name": "Taylor",
   "link": "http://www.facebook.com/btaylor"
}

You will need an authentication token for some of these, however. Speaking of authentication: facebook will move to OAUTH 2.0 in the future!

Web Hooks

“Don’t call us, we call you”: facebook will allow you to register Web Hooks. That will enable you ‘get a call’ from facebook every time certain information changes, so this allows you to synchronize with information as soon as they are updated in facebook by some user and vice versa. This is very interesting, but I’m very interested to see what limitations we will be seeing with this – I can think of several ways to abuse this.

Really Open?

The main question here is whether this graph is really open. As Zuck said, you can “write back information into facebook’s open graph”. Well. So the information goes back to facebook – this is quite beneficial for facebook, of course. On the other hand, facebook changed their policy so you’re no longer required to “forget” all user information after 24 hours. You’re now allowed to persist that information if the user allows you to do so.

One of the things that really trouble me, however, is that this will have facebook aggregate a LOT more information about you. Suppose you visited some site that you don’t want to share with your friends and that you really wouldn’t want to tell anyone about — facebook already knows it.

As of today, quite a number of projects that use facebook’s new technology are going to launch, among them Docs.com – an interesting product by Microsoft that let’s you share your office documents with your friends. Docs.com Apparently, Microsoft’s investment in facebook includes some large-scale co-operations.

They also added fancy charts (although I suspect these don’t work accurately yet, because of that peak you see in the screenshot) that look like Google Analytics a lot. The downside is that clicking on google search results will link to another google page which will then redirect you to the target – not exactly big trouble, but annoying when you’re used to copying links into your blog posts

That Domain-Issue

There is something peculiar about this I really don’t understand: If the domain is actually owned by the individual who suggested the concept, the FogCreek will never have a reliable revenue stream because the owner of the domain is the master — he could simply switch to a different technology, if he manages to keep the links intact, which might be a very important technology-factor when choosing some alternate technology.

On the other hand, if the domain is supposed to be owned by FogCreek, they have to buy every domain that is being suggested in the forum that exact second, otherwise domain grabbers will just … well, grab it (or even other users signed in). Democratic voting for domain names? I don’t think so. Personally, I would never even write down a domain name -especially not in a forum- if I like it and it’s not taken, I’d simply buy it – that gives me a year to think about it.

Not Convincing

Summing it up, I don’t think the move to their new model is a wise decision. My primary concerns are:

• The net is a trial-and-error place. You set something up, and see if it works. That might not be a wise model, but it is very democratic and has proven to work fine. Perhaps because it is not so ‘wise’. Note that wisdom grows from experience, inherently incapacitating certain types of innovation, namely those that completely contradict experience made.
• The code remains closed, so real customizations are not possible. This is a major flaw. I’ve got a dozen ideas of sites that could become really, really helpful to people, but they need some additional features or some heavy customization that is a major pain if you can’t access the sources. They don’t even have an XSLT processor you could use as of now…
• If a page does not offer any way to gain revenue, you can’t invest in it’s development, both technical development and marketing, where the first is already inhibited by the fact that customization is not possible from the technical side.
• This move will dramatically increase commitment to the development of open source clones such as OSQA, which looks very interesting by the way. There is also Shapado, and I think more will come. Crowd development can be extremely efficient and fast.

For all who operate a StackExchange site as of now, there is the cool thing that the site will operate for another three months, or if you have some real action going on, one year for free. That is great, because it allows for an easy transition to some other technology. It’s a bit sad to see that happen to such great technology, but I don’t believe this will work. Building a community remains a lot of effort, and people won’t put that effort in if they can’t call it their baby, can’t make any revenues, can’t make any customizations to the code, and can’t use the page to direct some traffic to other projects, I believe. We’ll see how it works out. I guess I will establish a small knowledge-base / link list at wikorum.net to aggregate some related information.

Motivation

One thing up front: We don’t need an object database for the simple tools we currently build, but we felt it was a good idea to get some acquaintance with the technology, because we will certainly need it for our (stealth) startup “pactas” soon. In pactas, the data structure is really complicated (with a fancy class hierarchy) and it probably will change very frequently.

Also, since I am such a big fan of reusability, we really develop a web engine – a framework that allows us to reuse a lot of the code for different projects and make sure most of our code has been tested thoroughly in the field. This proves to be a significant design decision when it comes to data modeling, and while I’m very happy with the decision, it certainly makes development harder.

The Problem

So, in time with the release of our simple backlink tracker, one of our development database files started to show strange behaviour – a certain query (via LINQ) would not order objects anymore – the orderby-clause seemingly was completely ignored. Our live server even came up with an “Invalid DateTimeKind specified” exception when trying to perform the query! What’s worse: The problem kept occurring from time to time, but it was not reproducible! Byzantine errors are clearly my favourite…

We thought the issue might be related to the current development/unstable versions of db4o that we were using (7.12 and 7.13). Using the stable version of db4o (7.4) proved difficult, because the old LINQ provider falls back to LINQ to Objects very often, which requires to fetch all objects in question from the database – that is very, very slow for a lot of objects, so we had to abandon that. We clearly wanted to stick to LINQ for a number of reasons (compile-time checking, readability, reusability).

Obviously the problem is related to the orderby operation on DateTime fields. I tried to modify the query, removed grouping because I feared it might be unstable (warning: the sort operation is, in fact, unstable! Unstable grouping would be useless, but the grouping is stable so that’s fine), even debugged the db4o code, but I couldn’t find any problem in there. Since the code is rather complex and that was the first time I took a look at it, I was happy to find some of the relevant code at all. Somewhere in the deeps of it, something screwed up. I didn’t want to spend too much time on that since I had to chew some particle physics on the side.

Solutions?

At the time of writing this (in fact, yesterday) I had something like a hotfix, but it turned out it’s completely nonsense – it worked around our internal bug in a very peculiar way. No more, no less.

When talking about this, somebody mentioned that it wasn’t such a good idea to use DateTime at all, and we should store the ticks instead. That lead to two long discussions with my co-ed Christian. We concluded: First, the power of object databases is that they do not force you to hack around and find different representations for your data (which raises the bar for object databases). The one big shortcoming of SQL is that it forces you to find a second, equally good, but different representation of your data and you need to translate between these representations all the time. You need to synchronize them. And you lose a lot of fancy features (such as lists, generics, inheritance, etc.) on the way.

Secondly, Christian suggested that object databases suffer from leaky abstractions badly, in that they break encapsulation in a way that leaks out a hell of a lot of implementation details. As Joel puts it: “All non-trivial abstractions, to some degree, are leaky.” The point is: With the error we’re currently encountering, it’s becoming a real problem. This is not an imperfect piece of architecture, it’s an exception in a database query. It kills the app dead! Boom!

In order to get activation [depth] straight, db4o needs to know how .NET’s containers work internally – that is clearly a detail that should be hidden, but db4o knows about it. db4o also takes care of that, but it leads to some messy issues. There is special code in db4o that handles containers, non-trivial objects such as strings, DateTime (which are non-trivial because they use 62 bits for the actual ticks and 2 bits for the DateTimeKind) and Lists. This is an implementation detail of the .NET framework, and it might change over time. There’s been a bug with Map, and I’m almost sure there is a bug with DateTime, too. Don’t get me wrong: The fact that some kind of mapping is needed is a somewhat generic (if not the) problem of serialization, it’s not really db4o-specific, and cannot be eliminated. In Hibernate, there is also a lot of code that handles the mapping of lists and the like but it doesn’t rely on implementation detail, thus it’s not (as) leaky.

Back to SQL?

Here’s the thing: I’d be beneficial to have a storage that is independent of the actual implementation on top, because it decouples the data store from the application which is not what db4o does. But wait, that is exactly what SQL is, right? Indeed: SQL forces you to map (or to cut down) your stuff to it’s internal features. A list becomes a foreign key on the other table, but that doesn’t play well with derived types, generics, etc… This is tricky as we all know, and you need lots of code to do that.

Worse, SQL forces you do that mapping for everything, including your own classes, and it demands a schema for every type of object – this is not what I want. I’d like to see a set of base objects in an object database which are natively understood by the DB. These objects can be mapped from and to by a layer which may be part of the db, or can be added manually if you need something very special. Still, it should allow to store your object in the database with all it’s magic, only that known objects will be translated, e.g. a DateTime will be stored as long Ticks and a DateTimeKind flag separated, making comparison operations easier (note that the comparison only works on the ticks: Whether the time is local or UTC is not considered by .NET in comparisons). Lists will be unfolded into an internal tree representation if indexed, so they become much easier to cope with for the database. That would also make managing m:n relations easier, since they could then be viewed as bilateral relationships – something you often need. Right now, they’re unilateral, thereby requiring some additional management on your behalf.

Migrating to SQL is clearly an option for this product, since data couldn’t fit SQL any better, but that doesn’t solve the issue for our pactas project, where we certainly will need schema-less data, complex object hierarchies, etc.

However, there are a few issues that remain unresolved as of now and they do qualify as show stoppers :

• There’s an irreproducible bug that potentially wrecks the db
• There’s a nasty performance issue with larger amounts of data

Alternatives

I’m quite dissatisfied that we have to abandon db4o at this stage, because I believe it’s the best kind of serialization I’ve ever experienced. It’s perfectly simple, it’s fast and most importantly, code-centric. If reusability is a concern, having the database structure and/or the ORM mapper dictate the objects is a major pain.

Entity Framework 4 (EF4) promises to handle this a lot better through “Model First” and “Code Only”, but I am still a bit afraid of EF because it doesn’t appear to be anything near lightweight and we will need schemaless storage for our future products anyways.

Versant, which bought db4o some time ago also offers its large-scale object database, which seems to suit large web-applications a lot better. First, it’s certainly made for huge amounts of data (unlike db4o, which comes from an embedded background and is aimed at database sizes in the low GB area, but supports up to 254 GB per file) and handles multi-threading better. Also, since db4o now moved to v3 of the GPL, it may not be freely usable in non-open source web applications anymore, so both solutions now have a price tag.

There is also an ISV/OEM empowerment program for Versant’s Object Database, which seems to make it affordable, but I haven’t looked at it in detail yet. Over the next weeks, I will have to evaluate a few of those other NoSQL solutions such as Cassandra and MongoDB, just to name two totally different options.

Aftermath

So what was the issue, after all? db4o did not screw up, we did: Take a blend of local and UTC time based on a completely random criterion, add two spoons of daylight savings time changes, add some misconfigured timezone on the server, bake for two weeks at 500 ° C and pling! You got yourself some really strange issue cake. Lessons learned:

• Debugging issue with object databases is harder than with RDBMS because the information is not chopped up.
• Do code reviews. Do code reviews.
• There aren’t too many alternatives to db4o, after all
• With the elaborated architecture and code-centric design we currently have, going back to SQL is a huge pain, and we won’t do it

All serialization is painful.

This is an interesting experiment… I’m interested to see how it works out