LESS CSS?! What?

Using LESS mixins, we can replace cumbersome CSS with a few simple statements. A mixin basically copies all properties over. For example, the following LESS will produce really huge, red text if applied:

.huge{
    font-size: 80px;
}

.hugeWarning{
    .huge;
    color: red;
}

OK, nice, but the CSS equivalent wouldn’t be very different. But you can also parametrize mixins, the classical example (taken directly from the LESS website) being rounded corners:

.rounded_corners (@radius: 5px) {
  -moz-border-radius: @radius;
  -webkit-border-radius: @radius;
  border-radius: @radius;
}

#header {
  .rounded_corners;
}

#footer {
  .rounded_corners(10px);
}

LESS in ASP .NET

If you want to use LESS from your shiny new ASP.NET MVC3 application, I suggest you grab a copy of Justin Etheredge’s brilliant zero-friction SquishIt library which compiles LESS to CSS, combines the files and minimizes them. Oh, yes, and of course also handles JavaScript combining and minification!

Now, since LESS also supports some mathematical operations, we can do all the nasty grid-calculation using LESS. For a fluid grid, this becomes something like this:

/* Fluid grid based on 960.gs syntax
This grid contains ONLY the fluid-container_12 class and it's associated grid, pull, push, suffix and prefix classes. Note that this CSS does not support nesting of grids.
*/

@columnCount: 12; // When you increase this, you also need to add some more classes below
@halfGutter: 1%; // half of the gutter width

.fluid-container_12 {
	margin-left: 0px;
	margin-right: 0px;
	width: 100%;
}

.grid (@n: 1)
{
	width: @n * 100% / @columnCount - 2.0 * @halfGutter;
	margin-left: @halfGutter;
	margin-right: @halfGutter;
}

.fluid-container_12 .grid_1{ .grid(1); }
.fluid-container_12 .grid_2{ .grid(2); }
.fluid-container_12 .grid_3{ .grid(3); }
.fluid-container_12 .grid_4{ .grid(4); }
.fluid-container_12 .grid_5{ .grid(5); }
.fluid-container_12 .grid_6{ .grid(6); }
.fluid-container_12 .grid_7{ .grid(7); }
.fluid-container_12 .grid_8{ .grid(8); }
.fluid-container_12 .grid_9{ .grid(9); }
.fluid-container_12 .grid_10{ .grid(10); }
.fluid-container_12 .grid_11{ .grid(11); }
.fluid-container_12 .grid_12{ .grid(12); }

.prefix(@n:1)
{
	padding-left: @n * 100% / @columnCount;
}

.fluid-container_12 .prefix_1 { .prefix(1); }
.fluid-container_12 .prefix_2 { .prefix(2); }
.fluid-container_12 .prefix_3 { .prefix(3); }
.fluid-container_12 .prefix_4 { .prefix(4); }
.fluid-container_12 .prefix_5 { .prefix(5); }
.fluid-container_12 .prefix_6 { .prefix(6); }
.fluid-container_12 .prefix_7 { .prefix(7); }
.fluid-container_12 .prefix_8 { .prefix(8); }
.fluid-container_12 .prefix_9 { .prefix(9); }
.fluid-container_12 .prefix_10 { .prefix(10); }
.fluid-container_12 .prefix_11 { .prefix(11); }
.fluid-container_12 .prefix_12 { .prefix(12); }

.suffix(@n:1)
{
	padding-right: @n * 100% / @columnCount;
}

.fluid-container_12 .suffix_1 { .suffix(1); }
.fluid-container_12 .suffix_2 { .suffix(2); }
.fluid-container_12 .suffix_3 { .suffix(3); }
.fluid-container_12 .suffix_4 { .suffix(4); }
.fluid-container_12 .suffix_5 { .suffix(5); }
.fluid-container_12 .suffix_6 { .suffix(6); }
.fluid-container_12 .suffix_7 { .suffix(7); }
.fluid-container_12 .suffix_8 { .suffix(8); }
.fluid-container_12 .suffix_9 { .suffix(9); }
.fluid-container_12 .suffix_10 { .suffix(10); }
.fluid-container_12 .suffix_11 { .suffix(11); }
.fluid-container_12 .suffix_12 { .suffix(12); }

.push(@n:1)
{
	left: @n * 100% / @columnCount;
}

.fluid-container_12 .push_1 { .push(1); }
.fluid-container_12 .push_2 { .push(2); }
.fluid-container_12 .push_3 { .push(3); }
.fluid-container_12 .push_4 { .push(4); }
.fluid-container_12 .push_5 { .push(5); }
.fluid-container_12 .push_6 { .push(6); }
.fluid-container_12 .push_7 { .push(7); }
.fluid-container_12 .push_8 { .push(8); }
.fluid-container_12 .push_9 { .push(9); }
.fluid-container_12 .push_10 { .push(10); }
.fluid-container_12 .push_11 { .push(11); }
.fluid-container_12 .push_12 { .push(12); }

.pull(@n:1)
{
	right: @n * 100% / @columnCount;
}

.fluid-container_12 .pull_1 { .pull(1); }
.fluid-container_12 .pull_2 { .pull(2); }
.fluid-container_12 .pull_3 { .pull(3); }
.fluid-container_12 .pull_4 { .pull(4); }
.fluid-container_12 .pull_5 { .pull(5); }
.fluid-container_12 .pull_6 { .pull(6); }
.fluid-container_12 .pull_7 { .pull(7); }
.fluid-container_12 .pull_8 { .pull(8); }
.fluid-container_12 .pull_9 { .pull(9); }
.fluid-container_12 .pull_10 { .pull(10); }
.fluid-container_12 .pull_11 { .pull(11); }
.fluid-container_12 .pull_12 { .pull(12); }

Nifty, huh?

Most importantly however, MVC3 introduces the Razor View Engine and separates MVC and WebPages, so the web rendering is no longer hard-wired into the ASP.NET subsystem.

Apart from its much cleaner syntax, Razor is also more flexible and it’s open source. Matthew Abbot and Ben of BuildStarted.com have written some interesting articles on how to use Razor without MVC and even without ASP.NET at all.

PDFs vs. HTML

Now, what does all that have to do with PDFs? Writing to PDFs itself isn’t necessarily too hard – you can easily draw some text here and there or draw lines, for example using the excellent iTextSharp library. But the real deal is typesetting!

Unfortunately, HTML and paper don’t make a very good fit. Breaking HTML tables across pages, for example, is an insufferable pain. HTML wasn’t made with paper in mind – a good thing for HTML, and a good reason not to abuse it for printing PDFs.

Introducing RazorTex

Now, Razor makes only a few assumptions about the type of data it renders – it doesn’t really have to be HTML… With RazorTex, you can create LaTeX-Views on the fly instead! A word of warning though: RazorTex is extremely young and unstable – more sample code than anything else – so be prepared! There’s a ton of features missing, but it works. If you’re ready for a rather rough ride, go ahead!

So how does it work, behind the scenes? RazorTex will use the Razor engine to compile code into a LaTeX file. A sample section from a .cstex file might look like this:

\frac{@Model.Factor.ToString("F2")}{@Model.Denominator.ToString("F2")} &= @Model.Result.ToString("F2") \\
\int_{@Model.LowerBound}^{@Model.UpperBound} @Model.Expression @Model.IntegrationVariable &= @Model.IntegrationResult \\

The '&=' and '\\' are special LaTeX-Commands, an alignment character and a line break, respectively.

If you don’t know LaTeX, the syntax can be really terrifying at first. However, there is no doubt LaTeX is extremely powerful when it comes to typesetting complicated document layouts. For example, it supports high-quality text rendering with microtypography, breaking tables across pages, typesetting mathematical formulas and rendering vector images, to name only a few. There are many additional packages you can use for practically any type of typesetting problem.

Running the Sample

Unfortunately, there are some smaller obstacles when running the sample. Also, you’ll have to install a bunch of things:

1. Download and install ASP.NET MVC3 Release Candidate from Microsoft. This includes the Razor view engine
2. Download and install MiKTeX 2.8
3. Download and install WinShell for LaTeX, an IDE for LaTeX [recommended]
4. Before running the sample, open the two included sample .TeX files and open them with WinShell. Run the PDF Compiler. This will install the required packages. If you don’t install the packages, the sample might dead-lock!
5. Adjust the settings in app.config according to your needs and create the temporary and output folders.
6. Finally, the application should run and create great pdfs from a console application…

The Architecture

Let’s take a quick look at the architecture of the Razor pipeline. Most importantly, Razor views are compiled – a huge advantage when you need to render a lot files with similar structure. However, we need to care for the compilation ourselves. Fortunately, Andrew Nurse has already accomplished that for us. For details, refer to his article. In short, we read the .cstex file, create a RazorParser and throw in our template string. This will create C# code which in turn can be compiled to binary using CodeDom.GenerateCodeFromCompileUnit. Of course, there are some subtleties involved such as assigning the correct base class, importing some namespaces, and so forth.

Once they are compiled, executing the latex view is blazing fast, but compilation takes some time. Therefore, it’s a good idea to cache these. Right now, there is some caching in RazorFlux, but it’s probably a good idea to compile all the latex-views in one go (and into a single assembly) – at least, that’s the way ASP.NET does it.

Additional Features, LatexHelper

Two particularly helpful features in HTML are the HtmlHelper and UrlHelper extensions. Of course, they don’t make a lot of sense in LaTeX, but there are some LaTeX-features where a custom helper would come in handy – a LatexHelper!?

Indeed, we can inject our own helpers through the base class:

Type baseType = (modelType == null)
            ? typeof(LatexTemplate)
            : typeof(LatexTemplate<>).MakeGenericType(modelType);

generator.GeneratedClass.BaseTypes.Add(baseType);

public abstract class LatexTemplate : ILatexTemplate
{
    public LatexHelper Latex { get; set; }
    public bool SuppressEmptyLines { get; set; }
}

So far, the LatexHelper is used primarily for “display templates”. Also, the LatexHelper certainly contains the worst hacks in the code. More about that later.

Drawbacks

Some disadvantages shouldn’t go unmentioned:

• While Razor doesn’t seem to make too many assumptions, it does assume that line breaks don’t play a role – true for HTML but damn wrong for LaTeX. I have had some trouble with that, but nothing that can’t be solved in the .cstex file. Still looking for a better solution though.
• Pdflatex isn’t too fast – there might be faster solutions for creating PDFs, but the quality of pdflatex is very high.
• It seems impossible to get pdflatex to work with streams instead of files, so you’ll have to write temporary files – no big deal, but a little painful. If anybody knows how to it, I’d be glad to know!

So I dug a bit deeper into MongoDB and NoRM, which is a nifty C# driver for MongoDB developed by Andrew Theken and several others. You might want to grab a copy at github, where you can see how incredibly active the project is! Now, back to the evaluation: what is the best way to evaluate a database? Of course, build a live product using it!

Since I was completely abusing db4o for said project (in fact, I am storing something you’d call documents there), I decided that this would be a great candidate for a migration. So now we’re migrating from an object database to a document database and from an ACID database to a NoSQL solution.

MongoDB is considered a NoSQL solution, while db4o is considered ‘soft’ NoSQL – see Stefan’s comment at the bottom. Why the distinction – both do not rely on, support or use SQL whatsoever?! But then again, that is not what the Term »NoSQL« is all about. It’s probably one of the most misleading terms ever coined and perhaps it should read »Not ACID« or just »Persistence without Prejudgement, PwoP«. db4o makes ACID guarantees, comes from an embedded background and offers single-server durability while MongoDB is made for the net, does not have single-server durability, supports MapReduce and is driven by JavaScript.

Hell, they couldn’t be more different. But then again, I can access both using identical interfaces:

// Linq is understood by both, so you could use the lines in both:
var u = (from Note n in container
         where n.Text == null select n).ToList();
var v = session.Query<Tag>().Where(p => p.Name != null);
// Store an object graph to Mongo using NoRM:
session.Add(testNote);
// Store an object graph to db4o:
container.Store(testNote);

Don’t be fooled – although these lines could all be used with MongoDB or db4o and all of them could even be used with the very same classes, they are still fundamentally different in behavior. Also, for anything but the most simple problems, you can’t just persist the same domain models.

What is a document now?

A document is not just an unstructured piece of data. It’s not a BLOB. Instead, an instance of a class, plus all it refers to, could be a document:

class UniqueIdObject
{
    public Guid Id { get; private set; }
    public UniqueIdObject() { Id = Guid.NewGuid(); }
}

class Report : UniqueIdObject
{
    public Report() : base() { Tags = new List<Tag>(); }
    public string Text { get; set; }
    public List<Tag> Tags { get; set; }
}

class Tag : UniqueIdObject
{
    public string Name { get; set; }
}

Now we might want to store a note in the database, and the code needed is just

using (NoRMSession session = new NoRMSession())
{
    Report newReport = new Report() { Text = "Hello World, MongoDB!" };
    newReport.Tags.Add(new Tag() { Name = "Tag 1" });
    newReport.Tags.Add(new Tag() { Name = "Tag 2" });
    session.Add(newReport);
}

That’s it! Wow! Of course, we haven’t taken care of indexation and stuff and since MongoDB is schemaless (or better, has a dynamic schema) we need to do that in code. But still, this is essentially all you need.

The important thing now is: What happens to those little `Tags` we deliberately put into a separate class? Now the mapper hides a bit of truth from us, because MongoDB works on so-called “Collections”. session.Add(newReport) will be session.Add<Report>(newReport), which will in turn put the newReport object into the Report-Collection! So the object graph, as it is, will be serialized into the Report collection, including our little Tag objects!

Each item with an orange border is an ‘atomic’ item in its respective data store:

db4o serialized graph

mongo serialized document

Let’s naively try to fetch all tags:

var v = session.Query<Tag>().Where(p => p.Name != null);

This does not work, v is null because there is no tag collection! Instead, the tag lists are part of the reports we put into the Report collection. Note that this would work in db4o, because db4o will store references as references, while ‘documents’ store the contained data instead of references. This is beautifully simple, but it’s also very different from what you might expect and it has lots of implications for your object structure.

Thoroughly think through your schemaless schema

MongoDB is made for scalability and simplicity, so it does not care for our foolish approach to fetch tags directly. There are ways to access that data directly, however. We could use a deep-graph query or write a javascript Map/Reduce instruction, but that is a bit out of scope right now. What’s more important is that it calls for changes to our domain model objects: If we really want to store a reference, we need to do so manually, in ye olde sql-way, by storing the associated Id instead of the object. Of course, that makes deserialization a bit more complicated because the object we now retrieve from the database aren’t ready-to-use as they come.

However, automating that process will induce several complications, among them the need to handle cyclic references, a concept for fetching or activating objects on-the-fly, called Transparent Activation in the db4o world and making sure we’re not inducing a massive performance hit along the way.

Also, updating objects can be painful. Suppose we stored a list of Reports for each user. Now we might want to put the list of reports directly into the user object, or store a list of ReportIds for each users and put the Reports into their very own ReportCollection. As usual, there is no silver bullet, so this decision depends on the specific needs of the application, but whatever decision we take will not be visible to users of the resulting objects. In fact, it leads to some unwanted strong coupling:

class ReportService
{
  private NoRMSession _session;

  // ...
  public void UpdateReportDetail()
  {
    this.ReportDetailXY = ComplicatedCalculation();
    this.LastChanged = DateTime.UtcNow;

    // If the reports are in their very own Report Collection, this
    // is fine. However, if they are contained as lists in the user
    // who owns them, we're in trouble and this will fail!
    _session.Update(this);
  }
}

This might not be a big problem for really small applications, and if you really need performance (why else would you choose a NoSQL system?) you have to fine-tune your objects anyways. Still, I have the feeling that there is some space for improvement here, and a basic wrapper could help in overcoming some of the issues raised. Some basic ideas on how to approach this will follow shortly.

Wrapping it up

Obviously, I’m comparing Apples and Oranges here: db4o is made to make persistence easier, especially with complex domain model objects. db4o, being an object database, behaves exactly the way you’d expect objects under serialization to behave, but that makes it quite complex. MongoDB is focused on simplicity and scalability. Through the document concept, you gain simplicity on the database, administration and wrapper (driver)-side, but you have to struggle with a slight impedance mismatch in code, especially against Linq, again.

Just to get that straight: I’m an avid db4o user and really love it. These issues are not critical and they don’t stop me from using or evangelizing db4o. However, I think there is some lack of awareness of some issues.

Also, I’d like to spawn some discussion about the issues below. Unfortunately, due to their changes to the forum system, most of the original discussions on the db4o forum are hard to find or possibly lost. You may want to vote on the issues you deem most pressing, which you can easily do in their issue tracker! I’m very interested in what you think about this little selection.

Selected Issues

» Don’t run SODA when no more constraints are present
I blogged about this already, because you experience this in very common scenarios, namely whenever you query a small subset of a larger candidate set. For example, consider selecting the last 50 posts on a blog/qa-site/etc. What will happen is that db4o runs the BTREE query for the sort operation (blazing), then hydrates (?) all objects, the returns the first 50 of them and throws away the rest. Thing is that there is no need to further inspect the items, and activating all them is basically a linear operation. Thus, this common type of query currently runs in O(n) instead of O(log n) which is an incredibly dramatic difference.

» LINQ-Implementation is not ‘thread’-safe
A very similar issue has been on LINQ-to-SQL’s todo list some time ago.
I’m not sure whether this is so much the typical use case. For the db4o case, it teaches us two things right now:

• Container reuse is non-trivial and should be approached with extreme care. You don’t want to run into this kind of byzantine error in a live app.
• The object identity problem is similar in both Object-Relational Mapping and OODBMS

» Scalable server architecture: multiple readers against the same file, transactional files
This sounds daunting, and it’s probably a huge one, as you can see from its age. I also believe this might be politically challenging, because this moves into the direction of Versant’s large-scale object database. However, there is a lot of movement into that direction from the user-side it seems – people are asking for features of this kind more and more lately, largely due to the ultra-cool LINQ integration db4o has. I suppose I’d be wise to focus on this kind of scenario as it could really become the preferred way of writing web applications: it’s extremely agile, supports rapid development, is flexible in that the same (LINQ) code could be used for different persistence layers if that should ever be needed and leads to clean, compile-time checked, type-safe code.

» Sanitize reflector design – remove core dependencies on generic reflector
Being able to get rid of the generic reflector seems important, I’m already building my own code for this. Here we have conflicting requirements: The GenericReflector makes db4o very easy to use and may help beginners. It is also required in client-server scenarios where the server doesn’t have the necessary model dlls, but for most applications I think you should try to avoid it. Storing data in a generic manner is slow and requires a lot more space, making it highly inefficient.

Current attack vector: Throw a Listener on the object created event on the server and make sure the server knows the type.

» Allow immediate TCP port reuse
When opening a server, the TCP port will be blocked in case the server crashes, the app is terminated, etc. Since that might happen quite often when you use the ‘integrated server’ where the server is actually created in your web application, a restart of the web application will fail because the TCP port is blocked. In a client-server scenario, on the other hand, a simple restart wouldn’t be possible because you need to assign a new port to clients or wait 6 minutes. This should be fairly simple, but I don’t know if that comes with any side-effects.

» Fast Collections
This is a huge one. The cool thing is that this could allow much more complicated queries to be executed in reasonable timeframes. However, I’m a bit worried about the issue “FastCollections: Inside BTree List implementation”, because that sounds really important, but is in state “Won’t fix”.

» Locking
Optimistic locking would be a nice-to-have thingie, but you do this yourself rather easy I think.

» A new object is stored upon value type updates
This is rated critical, so it’s not an item for a “wishlist”. I’m not sure if I understand its implications and I rarely use value types apart from Guids, and updating Guids is pointless – still a db4o user should probably know this and keep this in mind, so I felt it should go here ☠.

db4o Configuration

This one is not really in the issue tracker as a single item, and it’s more of a general remark. One of the rather messy things in db4o is configuration. Even with the new configuration interface, there is quite a bit of confusion among users. The reason, in my eyes, is mostly a combination of incomplete documentation and unexpected behaviour. Examples:

• The indexation setting for fields is the only configuration setting that is persistent. Everything else, including unique constraints, needs to be re-set when (or – more precisely – before) opening the ObjectContainer.
• Applying an option to a field that doesn’t exist will not trigger a warning or an Exception
• Some settings simply won’t have any effect when you perform them after opening the ObjectContainer, but they do not warn you.
• Certain settings must be applied on the server, a few must be applied on the client and with some … well, you just set them on both just to make sure. Here, db4o does throw exceptions, however!
• Several options need to be set before creating the object container (e.g. string encoding) and cannot be reset afterwards, again being completely silent about the ineffectiveness of the respective settings.
• Some settings, such as field-based cascade-on-activate, simply don’t seem to work at all

This leads to lots and lots of confusion. Most importantly, it is often hard to determine whether a certain setting was successfully applied, or not. Also, some defaults are unexpected:

• Default ActivationDepth is (completely random) 5. Why not 8? Or 2? This troubles beginners a lot. Either set it to infinity, or to zero. Everything else feels just random. You can still include a line ActivationDepth = 5; in beginner’s samples, thereby showing them that the setting is there and that they need to be aware of it.
• Default string encoding seems to be UTF-16 or UCS-2, probably the most useless encodings around, despite the Windows Kernel working with it. UTF-8 would come in as a reasonable default, but with UTF-16 half of your database is probably zeros, because even in non-english environments, there is still a lot of mostly ASCII-data to be stored (such as URLs, Email addresses, base64 encoded information, SHA-hashes, etc.). Also many languages have non-ASCII characters only sparingly, German being one example.

I think it’d be really cool if the configuration interface was a little more explicit and would throw Exceptions instead of silently ignoring requests that cannot be fulfilled.

Motivation

One thing up front: We don’t need an object database for the simple tools we currently build, but we felt it was a good idea to get some acquaintance with the technology, because we will certainly need it for our (stealth) startup “pactas” soon. In pactas, the data structure is really complicated (with a fancy class hierarchy) and it probably will change very frequently.

Also, since I am such a big fan of reusability, we really develop a web engine – a framework that allows us to reuse a lot of the code for different projects and make sure most of our code has been tested thoroughly in the field. This proves to be a significant design decision when it comes to data modeling, and while I’m very happy with the decision, it certainly makes development harder.

The Problem

So, in time with the release of our simple backlink tracker, one of our development database files started to show strange behaviour – a certain query (via LINQ) would not order objects anymore – the orderby-clause seemingly was completely ignored. Our live server even came up with an “Invalid DateTimeKind specified” exception when trying to perform the query! What’s worse: The problem kept occurring from time to time, but it was not reproducible! Byzantine errors are clearly my favourite…

We thought the issue might be related to the current development/unstable versions of db4o that we were using (7.12 and 7.13). Using the stable version of db4o (7.4) proved difficult, because the old LINQ provider falls back to LINQ to Objects very often, which requires to fetch all objects in question from the database – that is very, very slow for a lot of objects, so we had to abandon that. We clearly wanted to stick to LINQ for a number of reasons (compile-time checking, readability, reusability).

Obviously the problem is related to the orderby operation on DateTime fields. I tried to modify the query, removed grouping because I feared it might be unstable (warning: the sort operation is, in fact, unstable! Unstable grouping would be useless, but the grouping is stable so that’s fine), even debugged the db4o code, but I couldn’t find any problem in there. Since the code is rather complex and that was the first time I took a look at it, I was happy to find some of the relevant code at all. Somewhere in the deeps of it, something screwed up. I didn’t want to spend too much time on that since I had to chew some particle physics on the side.

Solutions?

At the time of writing this (in fact, yesterday) I had something like a hotfix, but it turned out it’s completely nonsense – it worked around our internal bug in a very peculiar way. No more, no less.

When talking about this, somebody mentioned that it wasn’t such a good idea to use DateTime at all, and we should store the ticks instead. That lead to two long discussions with my co-ed Christian. We concluded: First, the power of object databases is that they do not force you to hack around and find different representations for your data (which raises the bar for object databases). The one big shortcoming of SQL is that it forces you to find a second, equally good, but different representation of your data and you need to translate between these representations all the time. You need to synchronize them. And you lose a lot of fancy features (such as lists, generics, inheritance, etc.) on the way.

Secondly, Christian suggested that object databases suffer from leaky abstractions badly, in that they break encapsulation in a way that leaks out a hell of a lot of implementation details. As Joel puts it: “All non-trivial abstractions, to some degree, are leaky.” The point is: With the error we’re currently encountering, it’s becoming a real problem. This is not an imperfect piece of architecture, it’s an exception in a database query. It kills the app dead! Boom!

In order to get activation [depth] straight, db4o needs to know how .NET’s containers work internally – that is clearly a detail that should be hidden, but db4o knows about it. db4o also takes care of that, but it leads to some messy issues. There is special code in db4o that handles containers, non-trivial objects such as strings, DateTime (which are non-trivial because they use 62 bits for the actual ticks and 2 bits for the DateTimeKind) and Lists. This is an implementation detail of the .NET framework, and it might change over time. There’s been a bug with Map, and I’m almost sure there is a bug with DateTime, too. Don’t get me wrong: The fact that some kind of mapping is needed is a somewhat generic (if not the) problem of serialization, it’s not really db4o-specific, and cannot be eliminated. In Hibernate, there is also a lot of code that handles the mapping of lists and the like but it doesn’t rely on implementation detail, thus it’s not (as) leaky.

Back to SQL?

Here’s the thing: I’d be beneficial to have a storage that is independent of the actual implementation on top, because it decouples the data store from the application which is not what db4o does. But wait, that is exactly what SQL is, right? Indeed: SQL forces you to map (or to cut down) your stuff to it’s internal features. A list becomes a foreign key on the other table, but that doesn’t play well with derived types, generics, etc… This is tricky as we all know, and you need lots of code to do that.

Worse, SQL forces you do that mapping for everything, including your own classes, and it demands a schema for every type of object – this is not what I want. I’d like to see a set of base objects in an object database which are natively understood by the DB. These objects can be mapped from and to by a layer which may be part of the db, or can be added manually if you need something very special. Still, it should allow to store your object in the database with all it’s magic, only that known objects will be translated, e.g. a DateTime will be stored as long Ticks and a DateTimeKind flag separated, making comparison operations easier (note that the comparison only works on the ticks: Whether the time is local or UTC is not considered by .NET in comparisons). Lists will be unfolded into an internal tree representation if indexed, so they become much easier to cope with for the database. That would also make managing m:n relations easier, since they could then be viewed as bilateral relationships – something you often need. Right now, they’re unilateral, thereby requiring some additional management on your behalf.

Migrating to SQL is clearly an option for this product, since data couldn’t fit SQL any better, but that doesn’t solve the issue for our pactas project, where we certainly will need schema-less data, complex object hierarchies, etc.

However, there are a few issues that remain unresolved as of now and they do qualify as show stoppers :

• There’s an irreproducible bug that potentially wrecks the db
• There’s a nasty performance issue with larger amounts of data

Alternatives

I’m quite dissatisfied that we have to abandon db4o at this stage, because I believe it’s the best kind of serialization I’ve ever experienced. It’s perfectly simple, it’s fast and most importantly, code-centric. If reusability is a concern, having the database structure and/or the ORM mapper dictate the objects is a major pain.

Entity Framework 4 (EF4) promises to handle this a lot better through “Model First” and “Code Only”, but I am still a bit afraid of EF because it doesn’t appear to be anything near lightweight and we will need schemaless storage for our future products anyways.

Versant, which bought db4o some time ago also offers its large-scale object database, which seems to suit large web-applications a lot better. First, it’s certainly made for huge amounts of data (unlike db4o, which comes from an embedded background and is aimed at database sizes in the low GB area, but supports up to 254 GB per file) and handles multi-threading better. Also, since db4o now moved to v3 of the GPL, it may not be freely usable in non-open source web applications anymore, so both solutions now have a price tag.

There is also an ISV/OEM empowerment program for Versant’s Object Database, which seems to make it affordable, but I haven’t looked at it in detail yet. Over the next weeks, I will have to evaluate a few of those other NoSQL solutions such as Cassandra and MongoDB, just to name two totally different options.

Aftermath

So what was the issue, after all? db4o did not screw up, we did: Take a blend of local and UTC time based on a completely random criterion, add two spoons of daylight savings time changes, add some misconfigured timezone on the server, bake for two weeks at 500 ° C and pling! You got yourself some really strange issue cake. Lessons learned:

• Debugging issue with object databases is harder than with RDBMS because the information is not chopped up.
• Do code reviews. Do code reviews.
• There aren’t too many alternatives to db4o, after all
• With the elaborated architecture and code-centric design we currently have, going back to SQL is a huge pain, and we won’t do it

All serialization is painful.

Talking about speed and performance, I just came across an issue that was also discussed in db4o forums very recently: Sort operations on large datasets.
Note: This has nothing to do with linq or linq to db4o, it’s just the same for SODA queries.

Let’s take a very common example: Find some most recent items, for example most recent blog/forum posts or some other ‘top list’ on a very large amount of entries N:

        var mostRecentPosts = (from Posts o
              in ObjectContainer
              orderby o.Created descending
              select o).Take(100).ToList();

This query will be really really slow on a large amount of objects. Why? Essentially, the BTREE operation is very fast as it should be, but unfortunately db4o will invoke its SODA system on each of the objects, even if they are already outruled by the BTREE operation.

See this discussion on the db4o forum and the associated Jira-bug.

This is somewhat sad, because a query on a previously filtered set of items is blazing, about 100x faster:

        var mostRecentPosts = (from Posts o
              in ObjectContainer
              where o.Created > yesterday
              orderby o.Created descending
              select o).Take(100).ToList();

The latter operation plays roughly in the same league as SQL Server (don’t flame me – performance comparisons and profiling is really complicated and there is a zillion of factors that influence it, I know. That’s why I say ‘roughly the same league for this query’ and I’m talking about default setups).

This is a somewhat unfortunate situation, because it produces slow queries for many typical applications – unless you have a strong where clause that cuts down N from millions to hundreds. If you’re interested in seeing this issue fixed, head over to their issue tracker and vote for the issue to be fixed! Thanks!

I’ll be posting a bit more on db4o over the next few days I hope.

Part II of the article will show how a SerializationSurrogate works and explore wheter we can use it for generic de-serialization of DependencyProperties. Also, in this article I provide full source code for both parts of the article.

UPDATE: A much better version of this article can now be found at the CodeProject: DependencyProperty Serialization for Business Objects

Alternatively, directly get the source code here: http://www.emphess.net/wp-content/uploads/2009/03/bpboserialization1.zip

Well, let’s get started:

ISerializationSurrogate

This interface is designed as a stand-in for your class upon serialization and de-serialization.

It is quite simple and contains only two methods:

public void GetObjectData(object obj, SerializationInfo info, StreamingContext context)

and

public object SetObjectData(object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector)


This is quite interesting, since there is an explanation on MSDN why they chose not to provide a SetObjectData() method for the ISerializable interface, so this makes quite a difference.

Surrogate Serialization

For serialization, we use almost the same code we used in part I, with a few modifications only:

public void GetObjectData(object obj, SerializationInfo info, StreamingContext context)
{
    PropertyDescriptorCollection descriptors = TypeDescriptor.GetProperties(obj.GetType(),
            new Attribute[] { new PropertyFilterAttribute(PropertyFilterOptions.SetValues |
                                                          PropertyFilterOptions.UnsetValues |
                                                          PropertyFilterOptions.Valid ) });

    List<FieldInfo> fieldInformation = ReflectionHelper.GetFieldInformation(obj.GetType());

    foreach (FieldInfo fiRover in fieldInformation)
    {
        object[] attribs = fiRover.GetCustomAttributes(typeof(NonSerializedAttribute), false);

        if (attribs.Length > 0)
            continue; // Don't serialize this

        info.AddValue(fiRover.Name, fiRover.GetValue(obj));
    }

    foreach (PropertyDescriptor propertyDescriptor in descriptors)
    {
        SerializeProperty(obj, info, propertyDescriptor);
    }
}

(Please note that I clean the code up a little, in the source download there is a lot of debugging output, etc.)

Surrogate De-Serialization

Now, it seems as if an implementation of ISerializationSurrogate might do the job and help us to circumvent the necessity to provide boring constructors through the SetObjectData() method. So let’s give it a try by basically using the deserialization code from part I:

public object SetObjectData(object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector)
{
    Type thisType = obj.GetType();
    DependencyObject depObj = obj as DependencyObject;

    // De-Serialize Fields
    List<FieldInfo> fieldInformation = ReflectionHelper.GetFieldInformation(thisType);

    foreach (FieldInfo fiRover in fieldInformation)
    {
        object[] attribs = fiRover.GetCustomAttributes(typeof(NonSerializedAttribute), false);

        if (attribs.Length > 0)
            continue; // Don't deserialize this

        fiRover.SetValue(obj, info.GetValue(fiRover.Name, fiRover.FieldType));
    }

    // De-Serialize DependencyProperties
    PropertyDescriptorCollection descriptors = TypeDescriptor.GetProperties(thisType,
            new Attribute[] { new PropertyFilterAttribute(PropertyFilterOptions.SetValues |
                                                          PropertyFilterOptions.UnsetValues |
                                                          PropertyFilterOptions.Valid ) });

    // Try to use a mapping to set all DP values deferred. This will actually just
    // postpone the Exception
    List<DependencyValueMap> mappingList = new List<DependencyValueMap>();

    foreach (PropertyDescriptor propertyDescriptor in descriptors)
    {
        if (propertyDescriptor.Attributes[typeof(NonSerializedAttribute)] == null)
        {
            DependencyProperty dp = DependencyPropertyHelper.FindDependencyProperty(obj, propertyDescriptor.Name);

            if (null != dp)
            {
                mappingList.Add(new DependencyValueMap(dp, obj as DependencyObject, info.GetValue(propertyDescriptor.Name, propertyDescriptor.PropertyType)));
            }
            else
            {
                throw new SerializationException(String.Format(@"Failed to deserialize property '{0}' on object of type '{1}'.
Property could not be found. Version Conflict?", propertyDescriptor.Name, thisType));
            }
        }
    }

    foreach (DependencyValueMap dvRover in mappingList)
    {
        // This call will cause an InvalidOperationException:
        depObj.SetValue(dvRover.dp, dvRover.value);
    }

    return null;
}

Now, unfortunately, we cannot set the values of these DependencyProperties. The SetValue() method in DependencyObject will throw the oh-so-magic InvalidOperationException. I haven’t been able to find a way to go around this.

Summing it up

Well, that’s it basically. Where are we now?

• It is possible to use DependencyProperties for business objects and serialize and deserialize them.
• A reflection-based base class “SerializableDependencyProperty” does the job, as long as you stick to the naming convention and implement the deserialization constructor in all derived classes
• Circumventing the explicit deserialization constructor seems impossible due to limitations of the DependencyObject base class

The Source

The sample solutions contains two projects:
WPFDependencyPropertySerialization shows how the serialization of part I can be accomplished.
WPFDependencyPropertySerializationError shows the extension using an ISerializationSurrogate and, as mentioned before, will lead to an Exception … which will not be caught so you can really feel it rumbling:
“Current local value enumeration is outdated because one or more local values have been set since its creation.”

You can download the source at http://www.emphess.net/wp-content/uploads/2009/03/bpboserialization1.zip

Have Fun!

A post scriptum on DependencyProperties for business entities

The project I’m currently working on comes with a very complex data structure. It consists of a set of trees which can be connected in various different ways. We also have to account for circular dependencies between these objects. The user can create and modify all these different types of connections.
Moreover, the architecture of the application is highly sophisticated and is based on Microsoft’s composite application guidance. A lot of modules interact with each other, each having their own data, similar to a ViewModel yet managed by a Presenter. What sounds a little messy has been proven to be the best solution in a number of discussions.
These business entities are based on DependencyProperties and a lot of crucial features depend on the functionality provided by the DependencyProperties. It works well, though I should mention that I did not use a number of DependencyProperty features in the first place. For example, coercion and validation coupled with automated de-serialization might lead into trouble, so keep this in mind should you ever intend to go down the path…

MSDN: How To: Implement ICommandSource
MSDN: Implement ICommandSource Sample.

Not a big thing, but since google didn’t show anything for “ICommand.CanExecute NullReferenceException”, it might help somebody I hope…

http://www.codeplex.com/prism/Release/ProjectReleases.aspx?ReleaseId=12226

At the current release intervals, it’s hard to keep up

I will update my article on the CodeProject about OpenGL windows in Vista soon!

See http://www.taoframework.com/

Get the latest version via SVN: https://taoframework.svn.sourceforge.net/svnroot/taoframework