Would I recommend RavenDB?

RavenDB,Thoughtful Stuff 13 November 2013 | 8 Comments

Settle in for a long post, I’d make a cup of tea before starting…

With my new job there is a need to rework the existing solution to scale, one way we are going to achieve this is through migrating the web interface from ASP.NET WebForms to AngularJS with a nice REST API provided by NancyFX. This already shows great improvements in load times and user experience but we need more speed and scaling than SQL Server can provide.

Firstly, I consider myself pretty experienced with RavenDB, I’ve used it in past freelance projects and helped companies use RavenDB also. I’ve had great results with it so far.

My first project with RavenDB started back in 2011, to be accurate, 2011-10-26 according to Git. I started with version 499, hearing old war stories from the remaining early adopters, I’ve had a pretty pleasant time to what they went through. I’ve also been a personal, paying customer since 2012-2-25.

Let’s start with the positives about RavenDB, because I want this blog post to be balanced and not dismissed as “whining” which certain members of the mailing list community like to do if anyone questions the product.

The biggest wins

  • Have you ever installed SQL Server? If so, RavenDB is an exe you double click, that’s it.
  • No need for a DBA.
  • Client API is first class for .NET, rare in products these days.
  • It’s JSON, I work with AngularJS so it simplifies a lot around modeling.
  • It’s a second generation document store, you get features that make your life a little easier (like Includes, Transforms etc)
  • Much more, it is just awesome.

RavenDB is hands down the best database and client API experience I have worked with, it physically gave me headaches going back to SQL Server at my new job. It and NancyFX keep me on the .NET platform, else I would be moving on.

Hold your horses

Now, it’s not all plain sailing. There is a side to being a RavenDB user not many people like to talk about publicly, that is it often goes wrong.

Not in the “lose all my data” way, but in, ah, I’ve found a bug in feature X or when running in production you notice things like a memory leak or a feature is suddenly broken.

Steadily RavenDB has been getting a little better at releasing versions with less bugs but it isn’t enough.

Recently, bugs have been introduced when a new feature has been added but not tested properly or seriously breaks a core aspect of RavenDB.

For example, etags, a core part of how RavenDB works (basically a guid like id that tracks document changes). I just hit this etag one with verison 2750 and it is extremely frustrating to have an older part of your application break. I didn’t add unit tests around etags because, well, I didn’t expect a core aspect to break.

The real world, Phil.

Now, this is acceptable when you are working on bleeding edge projects or pet projects but I’m discussing RavenDB as an option with my new company and I have this nagging feeling that these issues will be a huge “no no” to us.

I love all the new shiny features going into RavenDB 3.0, the HTML5 studio, OWIN support and whatever is announced in tomorrows Mystery Feature #3 webinar (update: Java client, meh) but these don’t solve my issue with RavenDB, the lack of QA/testing/stability, whatever you choose to call it.

But, the thought of Voron replacing ESENT, a trusted for many years database, kind of scares me. I don’t like to think about what would happen when bugs happen in the storage engine…

I choose to hope that the engine has an extensive suite of unit tests, stress tests and is reviewed by community members who understand storage, I’m a web guy and don’t pretend to understand storage.

Going forward

Ok, so let’s be positive, how could RavenDB solve the issue of bugs?

Does RavenDB have dedicated testers?

I know it’s cool to say you have automated tests but it would be nice to have a team member trying his or her best to break it.

A developer evangelist

Some overlap with a tester but we need someone building apps in the public to showcase best practices, stretch the API and be a beta tester so we don’t have to gamble with our productions servers.

More focus on testing, not shiny new features

We all love new features but I’d like there to be more focus on improving the stability of the product.

Benchmarking and stress testing

We need a stress test, so we can pound RavenDB in our own environments. Also a benchmarking tool that we can run on our own hardware that spits out numbers so that we can compare and try to improve with different setups (SSDs etc) would be great.

Free licenses for staging environments

This seems strange, RavenDB has a “developer” machine exception for commercial licenses but as soon as us developers put it on a few VMs to check advanced features like replication work on different machines etc, we need a license? This really needs to change as it is going to cost us money to test your product actually works.

Better upgrade story

The current methods to upgrade RavenDB are either you take it offline or use replication to the new version. This is kind of OK for a new product but this story hasn’t improved since I started using RavenDB. I just wish there was an upgrade story that was automatic somehow and could be a single click, involve minimal risk and minimal downtime.

Conclusions

Would I recommend RavenDB? It depends.

For pet projects and small freelance projects, where a document database makes sense and where some bugs are expected, sure, it is a great choice of document database.

For corporate projects, well, no, not currently. Maybe when 2.5 has some stability work done (perhaps the next stable after 2750).

Currently as a business you really have to be willing to invest a lot in your own testing of RavenDB. This could be an extensive set of unit tests (that will actually need to test even the basic things like etags) and a staging/testing environment to put each update through its paces. Then I would feel comfortable recommending RavenDB, if you just expect it to work and be flawless, you’ll have a lot of 4am phone calls.

I hope this post comes across as constructive, I love RavenDB and want to see it succeed in bigger projects and environments but certain experiences so far in smaller projects haven’t been a great indicator for bigger projects. Hopefully, this gets better once 2.5 is stabilized and 3.0 is released and stability work is done.

What I really want is a promise from the RavenDB team and Oren to invest money in making RavenDB more stable rather than purely focusing on features.

I will do future blog posts if the situation changes.

 

 

8 Responses on “Would I recommend RavenDB?”

  1. Daniel Lang says:

    That’s a great post, Phil. As you remember, we had a couple of serious issues with bugs, memory leaks and broken upgrade stories. I cannot recommend RavenDB to anyone for serious stuff (things that pose significant threat to the health of your business if it doesn’t work for more than one day).

    The only reason why I haven’t written such a blog post, is because I like Oren and want him to succeed. I share your concerns about Voron, although I really wish to see a RavenDB Linux/Unix story and I think Voron is a prerequisite for that.

    Apart from stability, the other pressing issue I think RavenDB has to solve is to become independent from Oren. I know there are other people contributing, but all the key stuff is still being handled by Oren. Without him, RavenDB would die. Sure, he’s a great programmer, but I don’t want to base a big mission-critical project off on the fate of a single person.

  2. Totally reflect my experience and conclusions. The only difference: we didn’t run into the most serious of these problems before we where in production! So right now we’re scratching our heads to figure out if there is a way out (using some other doc store), which is no easy task when our code utilize Raven’s great client API. Even though it’s “just” json. It’s doable, but costly.

    I’ve spent countless hours in the RavenDB code (mostly because of lack of documentation, and looking for bugs), and found many bugs. Everything from simple equations that’s simply wrong (and not unit tested) to more serious memory issues, mostly because of .net’s garbage collector, which you can’t blame on Raven, but it still effects us badly.

    Currently Raven suffers from a serious trust issue, but I agree it’s a great product. Too bad I can’t say the same for the product quality.

  3. Yeah I agree with everything you said (I think I even suggested some of them myself :)). I want RavenDB to succeed, but I would be hesitant to recommend it to a business that requires something stable. As a Key/Value store RavenDB has NEVER failed me, which is a testament to the stability of Esent, but the indexes and map/reduces are another story.

    The question is “Would I still recommend RavenDB?” Yes I would, but the client would have to know that it is evolving and their are risks.

    Great post.

  4. Thomas says:

    Good to know, thanks for sharing your thoughts and experiences.

    We switched to RavenDB a couple of months ago, coming from MongoDB because of some of Ravens features like transactions and the power of the .NET Client (which doesn’t help you out when it comes to indexes, right? ;)).

    We are still in development, so we have no experiences under production circumstances for now, but there has always been some doubt.

    More than ever since we found some really fundamental bugs regarding handling long-types within indexes and how one answered to it … (“You put it in the Mailing List, so please provide a failing test by yourself, otherwise we will just ignore it.”).

    Thomas

    • cmh says:

      Documentation is another area that needs a tremendous amount of improvement. One should not need to spend hours pouring over source to figure out how to use the product to it’s fullest potential.

  5. Vlad Kosarev says:

    I’m curious as to what you use now instead of Raven. I agree with all your points and I’ve had many sleepless nights dealing with ‘stable’ releases that had colossal bugs in them. Lack of documentation, unpredictable behavior and ‘write a failing test’ kind of support all add up to zero trust of your database and that sucks. I absolutely love the idea of Raven and when it works it’s just glorious but while these highs are great the lows are just terrible.

  6. Phil S says:

    Very nice, informative post. Got here doing a search on licensing for Raven, looking to see if there is a lot of outrage about non-production environments requiring a license. I’ve worked for at least two companies where this was a non-starter for selecting a tool (SiteCore was one such tool we had to pass on, even though it otherwise looked like the best fit for our team).

    Your commentary on storage engine defects in new versions is very alarming. It sounds like others have similar complains. And as Jon Arild Tørresdal’s comment seems to indicate, there does seem to be some vendor lock-in in using the client API.

    What sucks is that I so desperately want to use Raven. It’s just an absolutely pleasure to code against. I have an upcoming project where a NoSQL solution would make sense, but I’m not sure I’m comfortable recommending Raven to my client.

  7. Raj says:

    You said “I hope this post comes across as constructive”. Hands down this is the best post I have come across after reading weeks worth of materials just to pick one.
    Many many thanks!

Leave a Reply to Raj