December 12, 2013

Moving from Google Appengine to NodeJS on Amazon EC2

nearby.lk moved the servers from Google App engine to Amazon EC2 a couple of months back, and the backend is now built with nodejs with mongodb as the database.

Backend does a lot of pre-computations and caching which gives slower start up times and faster response times. The migration decision was based on a number of factors such as ease of development, performance and cost. There were a bunch of disadvantages of using the app engine, a several benefits of using NodeJS and some things we miss after the migration.

App Engine: the bad #

Data store operations are slow #

There were not many data store writes. Almost none on most days since data was uploaded in bulk. Still, the data store reads were significantly slow. To improve performance we kept most of the data in memory, which worked great until the web traffic and database size started growing.

Too many instances #

App engine does load balancing for you. So when it feels like a single server cannot handle the load it creates multiple instances and balances the load. Although this sounds awesome, it wasn’t for us. Pre-computations at start up kept the new instances busy, and app engine was creating more and more instances to handle this. Worst part is that multiple instances weren’t actually required to handle the load, but App engine had a few instances running all the time, and we had to pay.

We tried warm-up requests to keep an extra instance running, to solve the slow start up issue, but it was not of much use.

Request Timeouts #

Start up pre-computations took more than the request time limit. This made us break down the start up process into smaller chunks and run each part on different requests. That is, an instance would only be ready after a couple of requests were sent to it. Breaking the start up process was a horrible coding experience, and the worst part is that the start up is taking most of it’s time reading the data store - the system would start up in within a second on my computer with app engine development server.

Memcache #

Memcache helped solve the start up issue a bit, but it had a stupid 1MB size limit per entry which made things really hard for the developers. Large objects had to be broken down into pieces smaller than 1MB and if one of them was removed from memcache, everything had to be recomputed. (Search index was larger than 1MB)

Uploading and downloading data #

This was literally a nightmare. bulk data uploads and downloads had to be broken down into tiny chunks because of the request time limits, and there was no way to access the datastore without writing code to do it.

Search API #

We HAD to use the search API for the last few months because we couldn’t keep our indexes in memory, because of the start up time (discussed earlier). You might expect the search API to be super awesome because it’s Google, but it was so slow. May be we didn’t use the proper design, but it was the best we could find with the documentation available.

NodeJS: the good #

One language #

Now the server and user interface are both in javascript, which makes it a lot easier for the developers to switch between the two.

Server-side rendering #

We don’t have to maintain different code to do server side rendering since we can use the same templates on both the server and client.

Portable #

We can now host our servers anywhere. We are not stuck with any platform, as we were with the app engine - where we had no choice to pay more if they were to change pricing. We are currently hosted on amazon and it is running smoothly. We might switch to a larger server on amazon soon.

Speed #

The start up time is much faster and requests are handled within a couple of milliseconds. We are using Nginx for all static content.

What we miss #

Management Console #

Appengine had a nice management console where we could look at logs, system status etc. Now we have to do it through ssh with unix commands. Although this is not a big issue for the developers, now we can’t ask someone else to check or restart the servers if needed.

Ease of deployment #

With app engine, all you have to do deploy is just to run a command. We need to write a small script to do this now.

Trying out new stuff #

With appengine you can easily create a new account or a new version without interfering with the main system, and do beta testing or A/B testing.

So far we are so glad that we moved away from appengine and we regret that we didn’t do it sooner. Development got so much easier and now we are working on stuff that actually adds value to users than hacking the system to deal with all the constraints such as memcache limits, slow datastore reads, etc.

Found these posts which also discuss problems with Appengine #

I wish I read one of these an year ago.

Hacker News Discussion #

302

Kudos

302