Distributed MapReduce with sharded MongoDB and SpringData

In MongoDB you have several options when it comes to aggregating data. There is the included JavaScript MapReduce Framework, the new Aggregation Framework in v 2.2+ and also the Hadoop Connector for really heavy lifting.

When using the integrated MapReduce Framework you have to be aware of the caveat that MongoDB’s JS Interpreter is single-threaded. This means that regardless of how many cores your server has only one gets utilized. That’s a bit of a bummer because vertical scaling might not decrease execution times significantly. So to really bring down execution times and make use of MapReduce’s parallel computing abilities you have to scale out and shard. This brings mainly two advantages. For one you have now more “threads” and also the dataset that each of these has to deal with is getting smaller in relation of how many shards you have.

The latest of stable version of MongoDB today is 2.0.6. which was initially used for our queries. Using this version queries that we successfully issued against a single MongoDB instance failed on the sharded setup.
It seems that we were hitting an issue similar or equal to https://jira.mongodb.org/browse/SERVER-5536.

As the issue states that it’s fixed in 2.1.2 we switched to the latest nightly build (2.1.2-pre) which worked fine.

Unfortunately after switching to 2.1.2 we were confrontend with https://jira.springsource.org/browse/DATAMONGO-378

The pragmatic albeit not beautiful was a hack that reimplements the following Interfaces: MongoOperations and ApplicationContextAware. It basically works around the type cast to Integer:

((Number) counts.get(“input”)).intValue(),
((Number) counts.get(“emit”)).intValue(),
((Number) counts.get(“output”)).intValue()

All of this resulted in the following performance improvements:

Performance Chart

These numbers show how much parallelism can actually result in a massive performance increase. MongoDB brings excellent out-of-the-box capabilites to simplify sharding and replication.

2 thoughts on “Distributed MapReduce with sharded MongoDB and SpringData

  1. Pingback: Follow up on distributed MapReduce with MongoDB « comSysto.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s