On the web facing side we use a pretty common old school setup of Tomcat which is hidden behind an Apache–mod_jk proxy. We make use of Apache’s mod_rewrite to clean out some URLs and provide protection of some special URLs of the app. Also mod_rewrite does a fantastic job of protecting the servlet container from recurring security issues in either Java, Tomcat or any of the used frameworks within the app. The appserver also runs an instance of the mongos router to provide access to MongoDB. Load balancing is realised with Amazon’s ELB. This by the way gives you IPv6 connectivity for free if your app can cope with it.
Shoe of the Day relies on MongoDB for persisting its data. For MongoDB we have a setup consisting of three EC2 instances. These three build one replica set which in turn represents one shard. MongoDB relies on so called replicas for data redundancy. Our partner 10gen advises you to never run on only one mongod instance in production. Replica sets provide automatic primary/secondary like distribution of data with automatic failover between nodes. At any given point in time there is only one primary within a replica set which provides all write operations. If the app can handle it you can also decide to allow reads from secondaries. As MongoDB per default does not guarantee real time consistency this can lead to unexpected behavior if you don’t have planned for it. By allowing reads from secondaries you can scale out read performance. In order to improve write performance MongoDB has a concept called “sharding” which automatically partitions data between a given number of nodes. These nodes should always represent a replica set. For this to work you need a mongos process (running on the app server) and a set of config servers which provide the metadata which data lives on which node. SOD has the config servers running on the MongoDB nodes as the only have a small footprint and we don’t want to run more instances than really needed. The MongoDB nodes store their database files on a RAID 10 over six EBS volumes which increases the value of random I/Os and another two EBS volumes on RAID 1 for the journal. Deployment / Provisioning
All hosts are run from custom built AMIs within EC2. For the deployment of new instances we make heavy use of boto which is a Python library that simplifies access to various AWS services. Boto is usually called from Fabric which is the Python relative of Capistrano. We use Fabric to create new instances and to control services on nodes. This enables us to set up new hosts with one call like “fab provision_mongodb_node:mongo01”. This for example starts a new instance of one of our own AMIs. After the instance has started up and accepts SSH access we attach eight EBS volumes which are partitioned and formatted and bundled together in a software RAID 10 with mdadm. After that the mongodb packages get installed and necessary configuration is deployed via chef.
In order to ensure that we have consistent configuration among all hosts we use Opscode’s chef. Following a lightweight approach we use chef-solo exclusively. Cookbooks are held in a git repository. When changes are made one can kick a chef-solo run via fabric.
Here we have several different components. For one we do collect classic system metrics and do monitoring as well as trending and alerting via XyMon. For MongoDB we also have an MMS account where you can easily monitor your whole MongoDB setup. In combination with a munin-node client you can also collect traditional metrics. This makes it easier to investigate potential problems in cooperation with 10gen’s great support team. We also want to know what’s happening within the application so currently we are evaluation both NewRelic and AppDynamics.
Monitor your spending
When hosting your systems and applications in the cloud you do not only have to monitor your performance or availability but also your costs. At EC2 you pay as you go so an out-of-control script or an app that consumes much more storage than expected can raise your bill more than you probably like. Also there is always the possibilty that your AWS credentials are compromised and you pay for someone else! For SOD we rely on cloudability. They have huge amount of supported providers and nice reporting and alerting.
DNS + proper Hostnames
EC2 will give your instances very odd looking private and public hostnames which are ephemeral. So to properly configure your instances and monitoring you want to give them custom names. For this we leverage Amazon’s DNS Service Route 53. Through an API we set and change internal and external DNS Records at boot time.
Before putting anything in production and if it’s your first deployment in EC2 you should get a feel for the different instance types fitting your workload you should do load testing. The guys from blitz.io provide a nice service where you can conduct load tests even from a browser plugin for Chrome!