This post is first one in a series that describes what I learned while trying to set up a simple, inexpensive Linux-based HA Server Cluster. The series begins with a description of an inexpensive Windows-2003-based cluster. This setup is inexpensive as far as Windows 2003 goes, but going with Linux should allow for savings of close to $1700 (or $3000 depending how you count it) per setup.
The goal of the setup is to run several Java application on Tomcat. These applications use MySql for persistence. In order to implement redundancy two hardware servers are used. Hardware and software are setup in a mixture of Active/Active (for almost all the components) and Active/Passive (for MySql) configurations (see Wiki:High Availability Cluster).
Being an inexpensive setup, only two hardware servers are used on production. Each of the two hardware machines has the following software installed:
Load Balancing and FailOver
In order that the two machines appear as a single cluster, Network Load Balancing (NLB), which comes standard with Windows 2003, is used. There are a bunch of “how to” tutorials on the web about how to set up an NLB cluster (for example this one). In my case two network cards per server were used – one each was reserved exclusively for the NLB. Altogether, this configuration needed 5 static IP addresses. (I have not been successful with setting up an NLB cluster with dynamically allocated IP addresses.) In production, all the requests to the cluster were done via the IP address assigned to the NLB.
NLB setup will handle the failover case where one on the machines becomes unresponsive – i.e. either it goes completely down or the network interface does not respond for whatever reason. Being a very simple load balancer, it is totally unaware of the state of the software (Apache, Tomcat) that processes the requests. Because of that, our simple HA Cluster uses JK Connector as a second failover point. JK Connector also handles session affinity, as NLB itself is only capable of handling the Client IP affinity.
A request to the the cluster arrives at the NLB, is from there routed to Apache on one of the machines. Within Apache, JK Connector then routes the request to a Tomcat on one of the machines. Each of the hardware servers have generally an identical setup of one or more Tomcats. Some of the applications on these Tomcats are stateless, while other may maintain sessions. JK Connector makes sure that session bound requests are routed to correct Tomcat instances.
MySql is configured in Master/SlaveA/SlaveB configuration. Master and Slave A are on the production server machines at a hosting location. Slave B, whose Master is Slave A is on a local machine, and servers two functions. It is a backup. It is also used by local data analysis / data mining server in read only mode.
All production Tomacs normally point to the Master DB server. The failover is achieved by a custom java class extending DataSource (from Spring). It automatically switches to Slave A DB if a request to Master DB fails.
The weakest link in this setup is Apache and that is due to the fact that NLB is not aware of application state. If Apache fails, for whatever reason, NLB will blissfully continue sending requests in the direction of the dead Apache. Fortunately, the Apache in this setup has a limited set of simple standard tasks to do like: forwarding all requests to JK Connector, authentication and access control, handling ssl/https, one or two rewrite rules. It is unlikely that it fail. If it fails, the failure will be noticed by the server monitoring software, which will first attempt to restart the Apache. If that does not help, monitoring software will then take that whole server out of the NLB cluster (note that Tomcats and MySql on that server will not be affected by it).
I have already mentioned a local machine (with Slave B DB) which is a companion to the two hosted production servers. In addition to this there is a local setup mirroring the production in software (but with slightly less expensive hardware) that is used for testing before deployments.
Cost Saved by Using Linux
By switching to Linux the following software costs can be spared:
- Production servers — 2x Windows 2003 R2 Server Standard Edition — 2x $642
- Companion/backup server — 1x Widows XP Professional — 1x $140
- Server Monitoring software — 1x $395
- Test servers — 2x Windows 2003 R2 Server Standard Edition — 2 x $642 (but this amount is distributed over several production clusters for different applications/clients)
This gives savings of up to $3100.
To be continued …
Filed under: Computers, Software | Tagged: apache, mysql, NLB, tomcat, windows 2003 | 1 Comment »