This afternoon I’m live blogging from TheServerSide Symposium, attending “Breaking all the Rules:  The Myth of Testing and Deployment in the Cloud” presented by fellow CodeRanch Andrew Monkhouse.

1.  CodeRanch History
Andrew opens his talk with a discussion of CodeRanch performance, traffic, and memory usage.  He mentions there have been serious issues in the past that haven’t temporarily brought the performance of the website down to a crawl.  CodeRanch uses a fork of JForum (2008) and staffs around 20 developers among 150 moderators, all volunteer participants, to build a custom community-based forum.

CodeRanch developers use a number of test strategies including smoke tests, load tests, and stress test.

2. Redeploying in the Cloud
The focus of Andrew’s presentation will be to demonstrate how CodeRanch would function in the cloud.  He has allocated a “small EC2 instance” (1.7GB memory, 1 compute unit, moderate I/O performance) within Amazon’s cloud that is useful for quick set up, but extremely small for practical use and reliable testing.

3. Benefits of the Cloud
The cloud provides a relatively clean environment in which it is ‘cheap’ to add extra servers, at least compared to running a local test server which may be allocated for additional processes.  Also, tests can run by anyone, anywhere, which is great for a distributed developer team, since Amazon allows you to assign rights to additional users.

You can easily create, set up, and desotry Amazon Machine Images (AMI), so you do not need to get it right the first time.  Lots of backup storage options, depending on your needs.

4.  Dangers of the Cloud
Testing in the Cloud can be difficult.  First off, it is hard to get absolute numbers because of the nature of the underlying change in hardware happening within the cloud.  Second, calibrating results is difficult.  Also, developers have to remember to shut down servers after they are down with them else other developers will run out of resources.  Finally, network traffic does cost money so testing can be expensive, although hardware is cheaper in cloud.

5.  Test Setup
The performance tests were set up into 3 servers, in part because it turned out to be cost effective to set it up this way based on the individual server options.

• Application Server running CodeRanch
• Database Server
• JMeter Test Server

JMeter was used at the testing suite software given its large variety of features, options, and plugins.

6.  Test Analysis
Andrew discussed some items to keep in mind when testing the server and understanding the results:

• It is quite possible for the Test Server to have greater load than the Application server, so be sure to monitor all your servers!
• Try different graphs, they may tell you different things
• JMeter plugins are your friend, they will help you identify problems with your application server and/or test set up.
• Don’t forget about ramp up times

7.  Where do we go from here?
Automated weekly performance tests using the latest code base.  Add more stress tests.  Andrew comments there are no plans to move the CodeRanch to the cloud at this time given that we require full-time administrators since we don’t have a dedicated, paid team of administrators.

Conclusion
Andrew presents a compelling argument why you should be testing your applications within the cloud.  It provides a sanitary environment in which to perform a large variety of tests across multiuple servers.  I’ve heard arguments of moving application servers to the cloud, but this is the first time I’ve heard it suggested to migrate all automated testing to the cloud.