Tag Archives: testing

smlunit – using without a python dependency

Posted on by
4

I’m currently taking the Coursera Programming Languages course which starts with ML.  Or more specifically SML/NJ.

Why I was looking at unit testing in SML

The instructor gave a test file with statements like:

val foo = function_to_write(1) = 2;

This is good in that it clearly shows the API and some of the expectations.  It’s not so good because you have to go thru the output to look whether all the values are set to true.  We were provided with 11 tests and told to write more.  I had 57 by the time I was done.  And I haven’t looked at the two challenge problems yet.  The instructor was very direct in saying we shouldn’t compare SML to languages we know because it impedes learning.  Well, I didn’t.  I compared the infrastructure.  And I learned more SML learning how this works than covered in week 1 of the class so I think I honored the spirit of this.

Choosing a SMLUnit implementation

I found two options in search

  1. SML Sharp‘s SML Unit – I found more references to this one on the internet.  However it requires steps to install and looks like it takes longer to get started with.
  2. SML Unit – This is the #1 hit on google for SML Unit and appears to be something written by a university student.  It’s very good though and I recommend it.

Python dependency

SMLUnit comes with good documentation and examples.  It also comes with a runner that uses Python to kick off the tests and report on the status.  This can’t be done purely in SML because reporting on the results requires state.

I’m sure it is a good runner, but I didn’t notice it existed.  (I started on the code page with the assertions not the main page that explains all this.)  Since I went to the (minor) trouble to create a runner in UNIX,  I’m posting it in case anyone wants to test SML on a machine that doesn’t have Python.  Not likely I know – most machines have Python.

The runner

The runner feeds your unit tests to SML and then reports on whether there were any failures.


# Since sml is a functional language, I couldn't figure out a way to output a summary
# at the end as to whether any tests failed. Wrapping in a shell script to avoid
# this problem. [realized afterwards it comes with a Python runner]
#
# Usage:
# smlunit_runner.sh name_of_tests.sml
#
# Assumes the sml test suite has "use" statements for all dependencies including smlunit

if [ $# -eq 0 ]
then
 echo "SML test file is mandatory"
 exit
fi

OUTPUT=`cat $1 | /usr/local/smlnj/bin/sml`

echo "$OUTPUT"

NUM_FAIL=`echo "$OUTPUT" | grep "^FAIL$" | wc -l`
NUM_PASS=`echo "$OUTPUT" | grep "^OK$" | wc -l`

echo ""
echo ""
echo "Test summary: (also look at last output to make sure not a stack trace)"
echo "$NUM_FAIL tests failed"
echo "$NUM_PASS tests passed"

What do the tests look like?

Code – helloWorld.sml

fun hello_world() = "hello"

fun add(a:int, b:int) = a + b

Tests – helloWorldTest.sml

use "asserts.sml";

<em id="__mceDel">use "helloWorld.sml";</em>

assertTrue true "assert true";
assertEqual "hello" (hello_world()) "compare two values";
assertEqual 3 (add(1,2)) "compare two values again";

And what about the asserts?

asserts.sml is a direct copy of smlunit.sml


(* from https://github.com/dellsystem/smlunit/blob/master/lib/smlunit.sml *)

fun roundish (x:real) = Real.realRound(x * 100000000000.0) / 100000000000.0;

fun assert expr1 expr2 (desc:string) function =
 (print ("*********** " ^ desc ^ " ***********\n" );
 (if function (expr1, expr2) then
 print "OK\n"
 else
 print "FAIL\n"
 );
 [expr1, expr2]);

fun assertEqual expr1 expr2 (desc:string) = assert expr1 expr2 desc (fn (x, y) => x = y);
fun assertRealEqual (expr1:real) (expr2:real) (desc:string) = assert expr1 expr2 desc (fn (x, y) => Real.== (roundish(x), roundish(y)));

fun assertTrue expr desc = (assertEqual expr true desc);
fun assertFalse expr desc = (assertEqual expr false desc);

(* Functions for comparing a lot of things in one list *)
fun assertReals f [] (desc:string) = []
 | assertReals f ((input, output, expl)::t) (desc:string) =
 (assertRealEqual (f input) output (desc ^ ": " ^ expl)) @ (assertReals f t (desc));

fun assertRegulars f [] (desc:string) = []
 | assertRegulars f ((input, output, expl)::t) (desc:string) =
 (assertEqual (f input) output (desc ^ ": " ^ expl)) @ (assertRegulars f t (desc));

Comments?

I have less than 5 hours experience with SML   (I did know LISP in college and recursion certainly isn’t new.)  So I’m sure there are naming conventions and the like that I’m not following.  I look forward to hearing comments on what I did.

Live from TSSJS – Testing in the Cloud with Andrew

Posted on by
1

This afternoon I’m live blogging from TheServerSide Symposium, attending “Breaking all the Rules:  The Myth of Testing and Deployment in the Cloud” presented by fellow CodeRanch Andrew Monkhouse.

1.  CodeRanch History
Andrew opens his talk with a discussion of CodeRanch performance, traffic, and memory usage.  He mentions there have been serious issues in the past that haven’t temporarily brought the performance of the website down to a crawl.  CodeRanch uses a fork of JForum (2008) and staffs around 20 developers among 150 moderators, all volunteer participants, to build a custom community-based forum.

CodeRanch developers use a number of test strategies including smoke tests, load tests, and stress test.

2. Redeploying in the Cloud
The focus of Andrew’s presentation will be to demonstrate how CodeRanch would function in the cloud.  He has allocated a “small EC2 instance” (1.7GB memory, 1 compute unit, moderate I/O performance) within Amazon’s cloud that is useful for quick set up, but extremely small for practical use and reliable testing.

3. Benefits of the Cloud
The cloud provides a relatively clean environment in which it is ‘cheap’ to add extra servers, at least compared to running a local test server which may be allocated for additional processes.  Also, tests can run by anyone, anywhere, which is great for a distributed developer team, since Amazon allows you to assign rights to additional users.

You can easily create, set up, and desotry Amazon Machine Images (AMI), so you do not need to get it right the first time.  Lots of backup storage options, depending on your needs.

4.  Dangers of the Cloud
Testing in the Cloud can be difficult.  First off, it is hard to get absolute numbers because of the nature of the underlying change in hardware happening within the cloud.  Second, calibrating results is difficult.  Also, developers have to remember to shut down servers after they are down with them else other developers will run out of resources.  Finally, network traffic does cost money so testing can be expensive, although hardware is cheaper in cloud.

5.  Test Setup
The performance tests were set up into 3 servers, in part because it turned out to be cost effective to set it up this way based on the individual server options.

  • Application Server running CodeRanch
  • Database Server
  • JMeter Test Server

JMeter was used at the testing suite software given its large variety of features, options, and plugins.

6.  Test Analysis
Andrew discussed some items to keep in mind when testing the server and understanding the results:

  • It is quite possible for the Test Server to have greater load than the Application server, so be sure to monitor all your servers!
  • Try different graphs, they may tell you different things
  • JMeter plugins are your friend, they will help you identify problems with your application server and/or test set up.
  • Don’t forget about ramp up times

7.  Where do we go from here?
Automated weekly performance tests using the latest code base.  Add more stress tests.  Andrew comments there are no plans to move the CodeRanch to the cloud at this time given that we require full-time administrators since we don’t have a dedicated, paid team of administrators.

Conclusion
Andrew presents a compelling argument why you should be testing your applications within the cloud.  It provides a sanitary environment in which to perform a large variety of tests across multiuple servers.  I’ve heard arguments of moving application servers to the cloud, but this is the first time I’ve heard it suggested to migrate all automated testing to the cloud.

Test smells breakout at the server side java symposium

Posted on by
Reply

While I’ve read about this topic a lot, I wanted to come to do the topic to see how Lasse Koskela presents. [I’ve learned so much from him over the years as a moderator at coderanch ] He speaks in the same voice that he writes – casual and easy to follow. I really liked the format: present an anti-pattern and have the audience brainstorm what is wrong with it/how to improve it.  I also like his speaking style, gets away from the podium, engages with the audience, good pace, etc. And good use of humor to get participation going “I’m from Finland, we’re ok with silence”

“it’s a good thing that tests break,it means they are testing something that changes”. “as long as they don’t break for the wrong reasons”

Assertions

  • Primitive assertions – Custom assertions, assertStringContains
  • Overly broad assertion – like the golden master, but not intentional, dangerous because have to change if any small detail changes, more specific are preferred
  • Irrelevant assertions – assert not null and then assert a property on the non null object – let junit throw the null pointer

Organization

  • Split personality – Tests two things, many ways to see – var reuse, calling method under test twice, a blank line for two blocks of code
  • Split logic – don’t use information from elsewhere, it is a mystery to the reader of the test what is happening
  • Note: sometimes a refactoring makes things worse and you need to go back or explore another solution.  Better to fix the problem with the refactored code because it is likely better than the original.

Transparency – how readable is it

  • Setup sermon – Extract overly long/complex setup methods, revisit design to get file system out of the picture, avoid magic numbers “why four”
  • Use multiple setup methods to make clear what is being setup
  • Big string – building an XML file in java is bad.  Instead of. Putting in external file, could create method that takes parameters and builds the XML file. This helps because can clearly deduce the important info without reading through reams of XML.

Maintainability

  • Absolute file path – Don’t depend on the environment, platform specific makes harder. For files could use a temp directory that you wipe clean. That way there isn’t test residue
  • Also, relying on path messes up paths when checking out from a branch, use relative path instead. Or getResourceAsStream()
  • Pixel perfection – a lot of assertions for each pixel in an image, any small change breaks the test even if not meaningful (like golden master pattern). One approach is to make the burden of maintaining less. Grab good image and compare to new image. Tis makes it obvious that it a golden master and requires visual inspection. Another way is to fix the test to use the proper language in the assertions.  that tests the lines are connected without testing the pixels directly  To implement, write custom matcher to test pixels by relative position.
  • Lasse feels adding assertion messages makes the tests harder to read and doesn’t add clarity.

I like how Lasse showed the benefits of looking at your old code. It shows that you’ve learned a lot since then.

I like that some of the examples were in languages other than java. I also learned a lot!  Sorry Scott: I wanted to be at your session, but this one was awesome!