‘Using’ disposable test helpers for C# unit testing Reply

In this post I want to describe a simple idiom for test setup and teardown based on the C# ‘using’ keyword.

Typical Example

Here is a typical sort of unit test in C#. Methods tagged with ‘TestInitialize’ and ‘TestCleanup’
are used to create and clean up external resources required for the test, saving these resources
as instance variables in the test class. The test code itself then refers to these instance variables.

	public class UnitTest1
		private DisposableObject _objectA;
		private DisposableObject _objectB;
		private DisposableObject _objectC;

		public void Initialize()
			_objectA = new DisposableObject();
			_objectB = new DisposableObject();
			_objectC = new DisposableObject();

		public void Cleanup()
			_objectA = null;

			_objectB = null;

			_objectC = null;
		public void DoWhateverTest()
			Assert.IsTrue( _objectA.DoWhatever(_objectB, _objectC));

Moving Fixtures to a Test Helper

There are several things I dislike in the previous example:

  • The setup/cleanup code, which may be dense (think Moq) is inline with the tests, which should be terse
  • The setup/cleanup code has to be duplicated if you have similar sets of tests in other files
  • Tests tend to be written in terms of individual object methods instead of in terms of application semantics.

To address these issues, I like to put all my test fixtures into a helper class.

Now there is just one instance variable. The helper class encapsulates all of the mocking and wiring of
components on which the test depends. We can easily re-use the same helper among multiple test files
if we want. And we have the option of making the resources themselves private to the class,
accessing them only via higher-level methods that use semantics of the problem domain.

I frequently parameterize the constructors of helper classes like these. For example, if I’m
wrapping the mock of a file system, I might specify a ‘bool canWriteFile’ argument in the constructor
and then set up my mocks accordingly. This further insulates actual test code from the mechanics
of setup and mocking.

public class TestHelper : IDisposable
		private DisposableObject _objectA;
		private DisposableObject _objectB;
		private DisposableObject _objectC;
		private bool _disposed;

		public TestHelper(bool shouldBehaveBadly=false)
			_objectA = new DisposableObject();
			_objectB = new DisposableObject();
			_objectC = new DisposableObject();			
                        if( shouldBehaveBadly )

		public void Dispose()
			if (!_disposed)
				_disposed = true;

		public bool A_DoWhatever_To_B_And_C()
			return _objectA.DoWhatever(_objectB, _objectC);

	public class UnitTest2
		private TestHelper _helper;

		public void Initialize()
			_helper = new TestHelper();

		public void Cleanup()
			_helper = null;

		public void DoWhateverTest()
			Assert.IsTrue( _helper.A_DoWhatever_To_B_And_C() );

‘Using’ instead of ‘TestInitialize’

The basic idea of the previous example, i.e. moving the test environment into a helper class, can be done in
any xUnit framework. In the specific case of C# it feels more idiomatic to ditch
the Setup/Cleanup methods and just ‘use’ the helper. In addition to making sure the test file now
contains only tests, this lets us specify different constructor arguments for each test.

	public class UnitTest3
		public void DoWhateverTest()
			using (TestHelper helper = new TestHelper())
		public void DoWhateverFailTest()
			using (TestHelper helper = new TestHelper(shouldBehaveBadly: true))


There, that’s better! Now our tests are short, self-contained, and expressed in the language of the domain rather than the language of the implementation. The disposable pattern has let us make our xUnit tests more behavioral and easier to maintain.

Opinionated Soak Testing Reply


“Opinionated Soak Testing” is a term I coined around 2011 for a particular style of randomized, black-box software testing in which:

  • a randomized test agent explores  interactions with the system (this is the randomized, black-box part, see also fuzz testing)
  • … at a fairly high transaction rate (this is the “soak” part)
  • … while doing the best it can to detect problems (this is the “opinionated” part, see also sanity testing)
 I have found this style of test to be easy to implement for client/server architectures and valuable for detecting problems that are missed by standard unit, integration, and manual tests.  When developing software, I like to run my opinionated soak tests at several points in the development process:
  • as a quick smoke test before checking in code
  • as a prolonged stability check before cutting a release
  • as a load driver when doing performance analysis
Opinionated soak testing is my second favorite kind of software testing, after unit testing.  

Soak Testing

“Soak Testing” is when you throw random input at your program to see how it holds up.  This is useful for testing stability, triggering out unexpected exceptions, and (with the aid of a profiler) conducting performance reviews.

In any client/server or SOA system, it’s instructive to (repeatedly) restart service tiers while running a soak test.  This will often reveal problems with initialization logic.


Example of Soak Testing

Suppose you are writing a system for securities trading, for example a dark pool matching engine.  A soak test for this system might instantiate one or more client connections (representing customers or brokerages) that submit random order flow.  The order flow (unlike in an integration test) does not need to be meaningful, but you probably do want to prefer realistic behavior.  So orders might be randomly priced, but more likely to be near the market than away.  The client connections should probably drop and reconnect, since real clients do that.  Some should fail authentication.  Part of the order flow should be nonsensical, with negative volumes or missing buy/sell instructions.  The goal is that, given enough time, the test will replicate every possible behavior or a real client.  A perhaps surprising benefit of this approach is that the client is quite simple to write — you don’t need to track your volume or worry about order validation, you can just generate navigate a state graph or generate some parameters in a random way.

When you run the soak test, you want to ask questions like these about the system under test:

  • Does the system continue to run?
  • Does the system dashboard accurately reflect the simulated client utilization?
  • Are there warnings, errors, or exceptions in any logs?
  • Does anything weird happen as you stop and start system components?  What if you restart back end components like a database server?  What if you enable and disable network connections?
As the final question suggests, you may need a soak test tool that performs administrative actions as well as client actions.  Or you can write two tools (perhaps a randomized client to run remotely, and a generator of random administrative actions to run on the server).  Be creative!

Opinionated Soak Testing

In order to make a soak test “Opinionated”, your test engine needs to examine system outputs and check their validity.  Because soak testing emphasizes performance over analysis, and because this is form of black-box testing, the amount of checking you can do may be limited.  Therefore, you’re more interested in detecting behavior that is blatantly wrong than in confirming that behavior is absolutely right.


Example of Opinionated Soak Testing

In the trading example above, it’s not possible to validate specific fill volumes or prices.  (Adequate control of the volume pool and market prices, like you would have in an integration test, would be inconsistent with writing clients that are random and fast.)  Despite this limitation, you can ask simple “sanity check” questions, such as:

  • If a client submits an IOC (“immediate or cancel”) order, does it ever fail to complete (fill or expire) within a reasonable amount of time (5 seconds)?
  • If a client submits a FOK (“fill or kill”) order, does it ever fill for less than its total volume?  (Or *more* than its total volume?!)
  • If the system reports positions, do the aggregate position and volume number correctly track reported executions?
The addition of these questions is what makes the test opinionated.  Examples from other domains might include:
  • When operating on a data structure, does the structure remain valid?  For example, if you perform random operations on a doubly-linked list implementation does it remain traversable from front to back and vice versa?
  • When rendering an image, is the average pixel density plausible?  (If your output is all black, you might have a problem.)
  • When navigating a web site, do all pages have expected navigation elements?  Does any operation produce a 404 error?

The simpler your domain is (for example a deterministic data structure) the more complete your validation can be.  The more complicated your domain (for example a peer-to-peer data network) the fuzzier your checks will become.   But there is always some opportunity to combine sanity checking with your soak testing.


First Draft of Web Codex (Rails) Reply

As a future addition to my Gallery here at Lost Bear Labs, I’ve been starting to think about a web service version of my Codex.  This would be a set of comparative implementations of basic functions in Rails, ASP.NET MVC, node.js, and possibly others.  The functionality covered would include basics like:

  • processing POST and GET parameters
  • server-side validation
  • accessing remote resources

Notably *not* included would be client-side logic (javascript, etc) or business logic.  What I’m interested in are the basic nuts and bolts of writing a server.

The project is still a bit wibbly in my mind, but here is a first crack at the Rails implementation:

Using the Minecraft API without a Raspberry Pi — CraftBukkit and RaspberryJuice 2


The Pi Edition of Minecraft includes a simple wire protocol API with bindings in Python.  (Bindings in other languages are also available, for example here is one for Ruby.  The language-specific APIs are just wrappers around some string commands that are transmitted to the Minecraft server via TCP/IP.  The Minecraft installation includes a description of the full syntax.)

If you don’t have a Raspberry Pi, then you can’t currently use the official Minecraft API.  (The desktop versions of the game don’t support it.)  This post describes how to get started with CraftBukkit (an open-source replacement for the Minecraft server) and Raspberry Juice (a CraftBukkit plugin that emulates the Minecraft API support) so that you can run Minecraft API programs using your desktop environment.

These instructions were developed on a Mac, but they should be easily adaptable to other environments.


Running CraftBukkit

There are good instructions for downloading and installing CraftBukkit at  http://wiki.bukkit.org/Setting_up_a_server.

I am running the most recent version of Minecraft (currently v1.5.1) so the most recent “recommended” version of CraftBukkit is not yet compatible.  Therefore I looked at the download page and selected the most recent “Beta Build” version for v1.5.1 (currently v1.5.1-RO2) and put this into my BukketServer folder.  I then modified my “start.command” file (as described in the instructions above) to load this version instead of the default version by writing:

java -Xms1024M -Xmx1024M -jar craftbukkit-1.5.1-R0.2.jar -o true


With my start.command file set up, I can then start the server by using a terminal window and typing:

cd ~/Desktop/BukkitServer

The server runs as a console program within the terminal window.  To stop the server, I type


For a list of other available commands, you can type


(The commands can also be used via Minecraft chat, but since I’m already switching around to my code window and keeping an eye on the terminal while coding, it’s just as easy to use this console directly.)


Connecting To Your Server

When you start Minecraft, select “Multiplayer” mode, and then define a server with the following properties:



When you then click the “Join Server” button, you will see a connection message in your CraftBukkit console window:

11:45:41 [INFO] TestUser[/] logged in with entity 
id 150 at ([world] -236.2425485963545, 67.0, 232.225211454399)


The Raspberry Juice Plugin 

Once you have a working server and can connect to it, you are ready to add the Raspberry Juice plugin that implements the Minecraft API.

Announcements about this plugin can be found here and the source code is on GitHub.

To install it, find the download link at the main plugin page, http://dev.bukkit.org/server-mods/raspberryjuice/ .  This will download a JAR file (currently raspberryjuice-1.2.jar).  Copy this JAR file into the folder called plugins under your main CraftBukkit folder.

Now when you restart your server, you should see a message like this and you are good to go:

1:44:02 [INFO] [RaspberryJuice] Enabling RaspberryJuice v1.2

Imitating Creative Mode

If your main goal is to experiment with script programming, then you may not want to deal with monsters, night-time, etc.  (This is how the Pi edition works… it runs only in creative mode.)  


To make your server default to monster-free, creative mode play, locate the file server.properties in your main CraftBukkit folder and set the following values:


 (Note:  the allow-flight setting is important.  If you don’t enable it, and then you run a script that erases the ground beneath you, the server will kick you out for “illegally” flying!)


To get rid of day/night transitions, install the Always Day plugin (copy the JAR file to your plugins folder).  Then just type the command “day” in your server’s console window.


Installing the Minecraft API

When you download Minecraft: Pi Edition, you will get a file called minecraft-pi-0.1.1.tar.gz.  Extract this file, and you will get a folder called mcpi  Within this is a subfolder called mcpi/api/python/mcpi.  This is the python API, and you can copy it around freely to your development folder.

Note that the neighboring folder, mcpi/api/python/spec, contains the description of the wire format used to send API commands to the server.


Writing your First Program

First, create a development folder for yourself, for example you could call it mcdev:

mkdir mcdev


 Now copy your downloaded API into this folder:

cd mcdev
cp -r ~/Downloads/mcpi/api/python/mcpi . 


 Within this folder, create a file called test1.py with the following contents:

# Sample program to create a pillar
import mcpi.minecraft as minecraft
import mcpi.block as block
# Connect to the Minecraft server
world = minecraft.Minecraft.create()
# Get the player's current position and store the coordinates
[x,y,z] = world.player.getPos()
# Set some variables to customize your pillar
height = 3
material = block.BRICK_BLOCK
# Build the pillar. It will be "height" blocks high and located one step away from the player.
for level in range(0, height):
world.setBlock( x+1, y+level, z+1, material )
level = level + 1;

(NOTE:  if you’re not a Python programmer [yet!] then be aware that the indentation in the last two lines is important.  These two lines must each be indented at the beginning with a tab (or 2 spaces) exactly as shown.  Unlike many other languages, Python pays attention to whitespace.)


Finally, execute your program by typing the following at the command line from within your mcdev folder.  

python test1.py


When you return to the game, your character will be standing next to a small brick pillar!


Untitled 2

In which Nokogiri Installs Just Fine Reply

 In response to my previous post, a friend who has done some professional development with rails wrote:

Yeah – Nokogiri on OSX is it’s own special adventure.  You can’t really blame ruby or rails for that – I think there is some way to run a non-compiled version of nokogiri, but the performance would be so painful, you’d really want the compiled libraries anyway.  I fought with it for about 4 days in Summer 2011 and have never had to muck with it since (once I got it to compile successfully)

I’m offended by the idea that professional developers might be wasting 4 days apiece to install a component that is so widely used.  But just as I was working up a head of steam to plow through the problem myself and document what I learned along the way, I got a perfectly smooth install.  Between my first attempt and now, I had installed all the prerequisites recommended by ‘rvm requirements’ (in the presence of homebrew), and that seems to have made all the difference.

Anyway, the beginning of today’s surprisingly easy experiment was to write a script that will do gem installation across all my installed rubies.  To run an install, it just repeatedly executes commands like this:

   $ rvm ruby-1.9.1-p431 do install rdoc

Running this with json_pure, which is an all ruby gem, I get output like the following (*see note 1*):

    $ ./TryGem.rb install json_pure
********************* ruby-1.8.6-p420 ******************
    Successfully installed json_pure-1.7.7 1 gem installed
Installing ri documentation for json_pure-1.7.7...
Installing RDoc documentation for json_pure-1.7.7...
********************* ruby-1.9.1-p431 ******************
Successfully installed json_pure-1.7.7
1 gem installed
Installing ri documentation for json_pure-1.7.7...
Installing RDoc documentation for json_pure-1.7.7...
... etc. 

To check whether a gem is installed, the script runs a command like this and looks for the desired gem name in the output:

    gem query –local

which yields a list like this:

    ruby-1.8.6-p420 json_pure (1.7.7)
    ruby-1.9.1-p431 json_pure (1.7.7)
    ruby-1.9.2-p320 json_pure (1.7.7)
ruby-1.9.3-head json_pure (1.7.7)
ruby-1.9.3-p125 json_pure (1.7.7)
ruby-1.9.3-p194 json_pure (1.7.7)
ruby-1.9.3-p286 json_pure (1.7.7)
ruby-1.9.3-p327 json_pure (1.7.7)
ruby-1.9.3-p362 json_pure (1.7.7)
ruby-1.9.3-p374 json_pure (1.7.7)
ruby-2.0.0-rc1 json_pure (1.7.7)
ruby-head json_pure (1.7.7)

Having tried all this with json_pure, I was ready to move on to nokogiri (which relies on native code components) and was delighted to get a clean install for all the up-to-date rubies:

    ruby-1.8.6-p420 (NOT INSTALLED) (*see note 2*)
ruby-1.9.1-p431 nokogiri (1.5.6)
ruby-1.9.2-p320 nokogiri (1.5.6)
ruby-1.9.3-head nokogiri (1.5.6)
ruby-1.9.3-p125 nokogiri (1.5.6)
ruby-1.9.3-p194 nokogiri (1.5.6)
ruby-1.9.3-p286 nokogiri (1.5.6)
ruby-1.9.3-p327 nokogiri (1.5.6)
ruby-1.9.3-p362 nokogiri (1.5.6)
ruby-1.9.3-p374 nokogiri (1.5.6)
ruby-2.0.0-rc1 nokogiri (1.5.6)
ruby-head nokogiri (1.5.6)


(*note 1*) Some fine print:  Each section of output actually begins with a line that says:

    warning: parenthesize argument(s) for future version 

 This seems only to affect v1.8.6 and it doesn’t seem to cause any other problem, so I’m ignoring it!


(*note 2*) The error with v1.8.6 is expected, and since I don’t actually need to run that version I haven’t pursued strategies such as installing earlier gem versions:

    $ rvm ruby-1.8.6-p420 do gem install nokogiri
ERROR: Error installing nokogiri:
nokogiri requires Ruby version >= 1.8.7.

Installing Ruby Versions on OS X (Mountain Lion) Reply

It had been a while since I played with Rails, so I was going back over the Rails Tutorial.  Inevitably, mid-way through, I got stuck because some step required some gem that depended on nokogiri, which would not install.

After the usual pause for frustration, I realized that I had never entirely gone back to first principles with the Ruby stack on OS X.  And despite my general intention to use Linux VMs for my execution environment, this seems like it should be a no-brainer.  The Ruby community relies heavily on OS X, surely everything should be seamless by now?

Or maybe not.  Like the language itself, this ecosystem is loosely coupled.  Allow the AppStore to upgrade your version of XCode, and the next thing you know ‘bundle install’ fails because some random gem that compiles from C++ source isn’t compatible with the new LLVM-based  compiler.  Or whatever.

After some flailing around research, I decided: Let the pillars of my universe be Homebrew and rvm.  I’d used rvm in the past, Homebrew didn’t look any more annoying than MacPorts, and a little experimentation showed that the ‘rvm requirements’ command rather pleasingly detected the presence of Homebrew and spat out some prerequisites, all of which installed fairly easily.

    Requirements for osx/10.8/x86_64
    # For update-system:
sudo brew update
sudo brew tap homebrew/dupes
    # For rvm:
sudo brew install bash curl git
    # For ruby:
sudo brew install autoconf automake apple-gcc42 libtool pkg-config
openssl readline libyaml sqlite libxml2 libxslt libksba

So then the question, what rubies could I run?  I wrote a little script that parses the output from ‘rvm known’ and ‘rvm list’ and can invoke ‘rvm install’ and ‘ruby -v’.

Using this, I was able to install all but one of the rubies that rvm claims to support on my system:

     ***** INSTALLED (12) ****
***** NOT INSTALLED (1) ****

We can double-check that these all execute ok using “rvm all”:

    $ rvm all do ruby -v
ruby 1.8.6 (2010-09-02 patchlevel 420) [i686-darwin12.2.1]
ruby 1.9.1p431 (2011-02-18 revision 30908) [i386-darwin12.2.1]
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-darwin12.2.1]
ruby 1.9.3p386 (2013-02-13) [x86_64-darwin12.2.1]
ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0]
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin12.2.1]
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12.2.1]
ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-darwin12.2.1]
ruby 1.9.3p327 (2012-11-10) [x86_64-darwin12.2.0]
ruby 1.9.3p362 (2012-12-25 revision 38607) [x86_64-darwin12.2.1]
ruby 1.9.3p374 (2013-01-15 revision 38858) [x86_64-darwin12.2.1]
ruby 2.0.0dev (2013-01-07 trunk 38733) [x86_64-darwin12.2.1]
ruby 2.0.0dev (2013-02-16) [x86_64-darwin12.2.1]

The only version that would not install is 1.8.7-371, which looks unlikely to be useful.  (rvm suggests “Please consider upgrading to ruby-1.9.3-p374″).  I can live with that.  So now I have a quick and repeatable way to test what rubies are actually available for me to work with.  

Next time:  Some gems?

Reviving the Codex Reply

The Codex is a set of small coding problems implemented across many languages.  I started writing these way back in 2007 and let them lapse for a couple years.  Since I’ve been adding new languages like CoffeeScript to my toolbox recently, it seemed like a good project to revive.

I don’t try to be exhaustive in covering these examples.  For example ‘FoldMapUCase’ exercises the basics of higher order functions;  it was (barely) possible to implement in PHP but not (at least then) in RealBasic, and certainly it’s been easy to re-implement in ruby and CoffeeScript.  ‘EchoSrv’ exercises basic networking, so it didn’t make sense to implement in (browser-hosted) JavaScript, but now I have implemented it in CoffeeScript using node.  If I think of a new example, I’ll implement it in the languages I’m using today and leave others by the wayside.

These examples originally lived on an SVN host, but now that we live in the future I’ve moved them all to github.

Exploring the CoffeeScript Unit Tests Reply

I’m a fan of CoffeeScript and used it last year to update all the little animations in my Gallery.  So this weekend I decided to spend some time learning how the interpreter is validated.  The answer (as you would hope) is that there are unit tests.  They are pleasantly simple to run and decipher.  After you install node, install CoffeeScript, and clone the CoffeeScript git repository, it’s a one-line operation to run the tests:

$ cake test
passed 431 tests in 1.10 seconds

There is also a browser-hosted file (./tests/test.html) which exercises the test cases in your browser instead of via the command-line node interpreter.

The unit tests themselves are defined in CoffeeScript files in the folder ./test, and an example test looks like this:

test "array splat expansions with assignments", ->
nums = [1, 2, 3]
list = [a = 0, nums..., b = 4]
eq 0, a
eq 4, b
arrayEq [0,1,2,3,4], list 

The harness to run these tests is defined directly within the project’s Cakefile:

  global.test = (description, fn) ->
fn.test = {description, currentFile}
catch e
e.description = description if description?
e.source = fn.toString() if fn.toString?
failures.push filename: currentFile, error: e
  global.eq = (a, b, msg) -> ok egal(a, b), msg
  global.arrayEq = (a, b, msg) -> ok arrayEgal(a,b), msg 

These function ‘ok’ used in these definitions was initially mysterious to me, until I noticed a line in the Cakefile that says:

global[name] = func for name, func of require 'assert'

This line imports the standard node ‘assert’ module and copies its methods into the global namespace.  A simpler, partial example of doing the same thing would be to write:

assert = require 'assert'
global.ok = assert.ok
ok false  # this is defined now, and raises an exception as expected

So that is a helpful reminder that “CoffeeScript is just javaScript” and that it plays well with other libraries.  In the browser-hosted version of the test suite (./tests/test.html) the node environment is not available so the ‘ok’ function must be defined explicitly:

@ok = (good, msg) ->
if good then ++success else throw Error say msg

Equilibrist v0.2 Reply

Equilibrist v0.2 is now available.  It fixes the two bugs mentioned previously:

  • on the “About” page, the project page link now points to a valid URL
  • while the timer is active, the phone will not go to sleep

This version also improves the main display so that the motion threshold and the motion history as shown graphically instead of with text:



Equilibrist v0.1 Reply

I’m please to announce that equilibrist, my first iOS application, is available for free on iTunes.


For the last couple years I’ve wanted an iPhone app that will let me know if I’m keeping the reformer carriage still while performing Pilates exercises such as “picking flowers” or “embrace the moon”.  I also envisioned using it for yoga balance practice on the ground and on the slackline.

Equilibrist v0.1 is my first crack at writing such an application.  It is a simple iOS app combining a timer with simple use of the accelerometer.  All of the code I needed to create it was available as snippets online, so there was nothing too difficult to figure out.  The app is free and open-source, and you can see the code on GitHub.

For a fuller description of what Equilibrist does and how to use it, please see the Equilibrist Project Page.  Bug reports, feature suggestions, and code pull requests are all welcome.


Version 0.1 already has two known bugs, so v0.2 will be on the way soon:

  • on the “About” page, the project page link points to an old URL
  • the application should prevent the phone from going to sleep while actively timing

I would also like to add a graphical display of the motion threshold and the detected motion, perhaps in a scrolling bar graph format.