The Problem
A while ago I and Jari got the opportunity to implement a small piece of software without any limitations on which tools to use. The assignment was to create a central server for an algorithm competition where teams have to implement a server that solves certain variants of Knapsack problems. The idea is that during a competion round, the competion server sends a bunch of challenges to the contestant servers and they have to produce a response in a given time. Each challenge is a Knapsack problem represented in a JSON message. If a contestant fails to produce a result in the given time, it is disqualified. Successful results are ranked and scored. The central server also visualizes the competition in real time by sending events to a web-based visualization client.
The Technology
We chose to use node.js for the server and HTML+CSS+JQuery for the visualization client, at least to start with. It turned out that this was a good match and we didn't have to go for Flash or such on the client side. To get the real-time updates to the client, we decided to give Socket.IO a try. And it was a good bet too. As supporters of TDD, we decided to do the whole thing test-driven, and to do most of the testing using integration tests. As a result, we had quite a complete integration test suite for the whole thing, without faking or mocking any part of the server. And the test suite took just 200 milliseconds to run, thanks to the asynchronous testing framework provided by Vows.
All this technology made us feel like kids in a candy store. Still, we spent most of our time working on the actual problem and not on accidential complexity like persistence, serialization or concurrency. Here's the stuff that made us happy:
- Node.js makes concurrent programming a non-issue by removing all concurrency in actual application code and making all "system" calls asynchronous. No need to think about thread-pools, locking, deadlocks or the treacherous Java Memory Model.
- Node.js also makes it very easy to create HTTP clients and servers.
- Vows makes testing of asynchronous things easy. No need to sleep 5 seconds before checking if something has happened. Instead, use a callback to react to the thing you're waiting for to happen.
- Socket.IO allows you to send messages between your browser client and your node.js server, both ways.
- Using JSON in all communications makes serialization a non-issue. Both node.js and JQuery have JSON serialization, so we just stringify() and parse() our objects and be happy.
TDD
I hate bugs, debugging and testing. That's why I always strive to go test-first. Especially when working with new technology. Even more so when working with a dynamic language like Javascript where there's no compiler going to save my ass. Lately I've been working with Haskell too, where there program usually works if I get it to compile..
Anyways, some things are easier to test than others and sometimes it may be practical to draw a line between what's test-driven and what's not. Here we had the prototype of an easily testable piece of software: a server that communicates in JSON with other servers, producing JSON events for a user interface client. So, we decided to fake the contender servers and the UI and test the whole thing from the outside.
So, let's fake. I mean, let's create fake instances of the contender servers, as well as the visualization client. Then, we'll build a test suite that uses these fakes to test-drive the actual server we're developing. I'll paste some slightly simplified pieces of code here for example. If you are really interested in the stuff, you'll find it all on Github.
Faking the Contender Servers
Anyways, some things are easier to test than others and sometimes it may be practical to draw a line between what's test-driven and what's not. Here we had the prototype of an easily testable piece of software: a server that communicates in JSON with other servers, producing JSON events for a user interface client. So, we decided to fake the contender servers and the UI and test the whole thing from the outside.
So, let's fake. I mean, let's create fake instances of the contender servers, as well as the visualization client. Then, we'll build a test suite that uses these fakes to test-drive the actual server we're developing. I'll paste some slightly simplified pieces of code here for example. If you are really interested in the stuff, you'll find it all on Github.
Faking the Contender Servers
Faking the contender servers was easy, as it only takes a few lines of code to create an HTTP server and have it reply to the challenges from the central server. (see contender.js) Here's the beef:
this.start = function() {
this.server = http.createServer(this.handler);
this.server.listen(this.port, this.host);
return this;
};
this.handler = function (req, res) {
extractContent(req, function(challengeJson) {
};
this.start = function() {
this.server = http.createServer(this.handler);
this.server.listen(this.port, this.host);
return this;
};
this.handler = function (req, res) {
extractContent(req, function(challengeJson) {
var challenge = JSON.parse(challengeJson);
testClient.challenges.push(challenge);
var result = resultStrategy(challenge)
var resultJson = JSON.stringify(result);
res.writeHead(200);
res.write(resultJson);
res.end();
})};
So, we create an HTTP server with a given handler function that parses the input JSON into challenge, then saves the challenge for later inspection and finally replies using a given resultStrategy. The extractContent function is a helper that consumes all input from the HTTP request and calls a callback when done (see contentextractor.js):
extractContent = function(response, contentHandler, encoding) {
if (encoding) response.setEncoding(encoding);
var content = "";
response.on('data', function (chunk) {
content += chunk;
});
response.on('end', function() {
contentHandler(content)
});
}
Faking the Socket.IO Client
Faking the visualization UI was a bit harder, as Socket.IO does not provide a client library for node.js, where we run our tests. Only a browser client is included. We quickly found Websocket Client, but ran into trouble pretty soon because there were no suitable examples for us. But after some guesswork we figured out how to set up a client and have it receive messages from the server. The amount of guesswork needed was actually the trigger for me to write this post, because I want to share this with the other TDD aficionados out there.
Faking the visualization UI was a bit harder, as Socket.IO does not provide a client library for node.js, where we run our tests. Only a browser client is included. We quickly found Websocket Client, but ran into trouble pretty soon because there were no suitable examples for us. But after some guesswork we figured out how to set up a client and have it receive messages from the server. The amount of guesswork needed was actually the trigger for me to write this post, because I want to share this with the other TDD aficionados out there.
So, here's the piece of code required to connect to the node.js server running Socket.IO. (see websocket-client.js)
var WebSocketImpl = require('websocket-client').WebSocket
WebSocketClient = function(port) {
return new WebSocketImpl(
'ws://localhost:' + port + '/socket.io/websocket')
}
var WebSocketImpl = require('websocket-client').WebSocket
WebSocketClient = function(port) {
return new WebSocketImpl(
'ws://localhost:' + port + '/socket.io/websocket')
}
To receive messages, you need to assign an onmessage handler as in (see message-queue.js)
function handler(message) { console.log(message) }
var ws = WebSocketClient(port);
ws.onmessage = function(message) {
/*
* The messages are prefixed with a header that contains
* the size of the actual payload. Since we just want the
* JSON from the message, it's easy to find the first
* curly brace and go from there.
*/
var data = message.data;
var jsonStart = data.indexOf('{');
if(jsonStart >= 0) {
var jsonString = data.substring(jsonStart)
var jsonObject = JSON.parse(jsonString)
handler(jsonObject)
}
};
function handler(message) { console.log(message) }
var ws = WebSocketClient(port);
ws.onmessage = function(message) {
/*
* The messages are prefixed with a header that contains
* the size of the actual payload. Since we just want the
* JSON from the message, it's easy to find the first
* curly brace and go from there.
*/
var data = message.data;
var jsonStart = data.indexOf('{');
if(jsonStart >= 0) {
var jsonString = data.substring(jsonStart)
var jsonObject = JSON.parse(jsonString)
handler(jsonObject)
}
};
Note that actual message handling is delegated to a function named handler in the above example.
Asynchronous Testing with Vows
This is not going to be an introduction to Vows, as you'll find such at the Vows site. This is more like a case study and a presentation of some modest tools we created.
The main test suite, in the file competition-round-test.js, sets up a server with 2 challenges that will be sent to 4 contestant servers. Two of the contestants are set up to return a simple answer with different strategies each. One contestant is set up to be a bit too slow so that the server should time-out before getting the answer from this one. The fourth contestant always sends an invalid answer. A fake visualization client is connected to the server, and it will be used to check that the server sends correct messages at each stage of the competition round.
When all this is set up, the server is started and it's output to both the contenders and the visualization client is monitored asynchronously using Vows. The very first test is like this:
var initialization = {
"When visualization client connects" :
expectMessage("initialization message is sent", {
message : "init",
contenders : [{name : "TestContender8200"},
{name : "TestContender8201"},
{name : "TestContender8202"},
{name : "TestContender8203"},
{name : "TestContender8204",
rabbit : true}],
challenges : [
{name : "Eka", numberOfItems : 2,
capacity : [99], timeout : 50},
{name : "Toka", numberOfItems : 2,
capacity: [991], timeout : 150}]})
};
Pretty clear? Should be. The expectMessage function there returns a Vows asynchronous test context that waits for a message to the fake visualization client and asserts that it is equal to the given message. In this case, it expects an initialization message that describes the contenders and the challenges. Here's the implementation:
function expectMessage(description, expectedMessage) {
return {
topic: function() {
messageQueue.waitForMessage(this.callback)
},
description: function(message, _) {
assert.deepEqual(message, expectedMessage);
}
};
}
It's a Vows convention that there's a topic function in each test context. This function is called by the framework. In expectMessage the topic registers a message listener to the messageQueue. It actually hands over the vows callback function (this.callback) that the messageQueue will call when a message is received. Vows will then run each test case and pass the arguments of the callback to the test cases. In this example, the description test case will receive the message from messageQueue and makes an assertion on the message. The extra underscore (_) parameter in the description function was added because Vows seems to require at least two parameters. I guess this is a bug in the Vows version we used.
Are you wondering what the messageQueue is? Well, that's a simple message queue of ours that's plugged into the fake visualization client. The source code for the message queue, as well as the rest of the fake visualization client can be found in message-queue.js.
The rest of the competion round test is pretty much the same as the first case. One more tool is introduced though, for situations where we are expecting, say, 5 messages, the ordering of which is not important. For that, we created MessageBatcher (also in message-queue.js) where the method waitForMessages calls the given callback when a given number of messages has been received. Using, this the results of the first challenge are tested like this:
var firstChallengeResults = {
"When contenders finish first challenge" : {
topic : function() {
messageBatcher.waitForMessages(5, this.callback);
},
'timeout was 50ms': function(_, _) {
assert.equal(round.challenges[0].timeout, 50)
},
'100ms contender fails': function(messages, _) {
assertContains(messages,
{message: "contenderFail", challengeName: "Eka",
contenderName : "TestContender8200"});
},
'client with ok result succeeds': function(messages, _) {
assertContains(messages,
{message: "contenderReady", challengeName: "Eka",
contenderName : "TestContender8201", value: 100,
weight : [10]});
},
'client with overweight fails': function(messages, _) {
assertContains(messages,
{message : "contenderFail", challengeName: "Eka",
contenderName : "TestContender8202"});
},
'invalid result fails': function(messages, _) {
assertContains(messages,
{message : "contenderFail", challengeName : "Eka",
contenderName : "TestContender8203"});
}
}
};
Lastly, we also want to verify that the contestants are presented with correct challenges. That's pretty easy, as we have our fake contenders capturing all input. So, in the last test batch, we'll do it like this:
var challengesSentToContenders = {
"For first challenge" : {
"correct challenge is sent" : function() {
assert.deepEqual(
client1.challenges[0], round.challenges[0])
}
},
"For second challenge" : {
"correct challenge is sent" : function() {
assert.deepEqual(
client1.challenges[1], round.challenges[1])
}
}
};
So, did it work?
The competion was a success. I actually also participated in the actual competition too, as a part of a great Haskell team and it was fun. The competion server and the visualization there worked pretty nice, except for some occasional hangs caused by contestants returning surprising answers like Haskell error messages instead of JSON :) So, maybe the competition server should have been tested more with different kinds of invalid answers..
Problems..
Unfortunately, the guys who ran the competion were unable to get the Vows tests running on their machines. So there might be some mismatch with the most recent node.js and Vows versions.
I'm running node 0.2.5 and vows 0.5.8 without problems. So, please clone competition-server, install as in README, then run the tests with run-tests.sh, and tell me if it worked ok.
This is not going to be an introduction to Vows, as you'll find such at the Vows site. This is more like a case study and a presentation of some modest tools we created.
The main test suite, in the file competition-round-test.js, sets up a server with 2 challenges that will be sent to 4 contestant servers. Two of the contestants are set up to return a simple answer with different strategies each. One contestant is set up to be a bit too slow so that the server should time-out before getting the answer from this one. The fourth contestant always sends an invalid answer. A fake visualization client is connected to the server, and it will be used to check that the server sends correct messages at each stage of the competition round.
When all this is set up, the server is started and it's output to both the contenders and the visualization client is monitored asynchronously using Vows. The very first test is like this:
var initialization = {
"When visualization client connects" :
expectMessage("initialization message is sent", {
message : "init",
contenders : [{name : "TestContender8200"},
{name : "TestContender8201"},
{name : "TestContender8202"},
{name : "TestContender8203"},
{name : "TestContender8204",
rabbit : true}],
challenges : [
{name : "Eka", numberOfItems : 2,
capacity : [99], timeout : 50},
{name : "Toka", numberOfItems : 2,
capacity: [991], timeout : 150}]})
};
Pretty clear? Should be. The expectMessage function there returns a Vows asynchronous test context that waits for a message to the fake visualization client and asserts that it is equal to the given message. In this case, it expects an initialization message that describes the contenders and the challenges. Here's the implementation:
function expectMessage(description, expectedMessage) {
return {
topic: function() {
messageQueue.waitForMessage(this.callback)
},
description: function(message, _) {
assert.deepEqual(message, expectedMessage);
}
};
}
It's a Vows convention that there's a topic function in each test context. This function is called by the framework. In expectMessage the topic registers a message listener to the messageQueue. It actually hands over the vows callback function (this.callback) that the messageQueue will call when a message is received. Vows will then run each test case and pass the arguments of the callback to the test cases. In this example, the description test case will receive the message from messageQueue and makes an assertion on the message. The extra underscore (_) parameter in the description function was added because Vows seems to require at least two parameters. I guess this is a bug in the Vows version we used.
Are you wondering what the messageQueue is? Well, that's a simple message queue of ours that's plugged into the fake visualization client. The source code for the message queue, as well as the rest of the fake visualization client can be found in message-queue.js.
The rest of the competion round test is pretty much the same as the first case. One more tool is introduced though, for situations where we are expecting, say, 5 messages, the ordering of which is not important. For that, we created MessageBatcher (also in message-queue.js) where the method waitForMessages calls the given callback when a given number of messages has been received. Using, this the results of the first challenge are tested like this:
var firstChallengeResults = {
"When contenders finish first challenge" : {
topic : function() {
messageBatcher.waitForMessages(5, this.callback);
},
'timeout was 50ms': function(_, _) {
assert.equal(round.challenges[0].timeout, 50)
},
'100ms contender fails': function(messages, _) {
assertContains(messages,
{message: "contenderFail", challengeName: "Eka",
contenderName : "TestContender8200"});
},
'client with ok result succeeds': function(messages, _) {
assertContains(messages,
{message: "contenderReady", challengeName: "Eka",
contenderName : "TestContender8201", value: 100,
weight : [10]});
},
'client with overweight fails': function(messages, _) {
assertContains(messages,
{message : "contenderFail", challengeName: "Eka",
contenderName : "TestContender8202"});
},
'invalid result fails': function(messages, _) {
assertContains(messages,
{message : "contenderFail", challengeName : "Eka",
contenderName : "TestContender8203"});
}
}
};
Lastly, we also want to verify that the contestants are presented with correct challenges. That's pretty easy, as we have our fake contenders capturing all input. So, in the last test batch, we'll do it like this:
var challengesSentToContenders = {
"For first challenge" : {
"correct challenge is sent" : function() {
assert.deepEqual(
client1.challenges[0], round.challenges[0])
}
},
"For second challenge" : {
"correct challenge is sent" : function() {
assert.deepEqual(
client1.challenges[1], round.challenges[1])
}
}
};
So, did it work?
The competion was a success. I actually also participated in the actual competition too, as a part of a great Haskell team and it was fun. The competion server and the visualization there worked pretty nice, except for some occasional hangs caused by contestants returning surprising answers like Haskell error messages instead of JSON :) So, maybe the competition server should have been tested more with different kinds of invalid answers..
Problems..
Unfortunately, the guys who ran the competion were unable to get the Vows tests running on their machines. So there might be some mismatch with the most recent node.js and Vows versions.
I'm running node 0.2.5 and vows 0.5.8 without problems. So, please clone competition-server, install as in README, then run the tests with run-tests.sh, and tell me if it worked ok.
