Unit Testing Prefix with V8, V8Sharp and PetaTest (and hints on what's coming next)

Tuesday, August 30th, 2011     #javascript #everything

In developing Prefix I needed a reliable, flexible and fast unit testing framework. To build this I used V8 (Google's JavaScript Engine from Chrome), V8Sharp (a .NET wrapper for V8) and PetaTest (my tiny unit testing framework).

This post is about Prefix a modern language that compiles to JavaScript. A language where C# is the inspiration - not the goal. Read more on the Prefix Project Page.

Unit Testing JavaScript

This isn't the first time I've had to run JavaScript code for unit testing. Previously what I've done is hosted the Microsoft Web Control, loaded a page with the appropriate JavaScript code and interoped to it through a Windows Form application. This is how I implemented the unit tests for the JavaScript port of MarkdownDeep - it works, but it's messy and very slow.

For Prefix, I wanted something better so I started looking around for a JavaScript engine that would be easy to host in .NET. I knew about Google's V8 JavaScript engine so that seemed the obvious place to start. I got it building and was about to start on some C++/CLR/IJW interop code for it when I stumbled upon V8Sharp - which does exactly what I needed.

Hosting V8 with V8Sharp is trivial, here's the bit to run some JavaScript code:

var v8 = v8sharp.V8Engine.Create();
v8.Register<TestConsole>("Console", console);
v8.Execute(jsActual);

TestConsole is a simple class that V8Sharp makes available to the script and I use it to route text output to the Console:

public class TestConsole
{
    public TestConsole()
    {
    }
    public void Write(object output)
    {
        Console.Write(output.ToString());
    }
    public void WriteLine(object output)
    {
        Console.WriteLine(output.ToString());
    }
}

So from JavaScript code I can now do this:

Console.WriteLine("Hello World (from JavaScript)");

Building the Rest of the Unit Testing Framework

Next I brought V8 and V8Sharp together with PetaTest. The unit test program has a single test fixture and a single parameterized test case. That one test case loads one of many test scripts from an embedded resource (it could be an external file if necessary). Currently I have about 350+ of these test scripts and I've setup PetaTest so that I can invoke one, all or a subset of tests - depending on what I'm working on.

Format of the Test Scripts

Each unit test script is a text file that looks like this:

extern dynamic Console
Console.WriteLine("Hello World from Prefix");
----
Hello World from Prefix

The test script is divided into two sections separated by a dashed ---- delimiter. The top half is the Prefix code to be compiled and executed. The bottom half is the expected output.

Testing Error Reporting

Of course a compiler needs to generate not just a correctly functioning output program, it should also generate errors when there are problems in the input program. To test these, the expected output can specify an exception that should be generated.

eg:

extern dynamic Console;

public enum MyEnum
{
    a,
    b,
    c,
}

MyEnum.a=12;

-----
// exception: The left hand side of an assignment must be a variable, property or indexer

In compiling this script the unit test will fail if the compiler doesn't generate an exception containing the specified text.

Closer Inspection of the Generated JavaScript

Usually the correctness of the generated script can be confirmed by examining the expected output. There are cases however where I need to ensure that the correct output was generated in a particular way. For this there are a couple of additional directives that can be embedded in the input script to ensure the generated script does or does not contain particular content.

// should generate: Foo.prototype.str = "OK";
// should not generate: this.str = 

extern dynamic Console;

class Foo
{
    public string str = "OK";
} 

Console.WriteLine(new Foo().str);

-----
OK

Note the should generate: and should not generate: directives at the top of the script.

Compiler Options

Finally there are a few other directives that can be used to tweak how the Prefix compiler is invoked.

  • // no warnings - by default the test cases are run with warnings as errors (so warnings can be tested). This option disables this.
  • // release mode - run the compiler in release mode.

(Note these are directives to the test case framework - not the actual Prefix compiler).

Example output

Here's the current set of unit test results. Right now (late Aug 2011) there are about 350 unit tests, all passing except 1 - which I'm working on.

If you click on the name of a unit test it will expand to show the input Prefix program, the expected output, the generated JavaScript and the results of the test.

Conclusion

I'm pretty happy with this testing environment. It's fast - 350+ unit tests in under 2 seconds, flexible and I get a nice output report thanks to PetaTest - the linked report above is exactly what PetaTest generates - I haven't modified or re-formatted it in any way.

« Older - Prefix takes One Giant Leap with Support for Generics Newer - Prefix - Support for Interfaces »