Link Search Menu Expand Document

Cmaj Unit Test Files

The test file format allows you to create test files which run multiple scripted tests of Cmajor code.

Javascript can be used to write custom functions that can perform kind of test, but a set of common helper functions are also provided.

A typical test might load a Cmaj program, check for compile errors, pass some inputs into it and check that the expected outputs are emitted.

Running Tests

To run a test file, you can use the command-line tool:

$ cmaj test my_tests/TestFile1.cmajtest

You can also provide a folder instead of a filename, and it will scan for all the tests within that folder, and print out a set of total results for all of them.

The command-line switches include:

--singleThread      Use a single thread to run the tests
--threads=n         Run with the given number of threads, defaults to the available cores
--runDisabled       Run all tests including any marked disabled
--testToRun=n       Only run the specified test number in the test files
--xmlOutput=file    Generate a JUNIT compatible xml file containing the test results
--iterations=n      How many times to repeat the tests

Run cmaj --help for more command-line option information.

File Format

A test file is split into blocks which are handled as separate tests.

Blocks are separated by a line starting with the characters ##, followed by a javascript function call which performs a test function on the remaining content of the block that follows it.

e.g.

// All the text before the first delimiter line is parsed as javascript, and is available globally to all the tests. This is where you'd put any shared helper functions that your tests need to use.

## testFunction()

...The chunk of text in each section is provided to the test function that is called above...

## expectError ("...")

...Likewise, this chunk of text is passed to the "expectError" function...

Any content at the start of the file, up to the first ## line, is parsed as Javascript. This is where you can define custom test functions which can be used multiple times within the file.

Special Section Delimiters

## global

The special delimiter ## global is used to declare a chunk of code which will be prefixed onto all the other tests in the file. So if you’re running many tests which all share a set of types or functions, you can avoid repeated code by putting the common definitions in a global section.

## disabled [test...]

To quickly disable a test, insert the token disabled in front of the test directive (anything following disabled on the line is ignored), and the test will be counted as disabled in the results.

Built-in Test Functions

There’s a built-in library of standard test functions, and you can have a closer look at their implementations in cmaj_test_functions.js.

The main functions include:

## testFunction

This wraps the block of code in a dummy namespace, and finds all the functions which take no parameters and return a bool. Each of these is called, and if any of them return false it is counted as a failure.

e.g.

## testFunction()

bool test1()    { return 1 + 1 == 2; }  // this will pass
bool test2()    { return 1 + 2 == 3; }  // this will fail
int notATest() {}  // this function will be ignored since it doesn't return a bool

## testProcessor()

This function will compile the block of code, and instantiate its main processor. The processor is expected to provide an output (either a stream or an event type) which emits int values.

The processor will then be run, and is expected to emit a stream of either 1 or 0 values, and then a -1 to indicate that the test is finished. If any 0 values are emitted, the test is registered as a fail, but if they’re all ones, it’s a pass.

e.g.

## testProcessor()

processor P
{
    output stream int out;

    void main()
    {
        out <- 1; advance();  // this is a pass
        out <- 0; advance();  // sending a zero will cause a fail to be logged

        out <- -1; advance(); // always send a -1 at the end to stop the test
    }
}

## testCompile()

This simply checks that the code compiles without any compiler errors.

## testConsole()

This instantiates a main processor from the code chunk that using the same system as testProcessor(). The output of the processor is ignored, but the content that was written to the console is checked against an expected value that is provided. e.g.

## testConsole ("hello world")

processor P
{
    output stream int out;
    void main()  { console <- "hello " <- "world"; out <- -1; advance(); }
}

## expectError ("<expected error message>")

This wraps the chunk of code in a dummy namespace (to allow you to easily write free functions without any boilerplate) and attempts to compile it. If the compiler error matches the one specified in the test directive, it’s a pass. If there’s no error, or the error doesn’t match, it’s a fail.

## expectError ("2:9: error: Cannot find symbol 'XX'")

void f (XX& x) {}

## runScript()

TODO

## testPatch()

This will attempt to load and compile a patch from a filename provided. Any compile errors will be registered as a failure. e.g.

## testPatch ("../../../my_patches/example_patch.cmajor_patch")

Auto-updating Expected Results

For tests like expectError or testConsole, the directive contains a string which is the expected outcome of the test. To easily update these, the tool will automatically insert the correct result and re-save the test file if you write the directive without any arguments.

For example, if you create a test file which looks like this:

## expectError

void f (XX& x) {}

and then run the test tool on it, it will rewrite the test file to look like this:

## expectError ("2:9: error: Cannot find symbol 'XX'")

void f (XX& x) {}

Creating Custom Test Functions

You can write your own javascript test functions at the start of the test file, and invoke them on a ## line.

If writing your own tests, it’s probably helpful to look at how the standard ones are implemented, in cmaj_test_functions.js.

The underlying javascript API used for managing the test status looks like this:

// A TestSection object provides functions that the current test can call. You get
// this object by calling the global function getCurrentTestSection()

class TestSection
{
    // logs a message to the console
    logMessage (message)

    // adds a failure to the test results, with a message
    reportFail (message)
    // adds a success to the test results, with a message
    reportSuccess (message)
    // adds a disabled test to the results, with a message
    reportDisabled (message)
    // notes that a test is unsupported on the current platform
    reportUnsupported (message)
    // logs an internal compiler error
    logCompilerError (error)

    // writes some stream data to a wav file
    writeStreamData (filename, streamData)
    // writes some event data to a JSON file
    writeEventData (filename, eventData)
    // reads some stream data from a wav file
    readStreamData (filename)
    // reads some event data from a JSON file
    readEventData (filename)

    // replaces the text of the current test invocation line with a new string
    updateTestHeader (newHeaderText)

    // converts a relative path from the test file to an absolute system path
    getAbsolutePath (relativePath)
}

function getCurrentTestSection()
function getDefaultEngineOptions()
function getEngineName()

The script also has access to the javascript bindings to create and render Cmajor processors.