Regression Testing

Running Pkg.test("Gadfly") evaluates all of the files in Gadfly/test/testscripts. Any errors or warnings are printed to the REPL. In addition, the figures that are produced are put into either the devel-output/ or master-output/ sub-directories. Nominally, the former represents the changes in a pull request while the latter are used for comparison. Specifically, runtests.jl examines the currently checked out git commit, and sets the output directory to master-output/ if it is the HEAD of the master branch or if it is detached. Otherwise, it assumes you are at the tip of a development branch and saves the figures to devel-output/. After running the tests on both of these branches, executing compare_examples.jl displays differences between the new and old figures. This script can dump a diff of the files to the REPL, open both figures for manual comparison, and/or, for SVG and PNG files, display a black and white figure highlighting the spatial location of the differences.

So the automated regression analysis workflow is then as follows:

In a branch other than master,
develop your new feature or fix your old bug,
commit all your changes,
Pkg.test("Gadfly"),
checkout master,
Pkg.test again,
Pkg.add("ArgParse") and, for B&W images, Cairo, Fontconfig, Rsvg, and Images as well,
check for differences with julia test/compare_examples.jl [--diff] [--two] [--bw] [-h] [filter]. For example, julia test/compare_examples.jl --bw .js.svg will show black and white images highlighting the differences between the svg test images.