Pkg.test("Gadfly") evaluates all of the files in
Gadfly/test/testscripts. Any errors or warnings are printed to the REPL. In addition, the figures that are produced are put into either the
master-output/ sub-directories. Nominally, the former represents the changes in a pull request while the latter are used for comparison. Specifically,
runtests.jl examines the currently checked out git commit, and sets the output directory to
master-output/ if it is the HEAD of the master branch or if it is detached. Otherwise, it assumes you are at the tip of a development branch and saves the figures to
devel-output/. After running the tests on both of these branches, executing
compare_examples.jl displays differences between the new and old figures. This script can dump a diff of the files to the REPL, open both figures for manual comparison, and/or, for SVG and PNG files, display a black and white figure highlighting the spatial location of the differences.
So the automated regression analysis workflow is then as follows:
- In a branch other than master,
- develop your new feature or fix your old bug,
- commit all your changes,
- checkout master,
Pkg.add("ArgParse")and, for B&W images, Cairo, Fontconfig, Rsvg, and Images as well,
- check for differences with
julia test/compare_examples.jl [--diff] [--two] [--bw] [-h] [filter]. For example,
julia test/compare_examples.jl --bw .js.svgwill show black and white images highlighting the differences between the svg test images.