Config File Syntax¶
Base¶
Each config file should start with the version of the syntax which is 1. Then the mandatory tests dictionary must exist. The following is a minimal example composed of 1 test which tests if the output is 3 when the binary is run with two arguments 1 and 2:
version: 1
tests:
- args: [1, 2]
stdout: 3
Filters¶
Global filters can be applied to all outputs of every tests. The outputs are stdout and stderr. Baygon features the following filters:
uppercase: All outputs are transformed into uppercase.lowercase: All outputs are transformed into lowercase.trim: Whitespace (spaces, tabs, newlines) is removed from both ends of the output.ignorespaces: All spaces are removed from all outputs (ignore-spacesis accepted as an alias).regex: A remplacement is made using regular expressions.replace: A remplacement is made using a string.
If more than one filter is set, they are applied in this order, whatever the order in the file: uppercase, lowercase, trim, ignorespaces, regex, replace.
A boolean filter set to false (e.g. uppercase: false) is not applied.
In the following, both standard output and standard error will be in lowercase and every occurrences of foo or foobar will be replaced by bar:
version: 1
filters:
lowercase: true
regex: ['foo(bar)?', bar]
tests:
- args: [1, 2]
stdout: 3
Filters can also be applied to a single test:
version: 1
tests:
- args: [1, 2]
stdout:
- contains: foo
- regex: f(oo|aa|uu)
filters:
lowercase: true
Or even at a stream output level:
stdout: [{ filters: { trim: true }, equals: 3 }]
Naming¶
All tests can be optionally named:
version: 1
tests:
- name: Test functionality of the additionner
args: [--add, 40, 2]
stdout: 42
- name: Test error if less than two arguments
args: []
exit: 2
Groups and subgroups¶
Tests can be grouped into sub sections, by nesting each test into categories:
version: 1
tests:
- name: Category 1
tests:
- args: [1, 2]
stdout: 3
- name: Category 2
tests:
- name: Subcategory 1
tests:
- args: [1, 2]
stdout: 3
Exit status¶
The exit status can be checked with the exit key followed with an integer. The following checks if the program returns 0
version: 1
tests:
- exit: 0
Standard input¶
stdin is the text given to the program on its standard input:
tests:
- stdin: "3\n4\n"
stdout: "7\n"
Input generated by a program¶
stdin can also be the output of another program, the generator. Its
arguments accept expressions, so each run of a repeated test
can get a different input:
version: 1
eval: true
executable: ./sort
tests:
- name: Sort random lists
repeat: 20
stdin:
executable: ./generate # Relative to the configuration file
args: ["{{ iter() }}"] # Seed: 0, 1, 2... one per run
stdout:
- same-as: ./reference # See "Reference program" below
The generator follows the same time and output limits
as the test. If it cannot start, exits with a status other than 0 or reaches a
limit, the test fails with Generator ./generate failed: ....
Standard outputs¶
Both stdout and stderr can be tested against multiple conditions, the
matchers:
tests:
- stdout:
- contains: foo # Must contain the word foo
- regex: f(oo|aa|uu) # Must match
stderr:
- equals: foobar # Must be exactly equal to foobar
Executable¶
In the case you want to specify a different executable name for a different test:
tests:
- name: Test ./foo
executable: ./foo
stdout: I am Foo
- name: Test ./bar
executable: ./bar
stderr: I am bar
- name: Group
executable: ./baz
tests:
- name: Test 1
exit: 0
A relative path is relative to the directory of the configuration file. A
bare name such as grep or gcc that is not a file in that directory is looked
up in the PATH, which is handy to check the source code.
Destructive programs (rm, mv, dd, mkfs, wget) are refused.
The executable is propagated through the test tree, and the nearest one wins: a group or a test can set its own executable even when a parent already has one.
version: 1
executable: ./app
tests:
- name: Run
stdin: "0\n"
- name: No compiler warning
executable: /bin/cat # Replaces ./app for this test only
args: [obj/gcc_output.cmd]
stdout:
- not:
- contains: 'warning:'
Note
Up to Baygon 0.7, overriding an inherited executable was an error
(Executable can't be overridden). An executable given on the command line
still cannot be combined with one at the top level of the configuration.
One approach is name the executable at the top level:
version: 1
executable: ./foobar
tests:
- name: Test ./foobar
exit: 0
Or even from the shell:
baygon ./foobar
Note that a relative executable in the config file is relative to the directory of the config file, while an executable given from the shell is relative to the current directory. In both cases the program runs in the current directory, so relative paths in args are relative to the directory Baygon is started from.
Time and output limits¶
A program that never ends, or that prints without end (a menu shown again and
again at the end of input is a classic), must not block the whole run. Two keys,
accepted at the top level, on a group or on a test, limit each run of a program.
Like executable, the nearest value wins.
timeout: wall-clock limit in seconds (an integer or a decimal number).max-output: number of bytes kept from each output stream (stdoutandstderr). A number of bytes, or a size such as64K,10Mor1G.0removes the limit.
version: 1
executable: ./app
timeout: 5 # Default of every test
tests:
- name: Exits with code 1 on EOF
timeout: 1 # This test only
stdin: ""
exit: 1
- name: Prints the menu
max-output: 64K
stdout:
- contains: Menu
When a limit is reached, the program and every process it started are killed,
the test fails with Timeout after 1 s. or Output limit exceeded, and the
rest of the suite goes on. A test with repeat stops at the first run that
reaches a limit.
There is no time limit by default. The --timeout option sets one for every
test whose configuration has none:
baygon --timeout 10
The output of each stream is capped at 16 MiB by default, so Baygon never holds an unbounded amount of output in memory.
Configuration file¶
By default Baygon will look for a file named baygon.yml in the current directory. You can specify a different file with the -c or --config option:
baygon --config other.yaml
Note
Baygon 0.5.x historically used the -t flag for selecting the configuration file. The CLI still accepts -t for backward compatibility, but newer releases prefer -c/--config.
Other names such as t, test or tests can be used, but the extension must be .json, .yml or .yaml. Here some valid configuration names:
baygon.yml
baygon.yaml
baygon.json
tests.yml
t.json
...
Matchers¶
A condition on stdout or stderr holds one or more matchers. Each matcher
applies to the output once the filters are applied.
| Matcher | Passes when the output... |
|---|---|
equals |
is exactly the given text |
contains |
contains the given text |
regex |
matches the Python regular expression |
number |
is a number equal to a value, or within a range |
length |
has the given number of characters |
lines |
has the given number of lines |
same-as |
is the same as the output of a reference program |
A plain value is a shortcut for equals: stdout: 3 is stdout: [equals: "3"].
An empty text checks that the output is empty: stderr: "".
tests:
- stdout:
- contains: foo
- regex: f(oo|aa|uu)
- equals: foobar
Numbers¶
number reads the whole output, spaces and newlines around it ignored, as a
number. Give a value, with an optional absolute tolerance, or a range with
min and/or max (both included):
tests:
- args: [2]
stdout:
- number: 6.28 # Equal (to within 1e-9)
- number: { value: 6.2832, tolerance: 0.001 }
- number: { min: 0, max: 10 }
- number: { value: "{{ 2 * pi }}", tolerance: 1e-6 } # An expression
When the number is surrounded by text, keep the number only with a regex
filter on the condition:
tests:
- stdout: # The program prints "Result: 6.2832"
- filters:
regex: ['^Result: ', '']
number: { value: 6.2832, tolerance: 0.001 }
The test fails with Output stdout is not a number: ... when the output is
not a number.
Length and lines¶
length counts the characters of the output and lines its lines. Give an
exact count or a range:
tests:
- stdout:
- lines: 3
- length: { max: 80 }
stderr:
- lines: 0 # Nothing on stderr
Reference program¶
same-as runs a reference program, for example the solution of the exercise,
with the same arguments, standard input and environment as the program under
test, and compares the two outputs on the same stream. The filters of the
condition are applied to both outputs:
version: 1
eval: true
executable: ./student
tests:
- repeat: 50
args: ["{{ randint(1, 1000) }}"]
stdout:
- filters: { trim: true }
same-as: ./solution
The reference runs once per run of the test, even when stdout and stderr
both use it. Its exit status does not matter, but if it cannot start or reaches a
limit, the test fails with
Reference ./solution failed: .... With -vvv or --pretty, its run is shown
as Command #n (reference).
Negation¶
not takes a list of matchers that must all fail. Each item holds one matcher:
tests:
- stdout:
- not:
- contains: error # Must not contain "error"
- regex: ^\s*$ # Must not be blank
- number: 0 # Must not be 0
same-as cannot be negated.
Expressions¶
When eval is set at the top level, the text between {{ and }} is
evaluated as Python in args, stdin, env, exit, the generator args and
the expected values of the matchers.
version: 1
eval: true
tests:
- name: Random addition
repeat: 10
args: ["{{ a = randint(0, 100) }}", "{{ b = randint(0, 100) }}"]
stdout:
- number: "{{ a + b }}"
- An assignment such as
{{ a = randint(0, 100) }}is replaced by its value, andacan be used in the next expressions. The arguments are evaluated first, then the standard input, the environment, the exit status and finally the matchers, once the program has run. - The functions of the
math,randomandstatisticsmodules are available. iter(start=0, step=1)returnsstart, thenstart + step... a new value each time the same expression is evaluated, e.g. one per run of a repeated test.- The delimiters and the code run beforehand can be changed:
eval:
start: "<<"
end: ">>"
init: "from datetime import date" # A string or a list of strings
eval: false, or no eval key, leaves the text as it is.