Skip to content

Config File Syntax

Base

Each config file should start with the version of the syntax which is 1. Then the mandatory tests dictionary must exist. The following is a minimal example composed of 1 test which tests if the output is 3 when the binary is run with two arguments 1 and 2:

version: 1
tests:
  - args: [1, 2]
    stdout: 3

Filters

Global filters can be applied to all outputs of every tests. The outputs are stdout and stderr. Baygon features the following filters:

  • uppercase: All outputs are transformed into uppercase.
  • lowercase: All outputs are transformed into lowercase.
  • trim: Whitespace (spaces, tabs, newlines) is removed from both ends of the output.
  • ignorespaces: All spaces are removed from all outputs (ignore-spaces is accepted as an alias).
  • regex: A remplacement is made using regular expressions.
  • replace: A remplacement is made using a string.

If more than one filter is set, they are applied in this order, whatever the order in the file: uppercase, lowercase, trim, ignorespaces, regex, replace.

A boolean filter set to false (e.g. uppercase: false) is not applied.

In the following, both standard output and standard error will be in lowercase and every occurrences of foo or foobar will be replaced by bar:

version: 1
filters:
  lowercase: true
  regex: ['foo(bar)?', bar]
tests:
  - args: [1, 2]
    stdout: 3

Filters can also be applied to a single test:

version: 1
tests:
  - args: [1, 2]
    stdout:
      - contains: foo
      - regex: f(oo|aa|uu)
    filters:
      lowercase: true

Or even at a stream output level:

stdout: [{ filters: { trim: true }, equals: 3 }]

Naming

All tests can be optionally named:

version: 1
tests:
  - name: Test functionality of the additionner
    args: [--add, 40, 2]
    stdout: 42
  - name: Test error if less than two arguments
    args: []
    exit: 2

Groups and subgroups

Tests can be grouped into sub sections, by nesting each test into categories:

version: 1
tests:
  - name: Category 1
    tests:
      - args: [1, 2]
        stdout: 3
  - name: Category 2
    tests:
      - name: Subcategory 1
        tests:
          - args: [1, 2]
            stdout: 3

Exit status

The exit status can be checked with the exit key followed with an integer. The following checks if the program returns 0

version: 1
tests:
  - exit: 0

Standard input

stdin is the text given to the program on its standard input:

tests:
  - stdin: "3\n4\n"
    stdout: "7\n"

Input generated by a program

stdin can also be the output of another program, the generator. Its arguments accept expressions, so each run of a repeated test can get a different input:

version: 1
eval: true
executable: ./sort
tests:
  - name: Sort random lists
    repeat: 20
    stdin:
      executable: ./generate # Relative to the configuration file
      args: ["{{ iter() }}"] # Seed: 0, 1, 2... one per run
    stdout:
      - same-as: ./reference # See "Reference program" below

The generator follows the same time and output limits as the test. If it cannot start, exits with a status other than 0 or reaches a limit, the test fails with Generator ./generate failed: ....

Standard outputs

Both stdout and stderr can be tested against multiple conditions, the matchers:

tests:
  - stdout:
      - contains: foo # Must contain the word foo
      - regex: f(oo|aa|uu) # Must match
    stderr:
      - equals: foobar # Must be exactly equal to foobar

Executable

In the case you want to specify a different executable name for a different test:

tests:
  - name: Test ./foo
    executable: ./foo
    stdout: I am Foo
  - name: Test ./bar
    executable: ./bar
    stderr: I am bar
  - name: Group
    executable: ./baz
    tests:
      - name: Test 1
        exit: 0

A relative path is relative to the directory of the configuration file. A bare name such as grep or gcc that is not a file in that directory is looked up in the PATH, which is handy to check the source code. Destructive programs (rm, mv, dd, mkfs, wget) are refused.

The executable is propagated through the test tree, and the nearest one wins: a group or a test can set its own executable even when a parent already has one.

version: 1
executable: ./app
tests:
  - name: Run
    stdin: "0\n"
  - name: No compiler warning
    executable: /bin/cat # Replaces ./app for this test only
    args: [obj/gcc_output.cmd]
    stdout:
      - not:
          - contains: 'warning:'

Note

Up to Baygon 0.7, overriding an inherited executable was an error (Executable can't be overridden). An executable given on the command line still cannot be combined with one at the top level of the configuration.

One approach is name the executable at the top level:

version: 1
executable: ./foobar
tests:
  - name: Test ./foobar
    exit: 0

Or even from the shell:

baygon ./foobar

Note that a relative executable in the config file is relative to the directory of the config file, while an executable given from the shell is relative to the current directory. In both cases the program runs in the current directory, so relative paths in args are relative to the directory Baygon is started from.

Time and output limits

A program that never ends, or that prints without end (a menu shown again and again at the end of input is a classic), must not block the whole run. Two keys, accepted at the top level, on a group or on a test, limit each run of a program. Like executable, the nearest value wins.

  • timeout: wall-clock limit in seconds (an integer or a decimal number).
  • max-output: number of bytes kept from each output stream (stdout and stderr). A number of bytes, or a size such as 64K, 10M or 1G. 0 removes the limit.
version: 1
executable: ./app
timeout: 5 # Default of every test
tests:
  - name: Exits with code 1 on EOF
    timeout: 1 # This test only
    stdin: ""
    exit: 1
  - name: Prints the menu
    max-output: 64K
    stdout:
      - contains: Menu

When a limit is reached, the program and every process it started are killed, the test fails with Timeout after 1 s. or Output limit exceeded, and the rest of the suite goes on. A test with repeat stops at the first run that reaches a limit.

There is no time limit by default. The --timeout option sets one for every test whose configuration has none:

baygon --timeout 10

The output of each stream is capped at 16 MiB by default, so Baygon never holds an unbounded amount of output in memory.

Configuration file

By default Baygon will look for a file named baygon.yml in the current directory. You can specify a different file with the -c or --config option:

baygon --config other.yaml

Note

Baygon 0.5.x historically used the -t flag for selecting the configuration file. The CLI still accepts -t for backward compatibility, but newer releases prefer -c/--config.

Other names such as t, test or tests can be used, but the extension must be .json, .yml or .yaml. Here some valid configuration names:

baygon.yml
baygon.yaml
baygon.json
tests.yml
t.json
...

Matchers

A condition on stdout or stderr holds one or more matchers. Each matcher applies to the output once the filters are applied.

Matcher Passes when the output...
equals is exactly the given text
contains contains the given text
regex matches the Python regular expression
number is a number equal to a value, or within a range
length has the given number of characters
lines has the given number of lines
same-as is the same as the output of a reference program

A plain value is a shortcut for equals: stdout: 3 is stdout: [equals: "3"]. An empty text checks that the output is empty: stderr: "".

tests:
  - stdout:
      - contains: foo
      - regex: f(oo|aa|uu)
      - equals: foobar

Numbers

number reads the whole output, spaces and newlines around it ignored, as a number. Give a value, with an optional absolute tolerance, or a range with min and/or max (both included):

tests:
  - args: [2]
    stdout:
      - number: 6.28 # Equal (to within 1e-9)
      - number: { value: 6.2832, tolerance: 0.001 }
      - number: { min: 0, max: 10 }
      - number: { value: "{{ 2 * pi }}", tolerance: 1e-6 } # An expression

When the number is surrounded by text, keep the number only with a regex filter on the condition:

tests:
  - stdout: # The program prints "Result: 6.2832"
      - filters:
          regex: ['^Result: ', '']
        number: { value: 6.2832, tolerance: 0.001 }

The test fails with Output stdout is not a number: ... when the output is not a number.

Length and lines

length counts the characters of the output and lines its lines. Give an exact count or a range:

tests:
  - stdout:
      - lines: 3
      - length: { max: 80 }
    stderr:
      - lines: 0 # Nothing on stderr

Reference program

same-as runs a reference program, for example the solution of the exercise, with the same arguments, standard input and environment as the program under test, and compares the two outputs on the same stream. The filters of the condition are applied to both outputs:

version: 1
eval: true
executable: ./student
tests:
  - repeat: 50
    args: ["{{ randint(1, 1000) }}"]
    stdout:
      - filters: { trim: true }
        same-as: ./solution

The reference runs once per run of the test, even when stdout and stderr both use it. Its exit status does not matter, but if it cannot start or reaches a limit, the test fails with Reference ./solution failed: .... With -vvv or --pretty, its run is shown as Command #n (reference).

Negation

not takes a list of matchers that must all fail. Each item holds one matcher:

tests:
  - stdout:
      - not:
          - contains: error # Must not contain "error"
          - regex: ^\s*$ # Must not be blank
          - number: 0 # Must not be 0

same-as cannot be negated.

Expressions

When eval is set at the top level, the text between {{ and }} is evaluated as Python in args, stdin, env, exit, the generator args and the expected values of the matchers.

version: 1
eval: true
tests:
  - name: Random addition
    repeat: 10
    args: ["{{ a = randint(0, 100) }}", "{{ b = randint(0, 100) }}"]
    stdout:
      - number: "{{ a + b }}"
  • An assignment such as {{ a = randint(0, 100) }} is replaced by its value, and a can be used in the next expressions. The arguments are evaluated first, then the standard input, the environment, the exit status and finally the matchers, once the program has run.
  • The functions of the math, random and statistics modules are available.
  • iter(start=0, step=1) returns start, then start + step... a new value each time the same expression is evaluated, e.g. one per run of a repeated test.
  • The delimiters and the code run beforehand can be changed:
eval:
  start: "<<"
  end: ">>"
  init: "from datetime import date" # A string or a list of strings

eval: false, or no eval key, leaves the text as it is.