Skip to content

Checking the source code

Baygon tests programs from the outside, but a test can run any program, not only the one under test. Checks on the source code of an assignment are therefore tests that run cat, grep, wc, a compiler or a formatter on the sources, and check their output or their exit status.

A bare name such as grep is looked up in the PATH (see Executable), and each test can use its own executable. The paths given in args are relative to the directory Baygon is started from.

The checks below run on a small C project made of main.c, sum.c and sum.h.

Program header

The file must start with a comment giving the author:

- name: Program header with the author
  executable: cat
  args: [main.c]
  stdout:
    - regex: '^/\*[\s\S]*?Author:\s*\S+[\s\S]*?\*/'

Include guard

The header must be protected against multiple inclusion, with the same name in #ifndef and #define:

- name: Include guard in sum.h
  executable: cat
  args: [sum.h]
  stdout:
    - regex: '#ifndef\s+(\w+)\s*\n#define\s+\1\b'

Forbidden functions

grep exits with 1 when it finds nothing, and prints the offending lines otherwise, which shows up in the failure message:

- name: No forbidden function
  executable: grep
  args: [-nE, '\b(gets|system)\s*\(', main.c, sum.c]
  exit: 1 # grep exits with 1 when it finds nothing

Code left in comments

A line comment that ends with a semicolon is most likely code that was commented out:

- name: No commented-out code
  executable: grep
  args: [-nE, '^\s*//.*;\s*$', main.c, sum.c]
  exit: 1

Length of a file

wc -l prints the number of lines followed by the file name. A regex filter keeps the number, which the number matcher compares:

- name: At most 100 lines in main.c
  executable: wc
  args: [-l, main.c]
  stdout:
    - filters:
        regex: ['^\s*(\d+).*$', '\1'] # Keep the number only
      number: { max: 100 }

Compiler warnings

- name: Compiles without warnings
  executable: gcc
  args: [-std=c11, -Wall, -Wextra, -Werror, -fsyntax-only, main.c, sum.c]
  exit: 0

A static analyser works the same way, for example cppcheck --error-exitcode=1 --enable=warning main.c sum.c with exit: 0.

Indentation and formatting

clang-format checks the files against the style given in a .clang-format file, and fails on the first difference with --Werror:

- name: Formatting follows .clang-format
  executable: clang-format
  args: [--dry-run, --Werror, main.c, sum.c, sum.h]
  exit: 0

pip install clang-format installs it when it is not available on the system.

Putting it together

The checks are tests like any other. Put them in a group to give them their own share of the points, and run them alone with --only:

version: 1
executable: ./a.out
tests:
  - name: Behaviour
    points: 15
    tests:
      - args: [1, 2]
        stdout: "3\n"
  - name: Code quality
    points: 5
    tests:
      - name: Compiles without warnings
        executable: gcc
        args: [-std=c11, -Wall, -Wextra, -Werror, -fsyntax-only, main.c, sum.c]
        exit: 0
      - name: No forbidden function
        executable: grep
        args: [-nE, '\b(gets|system)\s*\(', main.c, sum.c]
        exit: 1
baygon --only 2 --report quality.json