docs(bats): add info on how to use bats

documentation/.vscode/extensions.json

@@ -0,0 +1,7 @@
+{
+    "recommendations": [
+        "ms-vscode.makefile-tools",
+        "lextudio.restructuredtext",
+        "trond-snekvik.simple-rst"
+    ]
+}

documentation/.vscode/settings.json

@@ -0,0 +1,12 @@
+{
+    "files.watcherExclude": {
+        "**/_build/**": true,
+    },
+    "python.formatting.provider": "black",
+    "editor.rulers": [
+        80,
+        100,
+        120
+    ],
+    "esbonio.sphinx.confDir": ""
+}

documentation/index.rst

@@ -40,6 +40,7 @@ same structures.
 
    Installation Manual <installation/index>
    Reference Manual <ref-manual/index>
+   Testing Manual <testing/index>
    Boot Concepts <boot/index>
    Best Practices <best_practices/index>
 

documentation/testing/bats.rst

@@ -0,0 +1,354 @@
+.. index:: BATS
+
+************************************
+BATS - Bash Automated Testing System
+************************************
+
+The CoreOS distribution supports writing tests using shell syntax by providing the `bats` command.
+
+If you want to use `bats`, you will need the following CoreOS packages:
+
+- bats
+- bats-file
+- bats-assert
+
+Overview of BATS
+================
+
+A BATS test can be as simple as a single .bats file. For example:
+
+.. code-block:: bash
+
+    #!/usr/bin/env bats
+   
+    bats_load_library bats-support
+    bats_load_library bats-assert
+
+    @test "can output to stdout" {
+        run echo hello
+        assert_output 'hello'
+    }
+
+You can run it using the command `bats <filename>.bats`
+
+This will give you the following output:
+
+.. code-block:: bash
+
+    sam@SAVE:~/Projects/tests$ bats <filename>.bats 
+    <filename>.bats
+    ✓ can output to stdout
+
+    1 test, 0 failures
+
+The run command
+================
+
+In shell tests, you often need to run commands and capture their output, exit
+status, and error messages. The run command provided by `bats` allows you to
+execute commands within your test cases and collect this information for later
+assertion and validation.
+
+The run command will make the following variables available:
+
+- `${status}`: exit code of the command run by `run`
+- `${output}`: combined content of `stdout` and `stderr`
+- `${lines[@]}`: array of lines of the output
+- `${BATS_RUN_COMMAND}`: command run by the `run` command
+
+.. code-block:: bash
+
+    @test "invoking foo with a nonexistent file prints an error" {
+        run foo nonexistent_filename
+        [ "$status" -eq 1 ]
+        [ "$output" = "foo: no such file 'nonexistent_filename'" ]
+        [ "$BATS_RUN_COMMAND" = "foo nonexistent_filename" ]
+
+    }
+
+The `run` command accepts some parameters:
+
+- `-N`: Expect N as exit status and fail otherwise
+- `-!`: Expect non-zero exit status and fail if the command succeeds.
+- `--keep-empty-lines`: don't remove empty lines from `${lines}`
+- `--separate-stderr`: Use separate variables for stderr `${stderr}` and `${stderr_lines[@]}`
+
+.. code-block:: bash
+
+    @test "invoking foo without arguments prints usage" {
+        run -1 foo
+        [ "${lines[0]}" = "usage: foo <filename>" ]
+    }
+
+The bats-assert helper
+======================
+
+The `bats-assert` helper provides some functions to create more readable tests.
+These assertions use the variables created by the `run` command and can be used
+as follows:
+
+.. code-block:: bash
+
+    @test 'assert_output()' {
+        run echo 'have'
+        assert_output 'want'
+    }
+
+The following functions are provided:
+
+- `assert` and `refute`: Assert that a given expression evaluates to true or false.
+- `assert_equal`: Assert that two parameters are equal.
+- `assert_not_equal`: Assert that two parameters are not equal.
+- `assert_success` and `assert_failure`: Assert that the exit status is 0 or 1.
+- `assert_output` and `refute_output`: Assert that the output does (or does not) contain the given content.
+- `assert_line` and `refute_line`: Assert that a specific line of the output does (or does not) contain the given content.
+- `assert_regex` and `refute_regex`: Assert that a parameter matches (or does not match) the given pattern.
+
+The bats-file helper
+====================
+
+The `bats-file` helper provides functions to help work with files in tests:
+
+**Test File Types:**
+
+- `assert_exists` and `assert_not_exists`: Check if a file or directory exists.
+- `assert_file_exists` and `assert_file_not_exists`: Check if a file exists.
+- `assert_dir_exists` and `assert_dir_not_exists`: Check if a directory exists.
+- `assert_link_exists` and `assert_link_not_exists`: Check if a link exists.
+- `assert_block_exists` and `assert_block_not_exists`: Check if a block special file exists.
+- `assert_character_exists` and `assert_character_not_exists`: Check if a character special file exists.
+- `assert_socket_exists` and `assert_socket_not_exists`: Check if a socket exists.
+- `assert_fifo_exists` and `assert_fifo_not_exists`: Check if a fifo special file exists.
+
+**Test File Attributes:**
+
+- `assert_file_executable` and `assert_file_not_executable`
+- `assert_file_owner` and `assert_file_not_owner`
+- `assert_file_permission` and `assert_not_file_permission`
+- `assert_file_size_equals`
+- `assert_size_zero` and `assert_size_not_zero`
+- `assert_file_group_id_set` and `assert_file_not_group_id_set`
+- `assert_file_user_id_set` and `assert_file_not_user_id_set`
+- `assert_sticky_bit` and `assert_no_sticky_bit`
+
+**Test File Content:**
+
+- `assert_file_empty` and `assert_file_not_empty`
+- `assert_file_contains` and `assert_file_not_contains`
+- `assert_symlink_to` and `assert_not_symlink_to`
+
+**Working with a temporary directory:**
+
+- `temp_make` and `temp_del`
+
+Pre- and Post-test case hooks
+==============================
+
+In some cases, it's useful to have a function that runs before or after each test
+case in a bats file.
+
+A function named `setup` will run before each test case, and a function
+named `teardown` will run after each test case.
+
+This example creates a directory in the setup function but lacks a teardown
+that removes the directory. The second time the setup function is run, the
+setup will fail as the directory already exists:
+
+.. code-block:: bash
+
+    #!/usr/bin/env bats
+
+    bats_load_library bats-support
+    bats_load_library bats-assert
+    bats_load_library bats-file
+
+    setup() {
+        mkdir tmp
+        echo 'a' >> ./tmp/test
+    }
+
+    @test "test contains a single a I" {
+        assert_file_contains ./tmp/test '^a$'
+    }
+
+    @test "test contains a single a II" {
+        assert_file_contains ./tmp/test '^a$'
+    }
+
+.. code-block:: bash
+
+    sam@SAVE:~/Projects/tests$ bats test.bats 
+    test.bats
+    ✓ test contains a single a I
+    ✗ test contains a single a II
+    (from function `setup' in test file test.bats, line 8)
+        `mkdir tmp' failed
+    mkdir: cannot create directory ‘tmp’: File exists
+
+    2 tests, 1 failure
+
+This can be easily fixed by adding a teardown function:
+
+.. code-block:: bash
+
+    #!/usr/bin/env bats
+
+    bats_load_library bats-support
+    bats_load_library bats-assert
+    bats_load_library bats-file
+
+    setup() {
+        mkdir tmp
+        echo 'a' >> ./tmp/test
+    }
+
+    teardown() {
+        rm -rf ./tmp
+    }
+
+
+
+    @test "test contains a single a I" {
+        assert_file_contains ./tmp/test '^a$'
+    }
+
+    @test "test contains a single a II" {
+        assert_file_contains ./tmp/test '^a$'
+    }
+
+.. code-block:: bash
+
+    sam@SAVE:~/Projects/tests$ bats test.bats 
+    test.bats
+     ✓ test contains a single a I
+     ✓ test contains a single a II
+
+    2 tests, 0 failures
+
+Pre- and Post-test file hooks
+=============================
+
+To run some code before executing a test file or after executing it, the
+functions `setup_file` and `teardown_file` can be used.
+
+The last example could be refactored to only create the tmp directory once:
+
+.. code-block:: bash
+
+    #!/usr/bin/env bats
+
+    bats_load_library bats-support
+    bats_load_library bats-assert
+    bats_load_library bats-file
+
+    setup_file() {
+        export DIR="./tmp"
+        export FILE="${DIR}/test"
+        mkdir "${DIR}"
+    }
+
+    teardown_file() {
+        rm -rf "${DIR}"
+    }
+
+    setup() {
+        echo 'a' >> "${FILE}"
+    }
+
+    teardown() {
+        rm "${FILE}"
+    }
+
+    @test "test contains a single a I" {
+        assert_file_contains "${FILE}" '^a$'
+    }
+
+    @test "test contains a single a II" {
+        assert_file_contains "${FILE}" '^a$'
+    }
+
+Multiple files
+==============
+
+With `bats`, a file is a test suite. If you have multiple `bats` files in a
+directory and you provide the directory in the `bats` command line, `bats`
+will execute all the test suites.
+
+Example: `bats .` 
+
+.. code-block:: bash
+
+    sam@SAVE:~/Projects/tests$ bats .
+    ./first.bats
+    ✓ can run our script
+    ✗ second test
+    (in test file ./first.bats, line 27)
+        `false' failed
+    ./second.bats
+    ✓ multi file
+    ./test.bats
+    ✓ test contains a single a I
+    ✓ test contains a single a II
+
+    5 tests, 1 failure
+
+Pre- and Post-suite hooks
+=========================
+
+If you want to execute the same function before each test suite or after
+each test suite, create a file named `setup_suite.bash`. In this file,
+create a function named `setup_suite()` and another named `teardown_suite()`.
+
+Exporting the test results
+==========================
+
+Test results can be exported using the JUnit XML format. This can then be
+used in other tools and merged with other JUnit XML formats to generate a final
+test report.
+
+Example:
+
+.. code-block:: bash
+
+    sam@SAVE:~/Projects/tests$ bats . -F junit
+
+This will produce the following XML content on stdout:
+
+.. code-block:: xml
+
+    <?xml version="1.0" encoding="UTF-8"?>
+    <testsuites time="0.048">
+    <testsuite name="./first.bats" tests="2" failures="1" errors="0" skipped="0" time="0.025" timestamp="2023-08-16T14:22:15" hostname="SAVE">
+        <testcase classname="./first.bats" name="can run our script" time="0.013" />
+        <testcase classname="./first.bats" name="second test" time="0.012">
+            <failure type="failure">(in test file ./first.bats, line 27)
+    `false&#39; failed</failure>
+        </testcase>
+
+    </testsuite>
+    <testsuite name="./second.bats" tests="1" failures="0" errors="0" skipped="0" time="0.008" timestamp="2023-08-16T14:22:15" hostname="SAVE">
+        <testcase classname="./second.bats" name="multi file" time="0.008" />
+
+    </testsuite>
+    <testsuite name="./test.bats" tests="2" failures="0" errors="0" skipped="0" time="0.015" timestamp="2023-08-16T14:22:15" hostname="SAVE">
+        <testcase classname="./test.bats" name="test contains a single a I" time="0.008" />
+        <testcase classname="./test.bats" name="test contains a single a II" time="0.007" />
+
+    </testsuite>
+    </testsuites>
+
+Going further
+=============
+
+`bats` scripts can be checked with shellcheck for common mistakes.
+
+The `bats-assert` add-on provides many helper functions to perform
+assertions with a more readable syntax than the shell's built-in syntax.
+
+See https://github.com/bats-core/bats-assert
+
+The `bats-file` add-on provides helper functions to check for files. See
+https://github.com/bats-core/bats-file/
+
+You can find a list of projects using `bats` on this page:
+https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats

documentation/testing/index.rst

@@ -0,0 +1,15 @@
+
+==============================
+Belden CoreOS Testing Manual
+==============================
+
+This manual is a work on progress on how to test and how to write test for
+CoreOS or CoreOS based distribution.
+
+|
+
+.. toctree::
+   :caption: Table of Contents
+   :numbered:
+
+   bats