Testing¶
Test-driven development improves the ability to upgrade your code base to keep up with API changes, to refactor without incurring regression errors, or to validate when bugs in dependent libraries have been fixed.
Testing in lim
is driven by tox
from the Makefile
in the
top level repo directory, using the Bats-core: Bash Automated Testing System (2018)
testing framework. See Testing Your Shell Scripts, with Bats for
information on using BATS.
Note
Note that bats-core
is a more recent fork of the original
Bats: Bash Automated Testing System system, which is no longer
being actively maintained. bats-core
has more features than
the older bats
program that may come pre-installed with your
OS or package management system. The tests here rely on things
like the setup_file()
function to ensure data files are
present for runtime tests.
The Makefile
in this repo ensures that the newer bats-core
program is installed into /usr/local/bin
and that related
related libraries that are used by tests are installed locally
in the tests/lib
directory.
Run make help
to see the targets associated with testing. The
primary targets are test
for unit testing code, and
test-bats-runtime
for runtime integration testing.
The tox
tests are performed using the following configuration file:
[tox]
# In practice, you can optimize by first running basic tests
# 'pep8,bandit,docs' and only after those succeed go on to
# run the remaining (default) tests. E.g.,
# $ tox -e pep8,bandit,docs && tox -e py36,py37,py38,bats,pypi
# envlist = pep8,bandit,docs,py36,py37,py38,bats,pypi
envlist = py36,py37,py38,bats,pypi
skip_missing_interpreters = true
requires = tox-conda
setuptools>=42
setuptools_scm
[testenv]
setenv =
VIRTUAL_ENV={envdir}
BRANCH_NAME=master
PYTHONPATH={toxinidir}:{toxinidir}/lim
distribute = False
install_command = python -m pip install {opts} {packages}
conda_deps =
pytest
conda_channels =
conda-forge
# Make sure these match setup.py!
deps = -Ur{toxinidir}/requirements.txt
pbr>=5.4.5
pip>=20.2.2
pytest
commands = pytest {posargs}
; If you want to make tox run the tests with the same versions, create a
; requirements.txt with the pinned versions and uncomment the following lines:
; deps =
; -r{toxinidir}/requirements.txt
[testenv:pypi]
basepython = python3.8
whitelist_externals = make
deps = -Ur{toxinidir}/requirements.txt
twine
commands = make twine-check
[testenv:pep8]
basepython = python3.8
deps = -Ur{toxinidir}/requirements.txt
flake8>=3.8.3
commands = flake8 lim setup.py
[testenv:bandit]
basepython = python3.8
; Run security linter
deps = -Ur{toxinidir}/requirements.txt
bandit>=1.1.0
commands = bandit -c bandit.yaml -r lim -x tests -n5
[testenv:docs]
basepython = python3.8
deps = -Ur{toxinidir}/requirements.txt
commands = sphinx-build -b html docs docs/_build
[testenv:bats]
; Run bats unit tests
; Deal with this by requiring docutils==0.15:
; # Traceback (most recent call last):
; # File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/cliff/help.py", line 43, in __call__
; # factory = ep.load()
; # File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2444, in load
; # self.require(*args, **kwargs)
; # File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2467, in require
; # items = working_set.resolve(reqs, env, installer, extras=self.extras)
; # File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
; # raise VersionConflict(dist, req).with_context(dependent_req)
; # pkg_resources.ContextualVersionConflict: (docutils 0.16 (/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages), Requirement.parse('docutils<0.16,>=0.10'), {'botocore'})
deps = -Ur{toxinidir}/requirements.txt
docutils==0.15
whitelist_externals = make
commands = make test-bats
There are built-in unit tests and security tests for the Python lim
application, as well as runtime bats
tests that perform realtime
integration or system tests.
Note
The examples of output from bats
and tox
tests are truncated to 80
characters and do not represent the exact output you would see if you ran
these commands at the command line.
Data files¶
Some of the tests require PCAP files. One of the test files
(smallFlows_nopayloads.pcap
) comes from the Packet Café source repository.
If a clone of that directory exists at ~/git/packet_cafe
, the file will be
copied from the local clone. Otherwise, it will be downloaded from GitHub.
Other test files come from the CTU web server and are downloaded as needed
(using lim
, of course.)
Unit tests¶
Invoke unit tests with the helper Makefile
using the test
target (e.g., make test
). This target runs tox
tests for the Python code
(make test-tox
), followed by some basic unit tests performed with
bats
(make test-bats
).
If successful, you will see the following:
$ make test-tox
tox
GLOB sdist-make: /Users/dittrich/git/LiminalInfo/lim-cli/setup.py
py36 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py36 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py36 run-test-pre: PYTHONHASHSEED='1917477129'
py37 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py37 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py37 run-test-pre: PYTHONHASHSEED='1917477129'
py38 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py38 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py38 run-test-pre: PYTHONHASHSEED='1917477129'
pep8 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
pep8 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
pep8 run-test-pre: PYTHONHASHSEED='1917477129'
pep8 run-test: commands[0] | flake8 lim setup.py --exclude text.py
bandit inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/
bandit installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4
bandit run-test-pre: PYTHONHASHSEED='1917477129'
bandit run-test: commands[0] | bandit -c bandit.yaml -r lim -x tests -n5
[main] INFO profile include tests: None
[main] INFO profile exclude tests: B110,B101
[main] INFO cli include tests: None
[main] INFO cli exclude tests: None
[main] INFO using config: bandit.yaml
[main] INFO running on Python 3.8.5
Run started:2020-08-12 19:21:31.448772
Test results:
No issues identified.
Code scanned:
Total lines of code: 3872
Total lines skipped (#nosec): 9
Run metrics:
Total issues (by severity):
Undefined: 0.0
Low: 0.0
Medium: 0.0
High: 0.0
Total issues (by confidence):
Undefined: 0.0
Low: 0.0
Medium: 0.0
High: 0.0
Files skipped (0):
bats inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
bats installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
bats run-test-pre: PYTHONHASHSEED='1917477129'
bats run-test: commands[0] | make test-bats
[+] Running bats tests: 00_help.bats
1..6
ok 1 "lim about" contains "version"
ok 2 'lim help' can load all entry points
ok 3 "lim cafe --help" properly lists subcommands
ok 4 "lim ctu --help" properly lists subcommands
ok 5 "lim pcap --help" properly lists subcommands
ok 6 'lim --version' works
docs inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
docs installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
docs run-test-pre: PYTHONHASHSEED='1917477129'
docs run-test: commands[0] | sphinx-build -b html docs docs/_build
Running Sphinx v3.2.0
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 2 changed, 0 removed
reading sources... [ 50%] changes
reading sources... [100%] testing
/Users/dittrich/git/LiminalInfo/lim-cli/.tox/docs/lib/python3.8/site-packages/se
warnings.warn(
/Users/dittrich/git/LiminalInfo/lim-cli/docs/testing.rst:65: WARNING: Include fi
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 33%] changes
writing output... [ 66%] index
writing output... [100%] testing
generating indices... genindexdone
writing additional pages... searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.
The HTML pages are in docs/_build.
pypi inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
pypi installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
pypi run-test-pre: PYTHONHASHSEED='1917477129'
pypi run-test: commands[0] | python setup.py check --restructuredtext
running check
___________________________________ summary ____________________________________
py36: commands succeeded
py37: commands succeeded
py38: commands succeeded
pep8: commands succeeded
bandit: commands succeeded
bats: commands succeeded
docs: commands succeeded
pypi: commands succeeded
congratulations :)
Immediately following this are the bats
unit tests. Successful results
will look like this:
[+] Running bats tests: 00_help.bats
1..6
ok 1 "lim about" contains "version"
ok 2 'lim help' can load all entry points
ok 3 "lim cafe --help" properly lists subcommands
ok 4 "lim ctu --help" properly lists subcommands
ok 5 "lim pcap --help" properly lists subcommands
ok 6 'lim --version' works
Integration tests¶
The integration and system tests using bats
require a live network
connection and/or a running packet-cafe server. Because of this,
these tests are only run on demand and not as part of basic unit
testing and code analysis.
The bats
tests can then be run using the helper Makefile
target
test-bats-runtime
.
$ make test-bats-runtime
[+] Running bats runtime tests: runtime_10_ctu.bats runtime_20_pcap.bats runtime
1..35
ok 1 "lim -q ctu list CTU-Malware-Capture-Botnet-48" works
ok 2 "lim -q ctu list Botnet-48" works
ok 3 "lim -q ctu get Botnet-48 pcap --no-subdir " gets PCAP file to cwd
ok 4 "lim -q ctu get Botnet-48 pcap" gets PCAP file
ok 5 "lim pcap extract ips CTU-Malware-Capture-Botnet-48/botnet-capture-20110816
ok 6 "lim pcap extract ips --stdout CTU-Malware-Capture-Botnet-48/botnet-capture
ok 7 "lim pcap shift time CTU-Malware-Capture-Botnet-48/botnet-capture-20110816-
ok 8 VOL_PREFIX is exported
ok 9 packet-cafe Docker containers are running (via "docker ps")
ok 10 "lim -q cafe containers" reports Docker containers are running
ok 11 "lim cafe endpoints" includes "/api/v1/info"
ok 12 "lim cafe admin endpoints" includes "/v1/info"
ok 13 "lim cafe tools" includes "iqtlabs/"
ok 14 "lim cafe info" includes "hostname"
ok 15 "lim cafe admin info" includes "hostname"
ok 16 "lim cafe admin sessions" has no sessions
ok 17 "lim -q cafe admin sessions" returns failure
ok 18 "lim cafe upload --wait ~/git/packet_cafe/notebooks/smallFlows_nopayloads.
ok 19 "lim cafe status" contains "Complete"
ok 20 "lim cafe admin results" contains "metadata.json" files
ok 21 "lim cafe admin files" contains "smallFlows_nopayloads.pcap"
ok 22 "lim cafe upload --wait CTU-Malware-Capture-Botnet-48/botnet-capture-20110
ok 23 "lim -q cafe report --tool poof" fails
ok 24 "lim -q cafe report --tool p0f" works
ok 25 "lim cafe results --tool networkml" works
ok 26 "lim cafe admin sessions -f value" shows both sessions
ok 27 "lim cafe admin delete 22222222-2222-2222-2222-222222222222" removes sessi
ok 28 "lim cafe admin delete --all" leaves storage directory empty
ok 29 "lim cafe raw --tool p0f" fails
ok 30 "lim cafe raw --tool p0f --choose" fails (no tty)
ok 31 "lim cafe results --tool p0f" fails
ok 32 "lim cafe results --choose" fails (no tty)
ok 33 "lim cafe about" fails (no tty)
ok 34 "lim cafe ui" fails (no tty)
ok 35 Cleaning up /tmp/packet_cafe_status
Writing and debugging BATS tests¶
Files that are used by tests are loaded into the directory pointed
to by the runtime environment variable BATS_RUN_TMPDIR
. This is
to avoid cluttering up the repo directory with temporary files.
This directory is removed on exit (see source code).
To see the values of these variables, place the following command
in the .bats
file and run it with the -t
flag:
env | grep BATS_ >&3
The output will expose the runtime values:
$ bats -t tests/runtime_30_packet_cafe.bats
1..39
BATS_ROOT_PID=55272
BATS_TEST_SOURCE=/Users/dittrich/tmp/bats-run-55272/bats.55345.src
BATS_TMPDIR=/Users/dittrich/tmp
BATS_RUN_TMPDIR=/Users/dittrich/tmp/bats-run-55272
BATS_VERSION=1.2.1
BATS_TEST_PATTERN=^[[:blank:]]*@test[[:blank:]]+(.*[^[:blank:]])[[:blank:]]+\{(.*)$
BATS_CWD=/Users/dittrich/git/LiminalInfo/lim-cli
BATS_TEMPDIR_CLEANUP=1
BATS_TEST_FILTER=
BATS_TEST_PATTERN_COMMENT=[[:blank:]]*([^[:blank:]()]+)[[:blank:]]*\(?\)?[[:blank:]]+\{[[:blank:]]+#[[:blank:]]*@test[[:blank:]]*$
BATS_LIBEXEC=/usr/local/libexec/bats-core
BATS_ROOT=/usr/local
not ok 1 setup_file failed
# (from function `setup_file' in test file tests/runtime_30_packet_cafe.bats, line 25)
# `echo "Packet Cafe containers are not running ('lim cafe docker up'?)" >&2' failed
# [-] packet-cafe containers are not running
# Packet Cafe containers are not running ('lim cafe docker up'?)
# bats warning: Executed 1 instead of expected 39 tests