Testing

Test-driven development improves the ability to upgrade your code base to keep up with API changes, to refactor without incurring regression errors, or to validate when bugs in dependent libraries have been fixed.

Testing in lim is driven by tox from the Makefile in the top level repo directory, using the Bats-core: Bash Automated Testing System (2018) testing framework. See Testing Your Shell Scripts, with Bats for information on using BATS.

Note

Note that bats-core is a more recent fork of the original Bats: Bash Automated Testing System system, which is no longer being actively maintained. bats-core has more features than the older bats program that may come pre-installed with your OS or package management system. The tests here rely on things like the setup_file() function to ensure data files are present for runtime tests.

The Makefile in this repo ensures that the newer bats-core program is installed into /usr/local/bin and that related related libraries that are used by tests are installed locally in the tests/lib directory.

Run make help to see the targets associated with testing. The primary targets are test for unit testing code, and test-bats-runtime for runtime integration testing.

The tox tests are performed using the following configuration file:

[tox]
# In practice, you can optimize by first running basic tests
# 'pep8,bandit,docs' and only after those succeed go on to
# run the remaining (default) tests. E.g.,
# $ tox -e pep8,bandit,docs && tox -e py36,py37,py38,bats,pypi

# envlist = pep8,bandit,docs,py36,py37,py38,bats,pypi
envlist = py36,py37,py38,bats,pypi
skip_missing_interpreters = true
requires = tox-conda
           setuptools>=42
           setuptools_scm

[testenv]
setenv =
    VIRTUAL_ENV={envdir}
    BRANCH_NAME=master
    PYTHONPATH={toxinidir}:{toxinidir}/lim
distribute = False
install_command = python -m pip install {opts} {packages}
conda_deps =
       pytest
conda_channels =
       conda-forge
# Make sure these match setup.py!
deps = -Ur{toxinidir}/requirements.txt
       pbr>=5.4.5
       pip>=20.2.2
       pytest
commands = pytest {posargs}

; If you want to make tox run the tests with the same versions, create a
; requirements.txt with the pinned versions and uncomment the following lines:
; deps =
;     -r{toxinidir}/requirements.txt

[testenv:pypi]
basepython = python3.8
whitelist_externals = make
deps = -Ur{toxinidir}/requirements.txt
       twine
commands = make twine-check

[testenv:pep8]
basepython = python3.8
deps = -Ur{toxinidir}/requirements.txt
       flake8>=3.8.3
commands = flake8 lim setup.py

[testenv:bandit]
basepython = python3.8
; Run security linter
deps = -Ur{toxinidir}/requirements.txt
       bandit>=1.1.0
commands = bandit -c bandit.yaml -r lim -x tests -n5

[testenv:docs]
basepython = python3.8
deps = -Ur{toxinidir}/requirements.txt
commands = sphinx-build -b html docs docs/_build

[testenv:bats]
; Run bats unit tests
; Deal with this by requiring docutils==0.15:
; #   Traceback (most recent call last):
; #     File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/cliff/help.py", line 43, in __call__
; #       factory = ep.load()
; #     File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2444, in load
; #       self.require(*args, **kwargs)
; #     File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2467, in require
; #       items = working_set.resolve(reqs, env, installer, extras=self.extras)
; #     File "/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
; #       raise VersionConflict(dist, req).with_context(dependent_req)
; #   pkg_resources.ContextualVersionConflict: (docutils 0.16 (/Users/dittrich/git/python_secrets/.tox/bats/lib/python3.7/site-packages), Requirement.parse('docutils<0.16,>=0.10'), {'botocore'})
deps = -Ur{toxinidir}/requirements.txt
       docutils==0.15
whitelist_externals = make
commands = make test-bats

There are built-in unit tests and security tests for the Python lim application, as well as runtime bats tests that perform realtime integration or system tests.

Note

The examples of output from bats and tox tests are truncated to 80 characters and do not represent the exact output you would see if you ran these commands at the command line.

Data files

Some of the tests require PCAP files. One of the test files (smallFlows_nopayloads.pcap) comes from the Packet Café source repository. If a clone of that directory exists at ~/git/packet_cafe, the file will be copied from the local clone. Otherwise, it will be downloaded from GitHub.

Other test files come from the CTU web server and are downloaded as needed (using lim, of course.)

Unit tests

Invoke unit tests with the helper Makefile using the test target (e.g., make test). This target runs tox tests for the Python code (make test-tox), followed by some basic unit tests performed with bats (make test-bats).

If successful, you will see the following:

$ make test-tox
tox
GLOB sdist-make: /Users/dittrich/git/LiminalInfo/lim-cli/setup.py
py36 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py36 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py36 run-test-pre: PYTHONHASHSEED='1917477129'
py37 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py37 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py37 run-test-pre: PYTHONHASHSEED='1917477129'
py38 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
py38 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
py38 run-test-pre: PYTHONHASHSEED='1917477129'
pep8 inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
pep8 installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
pep8 run-test-pre: PYTHONHASHSEED='1917477129'
pep8 run-test: commands[0] | flake8 lim setup.py --exclude text.py
bandit inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/
bandit installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4
bandit run-test-pre: PYTHONHASHSEED='1917477129'
bandit run-test: commands[0] | bandit -c bandit.yaml -r lim -x tests -n5
[main]	INFO	profile include tests: None
[main]	INFO	profile exclude tests: B110,B101
[main]	INFO	cli include tests: None
[main]	INFO	cli exclude tests: None
[main]	INFO	using config: bandit.yaml
[main]	INFO	running on Python 3.8.5
Run started:2020-08-12 19:21:31.448772

Test results:
	No issues identified.

Code scanned:
	Total lines of code: 3872
	Total lines skipped (#nosec): 9

Run metrics:
	Total issues (by severity):
		Undefined: 0.0
		Low: 0.0
		Medium: 0.0
		High: 0.0
	Total issues (by confidence):
		Undefined: 0.0
		Low: 0.0
		Medium: 0.0
		High: 0.0
Files skipped (0):
bats inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
bats installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
bats run-test-pre: PYTHONHASHSEED='1917477129'
bats run-test: commands[0] | make test-bats
[+] Running bats tests: 00_help.bats
1..6
ok 1 "lim about" contains "version"
ok 2 'lim help' can load all entry points
ok 3 "lim cafe --help" properly lists subcommands
ok 4 "lim ctu --help" properly lists subcommands
ok 5 "lim pcap --help" properly lists subcommands
ok 6 'lim --version' works
docs inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
docs installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
docs run-test-pre: PYTHONHASHSEED='1917477129'
docs run-test: commands[0] | sphinx-build -b html docs docs/_build
Running Sphinx v3.2.0
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 2 changed, 0 removed
reading sources... [ 50%] changes
reading sources... [100%] testing

/Users/dittrich/git/LiminalInfo/lim-cli/.tox/docs/lib/python3.8/site-packages/se
  warnings.warn(
/Users/dittrich/git/LiminalInfo/lim-cli/docs/testing.rst:65: WARNING: Include fi
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 33%] changes
writing output... [ 66%] index
writing output... [100%] testing

generating indices...  genindexdone
writing additional pages...  searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in docs/_build.
pypi inst-nodeps: /Users/dittrich/git/LiminalInfo/lim-cli/.tox/.tmp/package/4/li
pypi installed: aiohttp==3.6.2,alabaster==0.7.12,anytree==2.8.0,appdirs==1.4.4,a
pypi run-test-pre: PYTHONHASHSEED='1917477129'
pypi run-test: commands[0] | python setup.py check --restructuredtext
running check
___________________________________ summary ____________________________________
  py36: commands succeeded
  py37: commands succeeded
  py38: commands succeeded
  pep8: commands succeeded
  bandit: commands succeeded
  bats: commands succeeded
  docs: commands succeeded
  pypi: commands succeeded
  congratulations :)

Immediately following this are the bats unit tests. Successful results will look like this:

[+] Running bats tests: 00_help.bats
1..6
ok 1 "lim about" contains "version"
ok 2 'lim help' can load all entry points
ok 3 "lim cafe --help" properly lists subcommands
ok 4 "lim ctu --help" properly lists subcommands
ok 5 "lim pcap --help" properly lists subcommands
ok 6 'lim --version' works

Integration tests

The integration and system tests using bats require a live network connection and/or a running packet-cafe server. Because of this, these tests are only run on demand and not as part of basic unit testing and code analysis.

The bats tests can then be run using the helper Makefile target test-bats-runtime.

$ make test-bats-runtime
[+] Running bats runtime tests: runtime_10_ctu.bats runtime_20_pcap.bats runtime
1..35
ok 1 "lim -q ctu list CTU-Malware-Capture-Botnet-48" works
ok 2 "lim -q ctu list Botnet-48" works
ok 3 "lim -q ctu get Botnet-48 pcap --no-subdir " gets PCAP file to cwd
ok 4 "lim -q ctu get Botnet-48 pcap" gets PCAP file
ok 5 "lim pcap extract ips CTU-Malware-Capture-Botnet-48/botnet-capture-20110816
ok 6 "lim pcap extract ips --stdout CTU-Malware-Capture-Botnet-48/botnet-capture
ok 7 "lim pcap shift time CTU-Malware-Capture-Botnet-48/botnet-capture-20110816-
ok 8 VOL_PREFIX is exported
ok 9 packet-cafe Docker containers are running (via "docker ps")
ok 10 "lim -q cafe containers" reports Docker containers are running
ok 11 "lim cafe endpoints" includes "/api/v1/info"
ok 12 "lim cafe admin endpoints" includes "/v1/info"
ok 13 "lim cafe tools" includes "iqtlabs/"
ok 14 "lim cafe info" includes "hostname"
ok 15 "lim cafe admin info" includes "hostname"
ok 16 "lim cafe admin sessions" has no sessions
ok 17 "lim -q cafe admin sessions" returns failure
ok 18 "lim cafe upload --wait ~/git/packet_cafe/notebooks/smallFlows_nopayloads.
ok 19 "lim cafe status" contains "Complete"
ok 20 "lim cafe admin results" contains "metadata.json" files
ok 21 "lim cafe admin files" contains "smallFlows_nopayloads.pcap"
ok 22 "lim cafe upload --wait CTU-Malware-Capture-Botnet-48/botnet-capture-20110
ok 23 "lim -q cafe report --tool poof" fails
ok 24 "lim -q cafe report --tool p0f" works
ok 25 "lim cafe results --tool networkml" works
ok 26 "lim cafe admin sessions -f value" shows both sessions
ok 27 "lim cafe admin delete 22222222-2222-2222-2222-222222222222" removes sessi
ok 28 "lim cafe admin delete --all" leaves storage directory empty
ok 29 "lim cafe raw --tool p0f" fails
ok 30 "lim cafe raw --tool p0f --choose" fails (no tty)
ok 31 "lim cafe results --tool p0f" fails
ok 32 "lim cafe results --choose" fails (no tty)
ok 33 "lim cafe about" fails (no tty)
ok 34 "lim cafe ui" fails (no tty)
ok 35 Cleaning up /tmp/packet_cafe_status

Writing and debugging BATS tests

Files that are used by tests are loaded into the directory pointed to by the runtime environment variable BATS_RUN_TMPDIR. This is to avoid cluttering up the repo directory with temporary files. This directory is removed on exit (see source code).

To see the values of these variables, place the following command in the .bats file and run it with the -t flag:

env | grep BATS_ >&3

The output will expose the runtime values:

$ bats -t tests/runtime_30_packet_cafe.bats
1..39
BATS_ROOT_PID=55272
BATS_TEST_SOURCE=/Users/dittrich/tmp/bats-run-55272/bats.55345.src
BATS_TMPDIR=/Users/dittrich/tmp
BATS_RUN_TMPDIR=/Users/dittrich/tmp/bats-run-55272
BATS_VERSION=1.2.1
BATS_TEST_PATTERN=^[[:blank:]]*@test[[:blank:]]+(.*[^[:blank:]])[[:blank:]]+\{(.*)$
BATS_CWD=/Users/dittrich/git/LiminalInfo/lim-cli
BATS_TEMPDIR_CLEANUP=1
BATS_TEST_FILTER=
BATS_TEST_PATTERN_COMMENT=[[:blank:]]*([^[:blank:]()]+)[[:blank:]]*\(?\)?[[:blank:]]+\{[[:blank:]]+#[[:blank:]]*@test[[:blank:]]*$
BATS_LIBEXEC=/usr/local/libexec/bats-core
BATS_ROOT=/usr/local
not ok 1 setup_file failed
# (from function `setup_file' in test file tests/runtime_30_packet_cafe.bats, line 25)
#   `echo "Packet Cafe containers are not running ('lim cafe docker up'?)" >&2' failed
# [-] packet-cafe containers are not running
# Packet Cafe containers are not running ('lim cafe docker up'?)
# bats warning: Executed 1 instead of expected 39 tests