Usage¶
Subcommand groups in lim are divided by categories reflecting (a) APIs for
services or data stores (e.g., cafe for packet-cafe or ctu for the
CTU Datasets), or (b) by file type for utilities that process files of that
data type (e.g., pcap for PCAP file processing).
There is generally an about subcommand that helps get you to documentation
about those subcommands, which in most cases leads you to the appropriate web
site with online documentation.
Getting help¶
To get help information on global command arguments and options, use
the help command or --help option flag. The usage documentation
below will detail help output for each command.
usage: lim [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug]
[-D <data-directory>] [--vol-prefix <vol_prefix>] [-e]
[-E <environment>] [-n <results_limit>]
LiminalInfo command line app.
optional arguments:
--version show program's version number and exit
-v, --verbose Increase verbosity of output. Can be repeated.
-q, --quiet Suppress output except warnings and errors.
--log-file LOG_FILE Specify a file to log output. Disabled by default.
-h, --help Show help message and exit.
--debug Show tracebacks on errors.
-D <data-directory>, --data-dir <data-directory>
Root directory for holding data files (Env:
``LIM_DATA_DIR``; default: /path/to/data)
--vol-prefix <vol_prefix>
Data volume mount for Packet Café containers (Env:
``VOL_PREFIX``; default:
/Users/dittrich/packet_cafe_data)
-e, --elapsed Include elapsed time (and ASCII bell) on exit
(default: False)
-E <environment>, --environment <environment>
Deployment environment selector (Env:
``LIM_ENVIRONMENT``; default: None)
-n <results_limit>, --limit <results_limit>
Limit result to no more than this many items (0 means
no limit; default: 0)
For help information on individual commands, use ``lim <command> --help``.
Several commands have features that will attempt to open a browser. See
``lim about --help`` to see help information about this feature and how
to control which browser(s) will be used.
Author: Dave Dittrich <dave.dittrich@gmail.com>
Copyright: 2018-2020, Dave Dittrich. 2019-2020, Liminal Information Corp.
License: Apache 2.0 License
URL: https://pypi.python.org/pypi/lim-cli
Commands:
about About the ``lim`` CLI (lim-cli)
cafe about Open packet-cafe documentation. (lim-cli)
cafe admin delete Delete data for a session. (lim-cli)
cafe admin endpoints List available packet-cafe admin endpoints. (lim-cli)
cafe admin files List files in packet-cafe server. (lim-cli)
cafe admin info Return basic information about the packet-cafe service. (lim-cli)
cafe admin results List all files produced by tools. (lim-cli)
cafe admin sessions List session IDs in packet-cafe service. (lim-cli)
cafe containers build Build Packet Café Docker containers. (lim-cli)
cafe containers down Bring down Packet Café Docker containers. (lim-cli)
cafe containers images List Packet Café related Docker images. (lim-cli)
cafe containers pull Pull Packet Café Docker containers. (lim-cli)
cafe containers show Show status of Packet Café Docker containers. (lim-cli)
cafe containers up Bring up Packet Café Docker containers. (lim-cli)
cafe endpoints List available packet-cafe API endpoints. (lim-cli)
cafe info Return basic information about the packet-cafe service. (lim-cli)
cafe raw Get raw output from a specific tool, session, and request. (lim-cli)
cafe report Produce a report on results of a session+request. (lim-cli)
cafe requests List request IDs for a specific session ID. (lim-cli)
cafe results Get the results from a tool for local storage or rendering. (lim-cli)
cafe status Return the status of all tools for a session and request ID. (lim-cli)
cafe stop Stop jobs of a request ID. (lim-cli)
cafe tools List details about workers in the packet-cafe server. (lim-cli)
cafe ui Open packet-cafe UI in a browser. (lim-cli)
cafe upload Upload a file to the packet-cafe service for processing. (lim-cli)
complete print bash completion command (cliff)
ctu get Get CTU dataset components. (lim-cli)
ctu list List CTU dataset metadata. (lim-cli)
ctu overview Get CTU dataset overview. (lim-cli)
ctu show Show scenario details. (lim-cli)
ctu stats List CTU dataset metadata. (lim-cli)
help print detailed help for another command (cliff)
pcap extract ips Extract source and destination IP addresses from PCAP file(s). (lim-cli)
pcap shift network Shift timestamps or source/destination addresses in PCAP files. (lim-cli)
pcap shift time Shift timestamps or source/destination addresses in PCAP files. (lim-cli)
version About the ``lim`` CLI (lim-cli)
Formatters¶
The cliff Command Line Formulation Framework provides a set of formatting options that facilitate accessing and using stored secrets in other applications. Data can be passed directly in a structured format like CSV, or passed directly to programs like Ansible using JSON.
Attention
The formatter options are shown in the --help output for individual
commands (e.g., lim cafe info --help). For the purposes of this
chapter, including the lengthy formatter options on every command would be
quite repetitive and take up a lot of space. For this reason, the
formatter options will be suppressed for commands as documented below. The
difference (WITH and WITHOUT the formatting options) would
look like this:
WITH formatting options
cafe info¶
Return basic information about the packet-cafe service.
lim cafe info [-f {json,shell,table,value,yaml}] [-c COLUMN] [--noindent] [--prefix PREFIX] [--max-width <integer>] [--fit-width] [--print-empty] [--choose] [--cafe-host <cafe_host_ip>] [--cafe-ui-port <cafe_ui_port>] [--cafe-admin-port <cafe_admin_port>]
-f<FORMATTER>,--format<FORMATTER>¶the output format, defaults to table
-cCOLUMN,--columnCOLUMN¶specify the column(s) to include, can be repeated to show multiple columns
--noindent¶whether to disable indenting the JSON
--prefix<PREFIX>¶add a prefix to all variable names
--max-width<integer>¶Maximum display width, <1 to disable. You can also use the CLIFF_MAX_TERM_WIDTH environment variable, but the parameter takes precedence.
--fit-width¶Fit the table to the display width. Implied if –max-width greater than 0. Set the environment variable CLIFF_FIT_WIDTH=1 to always enable
--print-empty¶Print empty table if there is no data to show.
--choose¶Choose session and request (default: False)
--cafe-host<cafe_host_ip>¶IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
--cafe-ui-port<cafe_ui_port>¶TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
--cafe-admin-port<cafe_admin_port>¶TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)Return basic information about the packet-cafe service.
Use this command to determine the last session ID and last request ID, if available.
$ lim cafe info +--------------+--------------------------------------+ | Field | Value | +--------------+--------------------------------------+ | url | http://127.0.0.1:80/api/v1/info | | last_session | 9a949fe6-6520-437f-89ec-e7af6925b1e0 | | last_request | 81778bb8a9b946ba82659732baacdb44 | | version | v0.1.0 | | hostname | bf1456253115 | +--------------+--------------------------------------+To programmatically obtain the last session ID for use in other scripts, do the following:
$ lim cafe info -f shell | grep last_ last_session="9a949fe6-6520-437f-89ec-e7af6925b1e0" last_request="81778bb8a9b946ba82659732baacdb44"See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-info
WITHOUT formatting options
cafe info¶
Return basic information about the packet-cafe service.
lim cafe info [--prefix PREFIX] [--choose] [--cafe-host <cafe_host_ip>] [--cafe-ui-port <cafe_ui_port>] [--cafe-admin-port <cafe_admin_port>]
--prefix<PREFIX>¶add a prefix to all variable names
--choose¶Choose session and request (default: False)
--cafe-host<cafe_host_ip>¶IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
--cafe-ui-port<cafe_ui_port>¶TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
--cafe-admin-port<cafe_admin_port>¶TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)Return basic information about the packet-cafe service.
Use this command to determine the last session ID and last request ID, if available.
$ lim cafe info +--------------+--------------------------------------+ | Field | Value | +--------------+--------------------------------------+ | url | http://127.0.0.1:80/api/v1/info | | last_session | 9a949fe6-6520-437f-89ec-e7af6925b1e0 | | last_request | 81778bb8a9b946ba82659732baacdb44 | | version | v0.1.0 | | hostname | bf1456253115 | +--------------+--------------------------------------+To programmatically obtain the last session ID for use in other scripts, do the following:
$ lim cafe info -f shell | grep last_ last_session="9a949fe6-6520-437f-89ec-e7af6925b1e0" last_request="81778bb8a9b946ba82659732baacdb44"See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-info
About¶
about¶
About the lim CLI
lim about [--readthedocs] [--browser BROWSER] [--force]
-
--readthedocs¶ Open a browser to the
lim-clireadthedocs page (default: False{}).
-
--browser<BROWSER>¶ Browser to use for viewing (default: None).
-
--force¶ Open the browser even if process has no TTY (default: False)
Shows information about the lim CLI.
$ lim about
lim version 21.2.7.dev14+g0928855.d20210226
It will also print out copyright and related information (which
isn’t easy to force autoprogram-cliff to parse correctly in
help output).
The --readthedocs option will open a browser to the lim
documentation web page.
ABOUT THE BROWSER OPEN FEATURE
This program uses the Python webbrowser module to open a
browser.
This module supports a large set of browsers for various operating
system distributions. It will attempt to chose an appropriate
browser from operating system defaults. If it is not possible to
open a graphical browser application, it may open the lynx text
browser.
You can choose which browser webbrowser will open using the
identifier from the set in the webbrowser documentation.
Either specify the browser using the --browser option on the
command line, or export the environment variable BROWSER
set to the identifier (e.g., export BROWSER=firefox).
It is also possible to set the BROWSER environment variable
to a full path to an executable to run. On Windows 10 with Windows
Subsystem for Linux, you can use this feature to open a Windows
executable outside of WSL. (E.g., using
export BROWSER='/c/Program Files/Mozilla Firefox/firefox.exe'
will open Firefox installed in that path).
Also note that when this program attempts to open a browser,
an exception may be thrown if the process has no TTY. If this
happens, use the --force option to bypass this behavior and
attempt to open the browser anyway.
Packet Cafe¶
cafe about¶
Open packet-cafe documentation.
lim cafe about [--browser BROWSER] [--force]
-
--browser<BROWSER>¶ Browser to use for viewing (default: None).
-
--force¶ Open the browser even if process has no TTY (default: False)
Opens up the packet-cafe documenation in a browser.
$ lim cafe about
[+] opening browser 'system default' for https://iqtlabs.gitbook.io/packet-cafe
To see help information about how the browser option works and
how you can configure it, see lim about --help.
cafe admin delete¶
Delete data for a session.
lim cafe admin delete
[--all]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id [sess_id ...]]
-
--all¶ Delete data for all sessions (careful with that flag, Eugene! default: False)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
Deletes all data and id directories for one or more sessions.
As a safety feature, you must provide a session ID on the command line or choose interactively. This command will not default like other commands.
To select specific sessions, provide them as arguments.
You can select the desired session ID from a list of
available sessions with the --choose option, or
delete all sessions at once with --all:
$ lim cafe admin delete --all
[+] deleted session 531f8bad-1f01-4b10-926b-a72aa27bcdba
[+] deleted session e6129371-ab97-4225-940e-5b18cd761da7
[+] deleted session 46d4f9a9-d5db-487e-a261-91764c044b44
[+] deleted session f44dc0e5-2ad0-4cbd-aac9-98a6c8233dff
[+] deleted session 5382b1b3-39f2-4563-9486-8efb99b56243
cafe admin endpoints¶
List available packet-cafe admin endpoints.
lim cafe admin endpoints
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
List the available admin endpoints for this packet-cafe server.
$ lim cafe admin endpoints
+-------------------+
| Endpoint |
+-------------------+
| /v1 |
| /v1/id/files |
| /v1/id/results |
| /v1/ids |
| /v1/info |
| /v1/logs/{req_id} |
+-------------------+
cafe admin files¶
List files in packet-cafe server.
lim cafe admin files
[--sort-ascending | --sort-descending]
[--tree]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--tree¶ Produce tree output rather than table (default: False)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
Lists all files uploaded into the packet-cafe server. This can produce
a large amount of output with very long lines, so you may want to use the
--fit-width option to break lines to fit the screen.
You can get a tree listing of files, which is much more compact and
readable, with the --tree option.
$ lim cafe admin files --tree
files
└── 791e1034-fdb9-4fa4-a410-e1dedef7c0b8
└── dcfe1b4dd2a04d559f6600902847a11a
├── tcprewrite_dot1q-2020-06-21-21_44_49.215175-UTC
│ ├── pcap-node-splitter-2020-06-21-21_44_53.389934-UTC
│ │ ├── clients
│ │ │ ├── combined.csv.gz
│ │ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap
│ │ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap.csv.gz
│ │ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-79-147-32-84-165-147-32-84-79-data-udp-frame-eth-ip-port-0.pcap
│ │ │ └── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-79-147-32-84-165-147-32-84-79-data-udp-frame-eth-ip-port-0.pcap.csv.gz
│ │ └── servers
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-118-228-148-32-118-228-148-32-147-32-84-165-2-4-5-4-1-1-4-2-tcp-frame-eth-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-123-126-51-33-123-126-51-33-147-32-84-165-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-123-126-51-57-123-126-51-57-147-32-84-165-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-123-126-51-64-123-126-51-64-147-32-84-165-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-123-126-51-65-123-126-51-65-147-32-84-165-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-147-32-80-9-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-147-32-84-165-147-32-84-165-147-32-84-79-data-udp-frame-eth-ip-port-0.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-147-32-84-255-147-32-84-165-147-32-84-255-nbns-frame-eth-wsshort-udp-ip-port-137.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-147-32-84-79-147-32-84-165-147-32-84-79-icmp-wsshort-frame-eth-ip.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-195-113-232-73-147-32-84-165-195-113-232-73-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-209-85-149-160-147-32-84-165-209-85-149-160-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-218-29-42-137-147-32-84-165-218-29-42-137-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-220-181-111-147-147-32-84-165-220-181-111-147-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-220-181-69-213-147-32-84-165-2-4-5-4-1-1-4-2-220-181-69-213-tcp-frame-eth-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-61-135-188-157-147-32-84-165-2-4-5-4-1-1-4-2-61-135-188-157-tcp-frame-eth-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-61-135-188-210-147-32-84-165-61-135-188-210-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ ├── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-61-135-188-212-147-32-84-165-61-135-188-212-wsshort-eth-tcp-http-frame-ip-port-80.pcap
│ │ └── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-server-ip-61-135-189-50-147-32-84-165-2-4-5-4-1-1-4-2-61-135-189-50-tcp-frame-eth-ip-port-80.pcap
│ └── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45.pcap
├── test.pcap
└── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45.pcap
See https://iqtlabs.gitbook.io/packet-cafe/design/api#v-1-id-files
cafe admin info¶
Return basic information about the packet-cafe service.
lim cafe admin info
[--prefix PREFIX]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--prefix<PREFIX>¶ add a prefix to all variable names
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
Return basic information about the packet-cafe service.
$ lim cafe admin info
+--------------+-------------------------------+
| Field | Value |
+--------------+-------------------------------+
| url | http://127.0.0.1:5001/v1/info |
| version | v0.1.0 |
| hostname | 5df1f9a14bff |
+--------------+-------------------------------+
Note that the last session ID and last request ID are found in the
output of lim cafe info (not lim cafe admin info).
See https://iqtlabs.gitbook.io/packet-cafe/design/api#v-1-info
cafe admin results¶
List all files produced by tools.
lim cafe admin results
[--sort-ascending | --sort-descending]
[--tree]
[-t <tool>]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--tree¶ Produce tree output rather than table (default: False)
-
-t<tool>,--tool<tool>¶ Only show results for specified tool (default: None)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
List files produced as a result of processing uploaded files.
This can produce a large amount of output with very long lines, so
you may want to use the --fit-width option to break lines to
fit the screen.
You can get a tree listing of files, which is much more compact and
readable, with the --tree option.
$ lim cafe admin results --tree
id
└── 791e1034-fdb9-4fa4-a410-e1dedef7c0b8
└── dcfe1b4dd2a04d559f6600902847a11a
├── mercury
│ └── metadata.json
├── networkml
│ └── metadata.json
├── p0f
│ └── metadata.json
├── pcap_stats
│ └── metadata.json
├── pcapplot
│ ├── metadata.json
│ └── trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap
│ ├── 1
│ │ └── map_ASN-trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap.png
│ ├── 2
│ │ └── map_Private_RFC_1918-trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap.png
│ ├── 3
│ │ └── map_Source_Ports-trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap.png
│ └── 4
│ └── map_Destination_Ports-trace_dcfe1b4dd2a04d559f6600902847a11a_2020-06-21_21_44_45-client-ip-147-32-84-165-147-32-80-9-147-32-84-165-wsshort-frame-eth-dns-udp-ip-port-53.pcap.png
└── snort
└── metadata.json
You can filter results by session, by request, or by tool.
Filtering matches lines that contain all of the specified values.
To show results for a specific session or a specific request,
provide them as arguments to the command. To show only results
for a given tool, specify it with the -tool option.
$ lim cafe admin results --tool networkml
+---------------------------------------------------------------------------------------------------+
| Results |
+---------------------------------------------------------------------------------------------------+
| /id/6f080abf-ef71-461d-b754-a81a54fb5ad5/d709256a73b44f4e82d45f6e4ffd03e5/networkml/metadata.json |
| /id/86f71039-e6e5-44e2-90b4-3eaf27253d6d/fa142a055de24896923cc69407feeaba/networkml/metadata.json |
| /id/278adaae-df30-4d7d-883a-990ddcf6ce88/a383d781275f4dbe9e2c78ec4b8abda4/networkml/metadata.json |
| /id/bd976556-fbc3-4e2e-a808-7024c0c0f69b/6bb276459cba45b3abce9043d0bc0ad9/networkml/metadata.json |
| /id/bd976556-fbc3-4e2e-a808-7024c0c0f69b/9e74cc6f818c47ea9cd8c8ab94ce93db/networkml/metadata.json |
+---------------------------------------------------------------------------------------------------+
See https://iqtlabs.gitbook.io/packet-cafe/design/api#v-1-id-results
cafe admin sessions¶
List session IDs in packet-cafe service.
lim cafe admin sessions
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
List the current session IDS in the packet-cafe service.
Returns shell exit code 0 if one or more sessiona are
present, or 1 if none are present.
Use the -q option to suppress the output table or error
message.
$ lim cafe admin sessions
+--------------------------------------+
| SessionId |
+--------------------------------------+
| 57b1484b-5502-4e3c-b6bc-854d4aeb2038 |
| 57be4843-32c0-4943-93d8-d1ec9bc0e792 |
| 2d222a53-5b01-4d5e-a659-7da7c21d3cf6 |
| 73d532d7-3b2b-4a93-9a68-ae7091af6a2f |
| 9a949fe6-6520-437f-89ec-e7af6925b1e0 |
| 7eedfd93-4f65-4422-8d70-a4edf47d7364 |
| a42ee6ab-d60b-4d8e-a1df-cb3dc6985c81 |
+--------------------------------------+
See https://iqtlabs.gitbook.io/packet-cafe/design/api#v-1-ids
cafe docker build¶
Build Packet Café Docker images.
lim cafe docker build
[-u | --ignore-dirty]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
-u,--update¶ Update the repository contents before rebuilding (default: False)
-
--ignore-dirty¶ Ignore a dirty repository (default: False)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
Build images from source locally rather than pulling them from Docker Hub.
This is used for local deployment or development and testing locally. If
you wish to use images from Docker Hub, use lim cafe docker pull
instead.
You will be notified if the GitHub repo has newer content and the program
will exit. Use the --update flag to update the repo before building.
cafe docker down¶
Bring down Packet Café Docker containers.
lim cafe docker down
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
Bring down the container stack associated with Packet Café services.
$ lim cafe docker down
[+] running 'docker-compose down' in /Users/dittrich/packet_cafe
Stopping packet_cafe_redis_1 ... done
Stopping packet_cafe_web_1 ... done
Stopping packet_cafe_workers_1 ... done
Stopping packet_cafe_ui_1 ... done
Stopping packet_cafe_admin_1 ... done
Stopping packet_cafe_messenger_1 ... done
Stopping packet_cafe_lb_1 ... done
Removing packet_cafe_redis_1 ... done
Removing packet_cafe_web_1 ... done
Removing packet_cafe_workers_1 ... done
Removing packet_cafe_mercury_1 ... done
Removing packet_cafe_ui_1 ... done
Removing packet_cafe_pcap-dot1q_1 ... done
Removing packet_cafe_admin_1 ... done
Removing packet_cafe_messenger_1 ... done
Removing packet_cafe_pcap-splitter_1 ... done
Removing packet_cafe_ncapture_1 ... done
Removing packet_cafe_pcapplot_1 ... done
Removing packet_cafe_lb_1 ... done
Removing packet_cafe_networkml_1 ... done
Removing packet_cafe_pcap-stats_1 ... done
Removing packet_cafe_snort_1 ... done
Removing network packet_cafe_default
Removing network admin
Removing network frontend
Removing network results
Removing network backend
Removing network analysis
Removing network preprocessing
[+] you can use 'lim cafe docker up' to restart the stack
After bringing the containers down, you can generally bring them back up without having to rebuild them.
Be aware that when you are doing development on a fork of Packet Café, you will need to rebuild the images when you make any changes that will affect things inside of running containers.
If you are just standing things up for the first time, are
doing local development editing files in your clone, or are
updating the repository with --update, you will need to
rebuild the images.
cafe docker images¶
List or delete Packet Café related Docker images.
lim cafe docker images
[--sort-ascending | --sort-descending]
[--rm]
[-a]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--rm¶ Remove the images from Docker (default: False)
-
-a,--all-columns¶ Include all available columns (default: False)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
List the images associated with Packet Café services and workers.
[+] listing images for service namespace "iqtlabs", tool namespace "iqtlabs"
+--------------+-------------------------------+--------+
| ID | Repository | Tag |
+--------------+-------------------------------+--------+
| 7808ad5f74f5 | iqtlabs/packet_cafe_workers | latest |
| 83fdfb8db32d | iqtlabs/packet_cafe_redis | latest |
| 93fc21bf376a | iqtlabs/packet_cafe_messenger | latest |
| 11bb63d0c705 | iqtlabs/packet_cafe_lb | latest |
| d9194c6daf5f | iqtlabs/packet_cafe_web | latest |
| 9fc447bc9fa4 | iqtlabs/packet_cafe_ui | latest |
| 8fe33a5eec27 | iqtlabs/packet_cafe_admin | latest |
| 1a5cec5e1dab | iqtlabs/tcprewrite_dot1q | latest |
| 39c6e9ac53a9 | iqtlabs/pcap_to_node_pcap | latest |
| adcc5b1f4213 | iqtlabs/pcap_stats | latest |
| 6732f33c5b25 | iqtlabs/ncapture | latest |
| 251346bde2eb | iqtlabs/networkml | v0.6.1 |
| 6d2d5d790715 | iqtlabs/mercury | latest |
| cedfd83f10dc | iqtlabs/snort | latest |
| b56a25f62851 | iqtlabs/pcapplot | v0.1.7 |
+--------------+-------------------------------+--------+
By default, only three columns are shown. If you wish to see all
available columns, use the -a option.
You can remove all of these images from Docker’s image storage
by using the --rm option.
If you are doing development on a fork of Packet Café and have pushed images to your own namespace on Docker Hub, use the namespace and version selection options or environment variables.
cafe docker ps¶
Show running status of Packet Café Docker containers.
lim cafe docker ps
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
Produces a table listing the Docker containers associated with
Packet Café (by virtue of the com.docker.compose.project
label being set to packet_cafe).
$ lim cafe docker ps
+-------------------------+------------+--------------------------------------+---------+
| name | short_id | image | status |
+-------------------------+------------+--------------------------------------+---------+
| packet_cafe_messenger_1 | ce4eed9e01 | iqtlabs/packet_cafe_messenger:latest | running |
| packet_cafe_workers_1 | 43fff494f6 | iqtlabs/packet_cafe_workers:latest | running |
| packet_cafe_ui_1 | 794eb87ed6 | iqtlabs/packet_cafe_ui:latest | running |
| packet_cafe_web_1 | a1f8f5f7cc | iqtlabs/packet_cafe_web:latest | running |
| packet_cafe_mercury_1 | 882b12e31f | iqtlabs/mercury:v0.11.10 | running |
| packet_cafe_ncapture_1 | 5b1b10f3e0 | iqtlabs/ncapture:v0.11.10 | running |
| packet_cafe_admin_1 | 73304f16cf | iqtlabs/packet_cafe_admin:latest | running |
| packet_cafe_redis_1 | c893c408b5 | iqtlabs/packet_cafe_redis:latest | running |
| packet_cafe_lb_1 | 4530125e8e | iqtlabs/packet_cafe_lb:latest | running |
+-------------------------+------------+--------------------------------------+---------+
To just get a return value (0 for “all running” and 1 if not),
use the -q option.
$ lim -q cafe docker ps
$ echo $?
0
cafe docker pull¶
Pull Packet Café Docker images.
lim cafe docker pull
[-u | --ignore-dirty]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
-u,--update¶ Update the repository contents before pulling (default: False)
-
--ignore-dirty¶ Ignore a dirty repository (default: False)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
Pull the images associated with Packet Café services and workers from Docker Hub to cache them locally.
cafe docker up¶
Bring the Packet Café Docker container stack up.
lim cafe docker up
[-u | --ignore-dirty]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
-
-u,--update¶ Update the repository contents before rebuilding (default: False)
-
--ignore-dirty¶ Ignore a dirty repository (default: False)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
Brings up the container and network stack associated with Packet Café
services and workers if they are not yet running. Messages from
docker-compose will be output to show progress. This can be
suppressed with the -q flag.
Prior to running docker-compose, the repository directory will
be created (if it does not yet exist) or a git fetch will be
attempted to check for updates.
$ lim cafe docker up
[+] branch 'master' is up to date
[+] running 'docker-compose up -d --no-build' in /Users/dittrich/packet_cafe
Creating network "packet_cafe_default" with the default driver
Creating network "admin" with the default driver
Creating network "frontend" with the default driver
Creating network "results" with the default driver
Creating network "backend" with the default driver
Creating network "analysis" with the default driver
Creating network "preprocessing" with the default driver
Creating packet_cafe_admin_1 ... done
Creating packet_cafe_ncapture_1 ... done
Creating packet_cafe_networkml_1 ... done
Creating packet_cafe_pcap-dot1q_1 ... done
Creating packet_cafe_pcap-splitter_1 ... done
Creating packet_cafe_snort_1 ... done
Creating packet_cafe_pcap-stats_1 ... done
Creating packet_cafe_ui_1 ... done
Creating packet_cafe_web_1 ... done
Creating packet_cafe_messenger_1 ... done
Creating packet_cafe_lb_1 ... done
Creating packet_cafe_redis_1 ... done
Creating packet_cafe_mercury_1 ... done
Creating packet_cafe_workers_1 ... done
Creating packet_cafe_pcapplot_1 ... done
[+] you can now use 'lim cafe ui' to start the UI
With either -q or normal verbosity, the containers will be run in
deamon mode (i.e., run in the background) and the command will immediately
return.
Adding -v or --debug will run the containers in the foreground and
produce a stream of log output from all of the containers. This assists in
debugging and development testing. If you interrupt with CTRL-C, the
containers will be halted and you will need to bring them back up. If
you bring them down by running lim cafe docker down in another
another terminal window, you can observe the shutdown process in the
log messages and the docker-compose process will then exit.
If new updates are available in the remote repository, you will see
messages about this and lim will suggest using the --update
option and exit before starting the containers. You can skip the
update and bring the containers up with the --ignore-dirty
option.
Note that if you are building images locally, you may not be able
to use the --update option with up due to the state of the
Git repository. While lim tries to deal with the situation, it
can’t handle things like merge conflicts. It also depends on what in
the repo gets changed during the update. In some cases, the local
images will not need to be rebuilt. In other cases, they will. Docker
may let you know if a rebuild is necessary.
cafe endpoints¶
List available packet-cafe API endpoints.
lim cafe endpoints
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
List the available API endpoints for this packet-cafe server.
$ lim cafe endpoints
+---------------------------------------------------------------------+
| Endpoint |
+---------------------------------------------------------------------+
| /api/v1 |
| /api/v1/id/{session_id}/{req_id}/{tool}/{pcap}/{counter}/{filename} |
| /api/v1/ids/{session_id} |
| /api/v1/info |
| /api/v1/raw/{tool}/{counter}/{session_id}/{req_id} |
| /api/v1/results/{tool}/{counter}/{session_id}/{req_id} |
| /api/v1/status/{session_id}/{req_id} |
| /api/v1/stop/{session_id}/{req_id} |
| /api/v1/tools |
| /api/v1/upload |
+---------------------------------------------------------------------+
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v1
cafe info¶
Return basic information about the packet-cafe service.
lim cafe info
[--prefix PREFIX]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--prefix<PREFIX>¶ add a prefix to all variable names
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
Return basic information about the packet-cafe service.
Use this command to determine the last session ID and last request ID, if available.
$ lim cafe info
+--------------+--------------------------------------+
| Field | Value |
+--------------+--------------------------------------+
| url | http://127.0.0.1:80/api/v1/info |
| last_session | 9a949fe6-6520-437f-89ec-e7af6925b1e0 |
| last_request | 81778bb8a9b946ba82659732baacdb44 |
| version | v0.1.0 |
| hostname | bf1456253115 |
+--------------+--------------------------------------+
To programmatically obtain the last session ID for use in other scripts, do the following:
$ lim cafe info -f shell | grep last_
last_session="9a949fe6-6520-437f-89ec-e7af6925b1e0"
last_request="81778bb8a9b946ba82659732baacdb44"
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-info
cafe raw¶
Get raw output from a specific tool, session, and request.
lim cafe raw
[-t <tool>]
[-P]
[-I INDENT]
[--no-color]
[-C <counter>]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
-t<tool>,--tool<tool>¶ Only show results for specified tool (default: None)
-
-P,--pprint¶ Print with pprint module (default: False)
-
-I<INDENT>,--indent<INDENT>¶ Indentation amount in characters (default: 2)
-
--no-color¶ Print without terminal coloring (default: False)
-
-C<counter>,--counter<counter>¶ Counter for selecting a specific file from a set (default: 1)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
Get raw output from a specific tool, session, and request.
To select the tool from which you want output, use the --tool option.
You must select a tool (from the list produced by lim cafe tools.)
$ lim cafe raw --tool networkml | head
[
{
"81778bb8a9b946ba82659732baacdb44": {
"valid": true,
"pcap_labels": "ip-147-32-84-79-147-32-84-165-147-32-84-79-data-udp-frame-eth-ip-port-0",
"decisions": {
"behavior": "normal",
"investigate": false
},
"classification": {
If there is more than one file, use --counter to select which one.
By default, JSON output is colored unless stdout is not a TTY (e.g.,
when piping output to another program, or redirecting output to a file.)
Disable colored output with --no-color, select pprint style
pretty-printing with --pprint, and control indentation with
--indent.
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-raw-tool-counter-sess_id-req_id
cafe report¶
Produce a report on results of a session+request.
lim cafe report
[--sort-ascending | --sort-descending]
[-t <tool>]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
-t<tool>,--tool<tool>¶ Only show results for specified tool (default: None)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
Produces a report of the results from one or more workers (tools) to summarize the contents of a PCAP file.
If no tool(s) are specified, reports from all supported tools will be produced.
This report is very high level and is intended to illustrate
how to get situational awareness about flows in a packet capture
to guide further more detailed analysis. It may not include all
details from a given tool. To see the full details from a worker,
use lim cafe raw --tool TOOL instead.
$ lim cafe report --tool p0f,networkml
[+] implicitly reusing last session id 46d4f9a9-d5db-487e-a261-91764c044b44
[+] implicitly reusing last request id a93591b554fe420ebbcf14b67fc8d298
************************************************************************************
Packet Cafe Report
Date produced: 2020-06-27T03:54:06.517174+00:00
Session ID: 46d4f9a9-d5db-487e-a261-91764c044b44
Request ID: a93591b554fe420ebbcf14b67fc8d298
File: trace_a93591b554fe420ebbcf14b67fc8d298_2020-06-21_21_44_45.pcap
Original File: test.pcap
************************************************************************************
Worker results: p0f
===================
+-----------------+----------------+----------+-------------------+---------+-------------------+
| source_ip | full_os | short_os | link | raw_mtu | mac |
+-----------------+----------------+----------+-------------------+---------+-------------------+
| 10.0.2.102 | Windows 7 or 8 | Windows | Ethernet or modem | 1500 | 08:00:27:5b:df:e1 |
| 202.44.54.4 | Windows XP | Windows | Ethernet or modem | 1500 | 52:54:00:12:35:02 |
| 190.110.121.202 | Windows XP | Windows | Ethernet or modem | 1500 | 52:54:00:12:35:02 |
| 112.213.89.90 | Windows XP | Windows | Ethernet or modem | 1500 | 52:54:00:12:35:02 |
+-----------------+----------------+----------+-------------------+---------+-------------------+
Worker results: networkml
=========================
+------------+-------------------+------------+-------------------+----------+-------------+
| source_ip | source_mac | role | confidence | behavior | investigate |
+------------+-------------------+------------+-------------------+----------+-------------+
| 10.0.2.102 | 08:00:27:5b:df:e1 | GPU laptop | 99.99999999539332 | normal | no |
+------------+-------------------+------------+-------------------+----------+-------------+
cafe requests¶
List request IDs for a specific session ID.
lim cafe requests
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
List current request IDs for a specific packet-cafe session ID. By default, the last used session ID will be the default. Otherwise, specify the session ID as an argument
$ lim cafe requests --fit-width
[+] implicitly reusing last session id bae5d69c-7180-445d-a8db-22a5ef0872e8
+--------------------------+--------------------------+-------------------+--------------------------+
| Id | Filename | Original_Filename | Tools |
+--------------------------+--------------------------+-------------------+--------------------------+
| 13394ad96ef3420094387a6a | trace_13394ad96ef3420094 | test.pcap | networkml,mercury,pcap- |
| a748490f | 387a6aa748490f_2020-05-1 | | stats,snort,p0f,pcapplot |
| | 5_07_25_48.pcap | | |
+--------------------------+--------------------------+-------------------+--------------------------+
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-ids-sess_id
cafe results¶
Get the results from a tool for local storage or rendering.
lim cafe results
[-t <tool>]
[-C <counter>]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
-t<tool>,--tool<tool>¶ Only show results for specified tool (default: None)
-
-C<counter>,--counter<counter>¶ Counter for selecting a specific file from a set (default: 1)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
Get the results from a tool (in the form of HTML) for local storage or rendering.
In this version, the contents are simply put on stdout and you must
redirect them to a file. (In future, this will be saved and a browser
opened to view the file, as if you had selected a result in the web UI.)
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-results-tool-counter-sess_id-req_id
cafe status¶
Return the status of all tools for a session and request ID.
lim cafe status
[--sort-ascending | --sort-descending]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
Return the status of all tools for a session and request ID.
By default, the last session ID and request ID (if available) are used.
$ lim cafe status
[+] implicitly reusing last session id bae5d69c-7180-445d-a8db-22a5ef0872e8
[+] implicitly reusing last request id c33c56abe4c743a8b77e0b76d9548c06
+---------------+----------+----------------------------------+
| Tool | State | Timestamp |
+---------------+----------+----------------------------------+
| snort | Complete | 2020-05-15T01:25:52.669640+00:00 |
| networkml | Complete | 2020-05-15T01:26:36.616426+00:00 |
| pcap-splitter | Complete | 2020-05-15T01:25:56.362483+00:00 |
| mercury | Complete | 2020-05-15T01:25:49.773921+00:00 |
| pcap-dot1q | Complete | 2020-05-15T01:25:47.988746+00:00 |
| ncapture | Complete | 2020-05-15T01:25:46.075214+00:00 |
| pcapplot | Complete | 2020-05-15T01:26:24.899752+00:00 |
| pcap_stats | Complete | 2020-05-15T01:25:48.251749+00:00 |
| p0f | Complete | 2020-05-15T01:26:48.456883+00:00 |
+---------------+----------+----------------------------------+
If no session ID is identified, you will be prompted to choose from those that are available:
$ lim cafe status
Chose a session:
→ <CANCEL>
57b1484b-5502-4e3c-b6bc-854d4aeb2038
57be4843-32c0-4943-93d8-d1ec9bc0e792
2d222a53-5b01-4d5e-a659-7da7c21d3cf6
73d532d7-3b2b-4a93-9a68-ae7091af6a2f
9a949fe6-6520-437f-89ec-e7af6925b1e0
7eedfd93-4f65-4422-8d70-a4edf47d7364
a42ee6ab-d60b-4d8e-a1df-cb3dc6985c81
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-status-sess_id-req_id
cafe stop¶
Stop jobs of a request ID.
lim cafe stop
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
[sess_id]
[req_id]
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
sess_id¶
-
req_id¶
Stop jobs of a request ID.
This is a placeholder for future functionality. See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-stop-sess_id-req_id
cafe tools¶
List details about workers in the packet-cafe server.
The API endpoint should be called “workers” if you ask me, since the “tool” is just part of the details returned.
lim cafe tools
[--sort-ascending | --sort-descending]
[--definitions]
[--docker-service-namespace <service_namespace>]
[--docker-service-version <service_version>]
[--docker-tool-namespace <tool_namespace>]
[--docker-tool-version <tool_version>]
[--packet-cafe-github-url <github_url>]
[--packet-cafe-repo-dir <repo_dir>]
[--packet-cafe-repo-remote <repo_remote>]
[--packet-cafe-repo-branch <repo_branch>]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--definitions¶ Show definitions from workers.json file, not live (default: False)
-
--docker-service-namespace<service_namespace>¶ Namespace for Packet Café service images (Env:
LIM_CAFE_SERVICE_NAMESPACE; default: None)
-
--docker-service-version<service_version>¶ Version (tag) for Packet Café service images (Env:
LIM_CAFE_SERVICE_VERSION; default: “latest”)
-
--docker-tool-namespace<tool_namespace>¶ Namespace for Packet Café tool images (Env:
LIM_CAFE_TOOL_NAMESPACE; default: None)
-
--docker-tool-version<tool_version>¶ Version (tag) for Packet Café tool images (Env:
LIM_CAFE_TOOL_VERSION; default: “latest”)
-
--packet-cafe-github-url<github_url>¶ URL for packet_cafe GitHub repository (Env:
LIM_CAFE_GITHUB_URL; default: https://github.com/iqtlabs/packet_cafe.git)
-
--packet-cafe-repo-dir<repo_dir>¶ Directory holding clone of packet_cafe repository (Env:
LIM_CAFE_REPO_DIR; default: /home/docs/packet_cafe)
-
--packet-cafe-repo-remote<repo_remote>¶ packet_cafe repository remote (Env:
LIM_CAFE_REPO_REMOTE; default: origin)
-
--packet-cafe-repo-branch<repo_branch>¶ packet_cafe repository branch (Env:
LIM_CAFE_REPO_BRANCH; default: master)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
List tools used by workers in the packet-cafe server.
$ lim cafe tools --fit-width
+---------------+---------------------------+---------+--------+---------------+----------------+----------+---------------+
| Name | Image | Version | Labels | Stage | ViewableOutput | Outputs | Inputs |
+---------------+---------------------------+---------+--------+---------------+----------------+----------+---------------+
| pcapplot | iqtlabs/pcapplot | v0.1.5 | | analysis | True | file | pcap-splitter |
| pcap-splitter | iqtlabs/pcap_to_node_pcap | v0.11.8 | | preprocessing | False | pcap | pcap-dot1q |
| ncapture | iqtlabs/ncapture | v0.11.8 | | preprocessing | False | pcap | pcap,pcapng |
| pcap-dot1q | iqtlabs/tcprewrite_dot1q | v0.11.8 | | preprocessing | False | pcap | ncapture |
| networkml | iqtlabs/networkml | v0.5.3 | | analysis | True | rabbitmq | pcap-splitter |
| snort | iqtlabs/snort | v0.11.8 | | analysis | True | rabbitmq | pcap,pcapng |
| pcap_stats | iqtlabs/pcap_stats | v0.11.8 | | analysis | True | rabbitmq | pcap,pcapng |
| mercury | iqtlabs/mercury | v0.11.8 | | analysis | True | rabbitmq | pcap,pcapng |
| p0f | iqtlabs/p0f | v0.11.8 | | analysis | True | rabbitmq | pcap-splitter |
+---------------+---------------------------+---------+--------+---------------+----------------+----------+---------------+
The --definitions option will show the definitions as found in
the workers.json file from the repository directory, rather
than from the running system via the API. The --packet-cafe-repo-dir
option controls which directory is used. This option is most useful
when developing and testing your own images to verify what images
will be used by workers after bringing up the stack.
$ lim cafe tools --definitions
[+] definitions from workers.json file in '/Users/dittrich/packet_cafe' (branch 'master')
. . .
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-tools
cafe ui¶
Open packet-cafe UI in a browser.
lim cafe ui
[--browser BROWSER]
[--force]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
-
--browser<BROWSER>¶ Browser to use for viewing (default: None).
-
--force¶ Open the browser even if process has no TTY (default: False)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
Opens up the packet-cafe UI in a browser.
To see help information about how the browser option works and
how you can configure it, see lim about --help.
cafe upload¶
Upload a file to the packet-cafe service for processing.
lim cafe upload
[--no-track]
[--ignore-errors]
[--wait]
[--reuse-session]
[--choose]
[--cafe-host <cafe_host_ip>]
[--cafe-ui-port <cafe_ui_port>]
[--cafe-admin-port <cafe_admin_port>]
pcap
[sess_id]
-
--no-track¶ Do not track worker status in real time (default: False)
-
--ignore-errors¶ Ignore job failures when tracking status (default: False)
-
--wait¶ Wait for processing to finish (default: False)
-
--reuse-session¶ Reuse the last session ID (default: False)
-
--choose¶ Choose session and request (default: False)
-
--cafe-host<cafe_host_ip>¶ IP address for packet_cafe server (Env:
LIM_CAFE_HOST; default: ‘127.0.0.1’)
-
--cafe-ui-port<cafe_ui_port>¶ TCP port for packet_cafe UI service (Env:
LIM_CAFE_UI_PORT; default: 80)
-
--cafe-admin-port<cafe_admin_port>¶ TCP port for packet_cafe admin service (Env:
LIM_CAFE_ADMIN_PORT; default: 5001)
-
pcap¶ Path to PCAP file to upload
-
sess_id¶ Optional session ID (default is to generate)
Upload a file to the packet-cafe service for processing.
By default, the file is added to a new session. To instead
add this file to an existing session, you can (a) specify the
session ID as an argument on the command line, (b) add the
--choose flag to interactively select the session ID from
existing sessions, (c) add the --reuse-session flag to
associate this file with the last session ID, or allow the
default behavior of generating a new session.
By default, basic status information is returned (including whether the call succeeded and the session ID + request ID for this request) and if the request was accepted, the progress of each worker is tracked in real time similar to the web UI.
$ lim cafe upload ~/git/packet_cafe/notebooks/smallFlows.pcap
[+] Upload smallFlows.pcap: success
[+] Session ID (sess_id): 30b9ce67-75a4-49e6-b484-c4646b72fbd9
[+] Request ID (req_id): 4e058115ed19491193eadf58f105032b
[+] pcap_stats: complete 2020-05-23T17:29:56.982084+00:00
[+] pcap-dot1q: complete 2020-05-23T17:29:55.773211+00:00
[+] ncapture: complete 2020-05-23T17:29:53.333307+00:00
[+] mercury: complete 2020-05-23T17:29:59.330288+00:00
[+] snort: complete 2020-05-23T17:30:02.781840+00:00
[+] pcap-splitter: complete 2020-05-23T17:31:10.060056+00:00
[+] networkml: complete 2020-05-23T17:32:13.648982+00:00
[+] p0f: complete 2020-05-23T17:32:21.438466+00:00
[+] pcapplot: complete 2020-05-23T17:33:05.999342+00:00
If -v (or more) is given, even more information is produced and
tracking is performed as well.
Adding the --elapsed option includes elapsed lap time (per worker)
and total time for all workers.
$ lim cafe upload CTU-Malware-Capture-Botnet-114-1/2015-04-09_capture-win2.pcap --elapsed
[+] Upload 2015-04-09_capture-win2.pcap: success
[+] Session ID (sess_id): 46d4f9a9-d5db-487e-a261-91764c044b44
[+] Request ID (req_id): a93591b554fe420ebbcf14b67fc8d298
[+] ncapture: complete 2020-05-27T03:26:53.894222+00:00 (00:00:05.07)
[+] pcap_stats: complete 2020-05-27T03:26:56.531330+00:00 (00:00:05.07)
[+] pcap-dot1q: complete 2020-05-27T03:26:56.311676+00:00 (00:00:05.07)
[+] mercury: complete 2020-05-27T03:26:59.670225+00:00 (00:00:07.10)
[+] snort: complete 2020-05-27T03:27:03.241917+00:00 (00:00:11.16)
[+] pcap-splitter: complete 2020-05-27T03:27:03.122224+00:00 (00:00:11.16)
[+] p0f: complete 2020-05-27T03:27:07.341062+00:00 (00:00:15.22)
[+] networkml: complete 2020-05-27T03:27:08.732745+00:00 (00:00:17.25)
[+] pcapplot: complete 2020-05-27T03:27:10.634384+00:00 (00:00:19.27)
[+] Elapsed time 00:00:22.86
Adding the --no-track option will return the upload status and both
session and request IDs. You can then check on the status as needed
using lim cafe status:
$ lim cafe upload test.pcap --no-track
[+] Upload test.pcap: success
[+] Session ID (sess_id): d7c9eaaa-6360-44d0-b821-097b17d1b4fb
[+] Request ID (req_id): 20c34e04b91a4fed9b4f876e67a218c9
$ lim cafe status
+------------+-------------+----------------------------------+
| Tool | State | Timestamp |
+------------+-------------+----------------------------------+
| snort | In progress | 2020-05-15T07:18:55.281469+00:00 |
| mercury | In progress | 2020-05-15T07:18:56.288996+00:00 |
| ncapture | Complete | 2020-05-15T07:18:56.881295+00:00 |
| pcap-dot1q | In progress | 2020-05-15T07:18:56.880669+00:00 |
| pcap_stats | In progress | 2020-05-15T07:18:56.923709+00:00 |
+------------+-------------+----------------------------------+
$ lim cafe status
+---------------+-------------+----------------------------------+
| Tool | State | Timestamp |
+---------------+-------------+----------------------------------+
| snort | Complete | 2020-05-15T07:19:02.913388+00:00 |
| networkml | In progress | 2020-05-15T07:19:07.484375+00:00 |
| pcap-splitter | Complete | 2020-05-15T07:19:07.994744+00:00 |
| mercury | Complete | 2020-05-15T07:19:00.197828+00:00 |
| pcap-dot1q | Complete | 2020-05-15T07:18:59.070603+00:00 |
| ncapture | Complete | 2020-05-15T07:18:56.881295+00:00 |
| pcapplot | In progress | 2020-05-15T07:19:07.046718+00:00 |
| pcap_stats | Complete | 2020-05-15T07:18:59.209291+00:00 |
| p0f | In progress | 2020-05-15T07:19:07.994061+00:00 |
+---------------+-------------+----------------------------------+
Using the -q flag will no produce any output and will also return
immediately without tracking processing. In circumstances where you are
performing lots of uploads, it may be better to wait until all processing
for each file is done is done before uploading the next file. Use the
--wait flag to do this.
By default when waiting for the status of jobs, any failures result in
an error message and the program will exit. You can disable this by
using the --ignore-errors flag, but be aware that doing so may
cause the program to hang indefinitely.
See https://iqtlabs.gitbook.io/packet-cafe/design/api#api-v-1-upload
CTU¶
ctu get¶
Get CTU dataset components.
lim ctu get
[--force]
[--no-subdir]
[-P <protocol-list>]
[-L <lines>]
[--cache-file CACHE_FILE]
[--ignore-cache]
scenario
{zip,labeled,binetflow,pcap,weblogng,all}
[{zip,labeled,binetflow,pcap,weblogng,all} ...]
-
--force¶ Force over-writing files if they exist (default:
False)
-
--no-subdir¶ Do not maintain scenario name subdirectory (default:
False)
-
-P<protocol-list>,--protocols<protocol-list>¶ Protocols to include, or ‘any’ (default:
icmp,tcp,udp)
-
-L<lines>,--maxlines<lines>¶ Maximum number of lines to get (default:
None)
-
--cache-file<CACHE_FILE>¶ Cache file path for CTU metadata (Env:
LIM_CTU_CACHE; default:/home/docs/.lim-ctu-cache.json)
-
--ignore-cache¶ Ignore any cached results (default:
False)
-
scenario¶
-
data¶
Get one or more data components from a scenario. These components are the raw PCAP file, Netflow file, and other analytic products from intrusion detection system processing, etc.
See lim ctu list --help for more on the scenario argument.
For the data argument, you can use all to recursively
download all scenario data, or one or more of the data
files by type: zip, labeled, binetflow, pcap, weblogng
By default, or when using the all attribute identifier,
the file(s) are placed in a subdirectory with the full name
of the scenario to better organize data across multiple
scenarios. You can override this when getting specific files
(i.e., not using all) with the --no-subdir option.
When publishing content derived from this data, make sure to respect
the Disclaimer at the bottom of the scenario Readme.* files:
These files were generated in the Stratosphere Lab as part of the
Malware Capture Facility Project in the CVUT University, Prague,
Czech Republic. The goal is to store long-lived real botnet traffic
and to generate labeled netflows files.
Any question feel free to contact us:
Sebastian Garcia: sebastian.garcia@agents.fel.cvut.cz
You are free to use these files as long as you reference this project
and the authors as follows:
Garcia, Sebastian. Malware Capture Facility Project. Retrieved
from https://stratosphereips.org
To cite the [CTU13] dataset please cite the paper “An empirical comparison of botnet detection methods” Sebastian Garcia, Martin Grill, Jan Stiborek and Alejandro Zunino. Computers and Security Journal, Elsevier. 2014. Vol 45, pp 100-123. http://dx.doi.org/10.1016/j.cose.2014.05.011
ctu list¶
List CTU dataset metadata.
lim ctu list
[--sort-ascending | --sort-descending]
[--cache-file CACHE_FILE]
[--ignore-cache]
[--date-starting <YYYY-MM-DD>]
[--date-ending <YYYY-MM-DD>]
[--fullnames]
[-a]
[--hash <{md5_hash|sha256_hash}>]
[--malware-includes <string>]
[--name-includes <string>]
[--description-includes <string>]
[scenario [scenario ...]]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--cache-file<CACHE_FILE>¶ Cache file path for CTU metadata (Env:
LIM_CTU_CACHE; default:/home/docs/.lim-ctu-cache.json)
-
--ignore-cache¶ Ignore any cached results (default:
False)
-
--date-starting<YYYY-MM-DD>¶ List scenarios starting from this date (default: ‘1970-01-01’)
-
--date-ending<YYYY-MM-DD>¶ List scenarios up to this date (default: ‘2021-02-26’)
-
--fullnames¶ Show full names
-
-a,--everything¶ Show all metadata columns (default : False)
-
--hash<{md5_hash|sha256_hash}>¶ Only list scenarios that involve a specific hash (default:
None)
-
--malware-includes<string>¶ Only list scenarios including this stringin the ‘Malware’ column (default:
None)
-
--name-includes<string>¶ Only list scenario including this stringin the ‘Capture_Name’ column (default:
None)
-
--description-includes<string>¶ Only list scenarios including this stringin the description (default:
None)
-
scenario¶
List scenarios (a.k.a., “captures”) and related metadata.
By default, all scenarios are listed. You can limit the output
by filtering on several attributes (e.g., by Capture_Name
field, by date range, contents of the malware name or web page
description, etc.) You can also limit the number of items
shown if necessary when the number of results is large.
The scenario argument equates to the field Capture_Name in
the index. This can be the scenario’s full name (e.g.,
CTU-IoT-Malware-Capture-34-1) or an abbreviated form of the
name (e.g., IoT-34-1 or just 34-1).
$ lim ctu list IoT-34-1 Botnet-42
+----------------+-------------------------------+---------+
| Infection_Date | Capture_Name | Malware |
+----------------+-------------------------------+---------+
| 2011-08-10 | CTU-Malware-Capture-Botnet-42 | Neeris |
| 2018-12-21 | CTU-IoT-Malware-Capture-34-1 | Mirai |
+----------------+-------------------------------+---------+
A larger number of attributes are available. You can get all of them
using the -a (--everything) flag. The subset of columns shown
by default is: infection_date, capture_name, malware
Valid column labels for options -c, --column, --sort-column,
or to be shown with -a, include:
infection_date, capture_name, malware, md5, sha256, capture_url, zip, labeled, binetflow, pcap, weblogng
Using lim ctu list -a produces very wide output. Even if many fields
are None and --fit-width is included, it is still unwieldy for just
one scenario as you can see here. Consider using lim ctu show instead.
$ lim ctu list --name-includes IoT --malware-includes muhstik --fit-width -a
+----------------+------------------------+---------+------------------------+------------------------+------------------------+------------------------+---------+------------------------+------------------------+-----------------------------+
| Infection_Date | Capture_Name | Malware | MD5 | SHA256 | Capture_URL | ZIP | LABELED | BINETFLOW | PCAP | WEBLOGNG |
+----------------+------------------------+---------+------------------------+------------------------+------------------------+------------------------+---------+------------------------+------------------------+-----------------------------+
| 2018-05-19 | CTU-IoT-Malware- | Muhstik | b8849fe97e39ae3afd6def | 5ce13670bc875e913e6f08 | https://mcfp.felk.cvut | fce7b8bbd1c1fba1d75b9d | None | 2018-05-21_capture.bin | 2018-05-21_capture.pca | 2018-05-21_capture.weblogng |
| | Capture-3-1 | | 618568bb09 | 7a4ac0a9e343347d5babb3 | .cz/publicDatasets/IoT | c1a60b25f49f68c9ec16b3 | | etflow | p | |
| | | | | b5c63e1d1b199371f69a | Datasets/CTU-IoT- | 656b52ed28290fc93c72.z | | | | |
| | | | | | Malware-Capture-3-1 | ip | | | | |
+----------------+------------------------+---------+------------------------+------------------------+------------------------+------------------------+---------+------------------------+------------------------+-----------------------------+
There are also a number of filters that can be applied, including MD5
and SHA256 hash, substrings in the Capture_Name or Malware
fields, start and end dates, or description of the scenario in its
web page.
The --hash option makes an exact match on any of the stored hash
values. This is the hash of the executable binary referenced in the
ZIP column. This example uses the most frequently occuring MD5
hash as seen in lim ctu stats --help:
$ lim ctu list --hash e515267ba19417974a63b51e4f7dd9e9
+----------------+----------------------------------+---------+
| Infection_Date | Capture_Name | Malware |
+----------------+----------------------------------+---------+
| 2015-03-04 | CTU-Malware-Capture-Botnet-110-1 | HTBot |
| 2015-03-04 | CTU-Malware-Capture-Botnet-110-2 | HTBot |
| 2015-03-09 | CTU-Malware-Capture-Botnet-110-3 | HTBot |
| 2015-03-09 | CTU-Malware-Capture-Botnet-111-2 | HTBot |
| 2015-04-09 | CTU-Malware-Capture-Botnet-110-4 | HTBot |
| 2015-04-09 | CTU-Malware-Capture-Botnet-111-3 | HTBot |
| 2015-04-22 | CTU-Malware-Capture-Botnet-110-5 | HTBot |
| 2015-04-22 | CTU-Malware-Capture-Botnet-111-4 | HTBot |
| 2015-04-23 | CTU-Malware-Capture-Botnet-110-6 | HTBot |
| 2015-06-09 | CTU-Malware-Capture-Botnet-111-5 | HTBot |
+----------------+----------------------------------+---------+
The --malware-includes option is rather simplistic, matching any
occurance of the substring (case insensitive) in the Malware field.
The same applies for the --name-includes option with respect to the
Capture_Name field. For more accurate matching, you may want to use
something like the -f csv option and match on regular expressions
using one of the grep variants. Or add regular expression handling
and submit a pull request! ;)
When publishing content derived from this data, make sure to respect
the Disclaimer at the bottom of the scenario Readme.* files:
These files were generated in the Stratosphere Lab as part of the
Malware Capture Facility Project in the CVUT University, Prague,
Czech Republic. The goal is to store long-lived real botnet traffic
and to generate labeled netflows files.
Any question feel free to contact us:
Sebastian Garcia: sebastian.garcia@agents.fel.cvut.cz
You are free to use these files as long as you reference this project
and the authors as follows:
Garcia, Sebastian. Malware Capture Facility Project. Retrieved
from https://stratosphereips.org
To cite the [CTU13] dataset please cite the paper “An empirical comparison of botnet detection methods” Sebastian Garcia, Martin Grill, Jan Stiborek and Alejandro Zunino. Computers and Security Journal, Elsevier. 2014. Vol 45, pp 100-123. http://dx.doi.org/10.1016/j.cose.2014.05.011
ctu overview¶
Get CTU dataset overview.
lim ctu overview
[--browser BROWSER]
[--force]
[--cache-file CACHE_FILE]
[--ignore-cache]
[scenario [scenario ...]]
-
--browser<BROWSER>¶ Browser to use for viewing (default: None).
-
--force¶ Open the browser even if process has no TTY (default: False)
-
--cache-file<CACHE_FILE>¶ Cache file path for CTU metadata (Env:
LIM_CTU_CACHE; default:/home/docs/.lim-ctu-cache.json)
-
--ignore-cache¶ Ignore any cached results (default:
False)
-
scenario¶
Opens a browser for the web page containing the scenario descriptions and data links.
Arguments are scenario names using either the full name
form (e.g., CTU-Malware-Capture-Botnet-123-1) or an
abbreviated form (e.g., Botnet-123-1).
The URL to use is the one seen in the SCENARIO_URL column
of the output of the lim ctu list command.
To see help information about how the browser option works and
how you can configure it, see lim about --help.
ctu show¶
Show scenario details.
lim ctu show
[--prefix PREFIX]
[--cache-file CACHE_FILE]
[--ignore-cache]
[scenario]
-
--prefix<PREFIX>¶ add a prefix to all variable names
-
--cache-file<CACHE_FILE>¶ Cache file path for CTU metadata (Env:
LIM_CTU_CACHE; default:/home/docs/.lim-ctu-cache.json)
-
--ignore-cache¶ Ignore any cached results (default:
False)
-
scenario¶
Shows details about an individual scenario in tabular form.
See lim ctu list --help for more on the scenario argument.
$ lim ctu show iot-3-1
+----------------+----------------------------------------------------------------------------------+
| Field | Value |
+----------------+----------------------------------------------------------------------------------+
| Infection_Date | 2018-05-19 |
| Capture_Name | CTU-IoT-Malware-Capture-3-1 |
| Malware | Muhstik |
| MD5 | b8849fe97e39ae3afd6def618568bb09 |
| SHA256 | 5ce13670bc875e913e6f087a4ac0a9e343347d5babb3b5c63e1d1b199371f69a |
| Capture_URL | https://mcfp.felk.cvut.cz/publicDatasets/IoTDatasets/CTU-IoT-Malware-Capture-3-1 |
| ZIP | fce7b8bbd1c1fba1d75b9dc1a60b25f49f68c9ec16b3656b52ed28290fc93c72.zip |
| LABELED | None |
| BINETFLOW | 2018-05-21_capture.binetflow |
| PCAP | 2018-05-21_capture.pcap |
| WEBLOGNG | 2018-05-21_capture.weblogng |
+----------------+----------------------------------------------------------------------------------+
ctu stats¶
List CTU dataset metadata.
lim ctu stats
[--sort-ascending | --sort-descending]
[--cache-file CACHE_FILE]
[--ignore-cache]
[{infection_date,capture_name,malware,md5,sha256,capture_url}]
-
--sort-ascending¶ sort the column(s) in ascending order
-
--sort-descending¶ sort the column(s) in descending order
-
--cache-file<CACHE_FILE>¶ Cache file path (default:
/home/docs/.lim-ctu-cache.json)
-
--ignore-cache¶ Ignore any cached results (default:
False)
-
attribute¶ Attribute to quantify (default:
infection_date)
Shows the selected dataset attribute and a count of unique instances in reverse order of occurance.
$ lim ctu stats md5 | head
+----------------------------------+-------+
| MD5 | Count |
+----------------------------------+-------+
| e515267ba19417974a63b51e4f7dd9e9 | 10 |
| - | 9 |
| e1090d7126dd88d0d1d39b68ea3aae11 | 6 |
| 05a00c320754934782ec5dec1d5c0476 | 6 |
| 48616dd47e12e369feef53a57830158a | 5 |
| 11bc606269a161555431bacf37f7c1e4 | 5 |
| bf08e6b02e00d2bc6dd493e93e69872f | 4 |
Possible attributes are those that come from the CTU index
file (infection_date, capture_name, malware, md5, sha256, capture_url).
To see more detailed descriptions of the CTU datasets as a whole,
use lim ctu overview to view the appropriate web page.
PCAP¶
pcap extract ips¶
Extract source and destination IP addresses from PCAP file(s).
lim pcap extract ips [--stdout] [pcap [pcap ...]]
-
--stdout¶ Write output to stdout (default: False).
-
pcap¶
Output is a sorted list of unique IP addresses. By default, the results are
written to a file with the same base name as the input, but ending
in .ips. To output to standard output, use the --stdout option.
pcap shift network¶
Shift timestamps or source/destination addresses in PCAP files.
lim pcap shift network [--start-time START_TIME] [pcap [pcap ...]]
-
--start-time<START_TIME>¶ New starting time for first packet (default: None).
-
pcap¶
Adjusts the timestamps in the ethernet frame headers of packets in a PCAP
file by rebasing them to the specified date. The --start-time is specified
in ISO 8601 date format, e.g., 2019-09-01T12:00:00Z or
2019-09-01T20:00:00.00-08:00.
To see the old and new timestamps for each packet as they are converted,
use -vv.
NOTE 1: Keep in mind that this utility only maniuplates the packet headers. This means that any embedded timestamps in the body of ethernet frames (e.g., in the UDP or TCP data portion of the packet) do not get adjusted.
NOTE 2: The network address shifting logic has not been completed yet.
The program raises an exception with a message to that effect.
pcap shift time¶
Shift timestamps or source/destination addresses in PCAP files.
lim pcap shift time [--start-time START_TIME] [pcap [pcap ...]]
-
--start-time<START_TIME>¶ New starting time for first packet (default: None).
-
pcap¶
Adjusts the timestamps in the ethernet frame headers of packets in a PCAP
file by rebasing them to the specified date. The --start-time is specified
in ISO 8601 date format, e.g., 2019-09-01T12:00:00Z or
2019-09-01T20:00:00.00-08:00.
To see the old and new timestamps for each packet as they are converted,
use -vv.
NOTE 1: Keep in mind that this utility only maniuplates the packet headers. This means that any embedded timestamps in the body of ethernet frames (e.g., in the UDP or TCP data portion of the packet) do not get adjusted.
NOTE 2: The network address shifting logic has not been completed yet.
The program raises an exception with a message to that effect.