diff --git a/develop/search/search_index.json b/develop/search/search_index.json index e2240af16..c283595b8 100644 --- a/develop/search/search_index.json +++ b/develop/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Introduction","text":""},{"location":"#introduction","title":"Introduction","text":"

IntelMQ is a solution for IT security teams (CERTs & CSIRTs, SOCs abuse departments, etc.) for collecting and processing security feeds (such as log files) using a message queuing protocol. It's a community driven initiative called IHAP1 (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.

IntelMQ is frequently used for:

The design was influenced by AbuseHelper however it was re-written from scratch and aims at:

It follows the following basic meta-guidelines:

"},{"location":"#contribute","title":"Contribute","text":"

  1. Incident Handling Automation Project, mailing list: ihap@lists.trusted-introducer.org\u00a0\u21a9

"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#changelog","title":"CHANGELOG","text":""},{"location":"changelog/#331-unreleased","title":"3.3.1 (unreleased)","text":""},{"location":"changelog/#configuration","title":"Configuration","text":""},{"location":"changelog/#core","title":"Core","text":""},{"location":"changelog/#development","title":"Development","text":""},{"location":"changelog/#data-format","title":"Data Format","text":""},{"location":"changelog/#bots","title":"Bots","text":""},{"location":"changelog/#collectors","title":"Collectors","text":""},{"location":"changelog/#parsers","title":"Parsers","text":""},{"location":"changelog/#experts","title":"Experts","text":""},{"location":"changelog/#outputs","title":"Outputs","text":""},{"location":"changelog/#documentation","title":"Documentation","text":""},{"location":"changelog/#packaging","title":"Packaging","text":""},{"location":"changelog/#tests","title":"Tests","text":""},{"location":"changelog/#tools","title":"Tools","text":""},{"location":"changelog/#contrib","title":"Contrib","text":""},{"location":"changelog/#known-issues","title":"Known issues","text":""},{"location":"changelog/#330-2024-03-01","title":"3.3.0 (2024-03-01)","text":""},{"location":"changelog/#configuration_1","title":"Configuration","text":""},{"location":"changelog/#core_1","title":"Core","text":""},{"location":"changelog/#development_1","title":"Development","text":""},{"location":"changelog/#data-format_1","title":"Data Format","text":""},{"location":"changelog/#bots_1","title":"Bots","text":""},{"location":"changelog/#collectors_1","title":"Collectors","text":""},{"location":"changelog/#parsers_1","title":"Parsers","text":""},{"location":"changelog/#experts_1","title":"Experts","text":""},{"location":"changelog/#outputs_1","title":"Outputs","text":""},{"location":"changelog/#documentation_1","title":"Documentation","text":""},{"location":"changelog/#packaging_1","title":"Packaging","text":""},{"location":"changelog/#tests_1","title":"Tests","text":""},{"location":"changelog/#tools_1","title":"Tools","text":""},{"location":"changelog/#contrib_1","title":"Contrib","text":""},{"location":"changelog/#known-issues_1","title":"Known issues","text":""},{"location":"changelog/#321-2023-08-28","title":"3.2.1 (2023-08-28)","text":""},{"location":"changelog/#core_2","title":"Core","text":""},{"location":"changelog/#bots_2","title":"Bots","text":""},{"location":"changelog/#experts_2","title":"Experts","text":""},{"location":"changelog/#320-2023-07-18","title":"3.2.0 (2023-07-18)","text":""},{"location":"changelog/#core_3","title":"Core","text":""},{"location":"changelog/#development_2","title":"Development","text":""},{"location":"changelog/#bots_3","title":"Bots","text":""},{"location":"changelog/#collectors_2","title":"Collectors","text":""},{"location":"changelog/#parsers_2","title":"Parsers","text":""},{"location":"changelog/#experts_3","title":"Experts","text":""},{"location":"changelog/#outputs_2","title":"Outputs","text":""},{"location":"changelog/#documentation_2","title":"Documentation","text":""},{"location":"changelog/#tests_2","title":"Tests","text":""},{"location":"changelog/#tools_2","title":"Tools","text":""},{"location":"changelog/#known-issues_2","title":"Known Issues","text":"

This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382). - Breaking changes in 'rt' library (#2367). - Stomp collector failed (#2342). - Type error with SQL output bot's prepare_values returning list instead of tuple (#2255). - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).

"},{"location":"changelog/#310-2023-02-10","title":"3.1.0 (2023-02-10)","text":""},{"location":"changelog/#core_4","title":"Core","text":""},{"location":"changelog/#development_3","title":"Development","text":""},{"location":"changelog/#bots_4","title":"Bots","text":""},{"location":"changelog/#collectors_3","title":"Collectors","text":""},{"location":"changelog/#parsers_3","title":"Parsers","text":""},{"location":"changelog/#experts_4","title":"Experts","text":""},{"location":"changelog/#outputs_3","title":"Outputs","text":""},{"location":"changelog/#documentation_3","title":"Documentation","text":""},{"location":"changelog/#packaging_2","title":"Packaging","text":""},{"location":"changelog/#tests_3","title":"Tests","text":""},{"location":"changelog/#tools_3","title":"Tools","text":""},{"location":"changelog/#contrib_2","title":"Contrib","text":""},{"location":"changelog/#known-issues_3","title":"Known issues","text":"

This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - Missing commas in SQL query for separate Events table (#2125). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).

"},{"location":"changelog/#302-2021-09-10","title":"3.0.2 (2021-09-10)","text":""},{"location":"changelog/#core_5","title":"Core","text":""},{"location":"changelog/#bots_5","title":"Bots","text":""},{"location":"changelog/#experts_5","title":"Experts","text":""},{"location":"changelog/#documentation_4","title":"Documentation","text":""},{"location":"changelog/#known-issues_4","title":"Known issues","text":"

See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).

"},{"location":"changelog/#301-2021-09-02","title":"3.0.1 (2021-09-02)","text":""},{"location":"changelog/#configuration_2","title":"Configuration","text":""},{"location":"changelog/#core_6","title":"Core","text":""},{"location":"changelog/#bots_6","title":"Bots","text":""},{"location":"changelog/#collectors_4","title":"Collectors","text":""},{"location":"changelog/#parsers_4","title":"Parsers","text":""},{"location":"changelog/#experts_6","title":"Experts","text":""},{"location":"changelog/#outputs_4","title":"Outputs","text":""},{"location":"changelog/#documentation_5","title":"Documentation","text":""},{"location":"changelog/#packaging_3","title":"Packaging","text":""},{"location":"changelog/#tests_4","title":"Tests","text":""},{"location":"changelog/#tools_4","title":"Tools","text":""},{"location":"changelog/#known-issues_5","title":"Known issues","text":"

See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).

"},{"location":"changelog/#300-2021-07-02","title":"3.0.0 (2021-07-02)","text":""},{"location":"changelog/#configuration_3","title":"Configuration","text":""},{"location":"changelog/#core_7","title":"Core","text":""},{"location":"changelog/#development_4","title":"Development","text":""},{"location":"changelog/#data-format_2","title":"Data Format","text":"

The IntelMQ Data Harmonization (\"DHO\") is renamed to IntelMQ Data Format (\"IDF\"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810). Update allowed classification fields to version 1.3 (2021-05-18) (by Sebastian Wagner, fixes #1409, #1476). - The taxonomy abusive content has been renamed to abusive-content. - The taxonomy information content security has been renamed to information-content-security. - The validation of type unauthorised-information-access has been fixed, a bug prevented the use of it. - The validation of type unauthorised-information-modification has been fixed, a bug prevented the use of it. - The type leak has been renamed to data-leak. - The type dropzone has been removed. Taxonomy other with type other and identifier dropzone can be used instead. Ongoing discussion in the RSIT WG. - The taxonomy intrusion attempts has been renamed to intrusion-attempts. - For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409): - The type compromised has been renamed to system-compromise. - The type unauthorized-command has been merged into system-compromise. - The type unauthorized-login has been merged into system-compromise. - The type backdoor has been merged into system-compromise (PR#1995 by Sebastian Wagner, addresses #1409). - The type defacement has been merged into taxonomy information-content-security, type unauthorised-information-modification (PR#1994 by Sebastian Wagner, addresses #1409). - The taxonomy information gathering has been rename to information-gathering. - The taxonomy malicious code has been renamed to malicious-code. - The type c2server has been renamed to c2-server. - The type malware has been integrated into infected-system and malware-distribution, respectively (PR#1917 by Sebastian Wagner addresses #1409). - The type ransomware has been integrated into infected-system. - The type dga domain has been moved to the taxonomy other renamed dga-domain (PR#1992 by Sebastian Wagner fixes #1613). - For the taxonomy 'availability', the type misconfiguration is new. - For the taxonomy 'other', the type unknown has been renamed to undetermined. - For the taxonomy 'vulnerable': - The type vulnerable client has been renamed to vulnerable-system. - The type vulnerable service has been renamed to vulnerable-system.

"},{"location":"changelog/#bots_7","title":"Bots","text":""},{"location":"changelog/#collectors_5","title":"Collectors","text":""},{"location":"changelog/#parsers_5","title":"Parsers","text":""},{"location":"changelog/#experts_7","title":"Experts","text":""},{"location":"changelog/#outputs_5","title":"Outputs","text":""},{"location":"changelog/#documentation_6","title":"Documentation","text":""},{"location":"changelog/#packaging_4","title":"Packaging","text":""},{"location":"changelog/#tests_5","title":"Tests","text":""},{"location":"changelog/#tools_5","title":"Tools","text":""},{"location":"changelog/#contrib_3","title":"Contrib","text":""},{"location":"changelog/#known-issues_6","title":"Known issues","text":""},{"location":"changelog/#233-2021-05-31","title":"2.3.3 (2021-05-31)","text":""},{"location":"changelog/#core_8","title":"Core","text":""},{"location":"changelog/#bots_8","title":"Bots","text":""},{"location":"changelog/#parsers_6","title":"Parsers","text":""},{"location":"changelog/#experts_8","title":"Experts","text":""},{"location":"changelog/#outputs_6","title":"Outputs","text":""},{"location":"changelog/#documentation_7","title":"Documentation","text":""},{"location":"changelog/#tests_6","title":"Tests","text":""},{"location":"changelog/#tools_6","title":"Tools","text":""},{"location":"changelog/#known-issues_7","title":"Known issues","text":""},{"location":"changelog/#232-2021-04-27","title":"2.3.2 (2021-04-27)","text":""},{"location":"changelog/#core_9","title":"Core","text":""},{"location":"changelog/#bots_9","title":"Bots","text":""},{"location":"changelog/#collectors_6","title":"Collectors","text":""},{"location":"changelog/#parsers_7","title":"Parsers","text":""},{"location":"changelog/#experts_9","title":"Experts","text":""},{"location":"changelog/#outputs_7","title":"Outputs","text":""},{"location":"changelog/#documentation_8","title":"Documentation","text":""},{"location":"changelog/#tests_7","title":"Tests","text":""},{"location":"changelog/#known-issues_8","title":"Known issues","text":""},{"location":"changelog/#231-2021-03-25","title":"2.3.1 (2021-03-25)","text":""},{"location":"changelog/#configuration_4","title":"Configuration","text":""},{"location":"changelog/#core_10","title":"Core","text":""},{"location":"changelog/#bots_10","title":"Bots","text":""},{"location":"changelog/#collectors_7","title":"Collectors","text":""},{"location":"changelog/#parsers_8","title":"Parsers","text":""},{"location":"changelog/#documentation_9","title":"Documentation","text":""},{"location":"changelog/#tests_8","title":"Tests","text":""},{"location":"changelog/#tools_7","title":"Tools","text":""},{"location":"changelog/#known-issues_9","title":"Known issues","text":""},{"location":"changelog/#230-2021-03-04","title":"2.3.0 (2021-03-04)","text":"

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.

"},{"location":"changelog/#configuration_5","title":"Configuration","text":""},{"location":"changelog/#core_11","title":"Core","text":""},{"location":"changelog/#development_5","title":"Development","text":""},{"location":"changelog/#bots_11","title":"Bots","text":""},{"location":"changelog/#collectors_8","title":"Collectors","text":""},{"location":"changelog/#parsers_9","title":"Parsers","text":""},{"location":"changelog/#experts_10","title":"Experts","text":""},{"location":"changelog/#outputs_8","title":"Outputs","text":""},{"location":"changelog/#documentation_10","title":"Documentation","text":""},{"location":"changelog/#packaging_5","title":"Packaging","text":""},{"location":"changelog/#tests_9","title":"Tests","text":""},{"location":"changelog/#tools_8","title":"Tools","text":""},{"location":"changelog/#contrib_4","title":"Contrib","text":""},{"location":"changelog/#known-issues_10","title":"Known issues","text":""},{"location":"changelog/#223-2020-12-23","title":"2.2.3 (2020-12-23)","text":""},{"location":"changelog/#documentation_11","title":"Documentation","text":""},{"location":"changelog/#harmonization","title":"Harmonization","text":""},{"location":"changelog/#bots_12","title":"Bots","text":""},{"location":"changelog/#collectors_9","title":"Collectors","text":""},{"location":"changelog/#parsers_10","title":"Parsers","text":""},{"location":"changelog/#experts_11","title":"Experts","text":""},{"location":"changelog/#tests_10","title":"Tests","text":""},{"location":"changelog/#known-issues_11","title":"Known issues","text":""},{"location":"changelog/#222-2020-10-28","title":"2.2.2 (2020-10-28)","text":""},{"location":"changelog/#core_12","title":"Core","text":""},{"location":"changelog/#bots_13","title":"Bots","text":""},{"location":"changelog/#parsers_11","title":"Parsers","text":""},{"location":"changelog/#experts_12","title":"Experts","text":""},{"location":"changelog/#documentation_12","title":"Documentation","text":""},{"location":"changelog/#packaging_6","title":"Packaging","text":""},{"location":"changelog/#tests_11","title":"Tests","text":""},{"location":"changelog/#tools_9","title":"Tools","text":""},{"location":"changelog/#contrib_5","title":"Contrib","text":""},{"location":"changelog/#known-issues_12","title":"Known issues","text":""},{"location":"changelog/#221-2020-07-30","title":"2.2.1 (2020-07-30)","text":""},{"location":"changelog/#core_13","title":"Core","text":""},{"location":"changelog/#development_6","title":"Development","text":""},{"location":"changelog/#bots_14","title":"Bots","text":""},{"location":"changelog/#collectors_10","title":"Collectors","text":""},{"location":"changelog/#parsers_12","title":"Parsers","text":""},{"location":"changelog/#experts_13","title":"Experts","text":""},{"location":"changelog/#outputs_9","title":"Outputs","text":""},{"location":"changelog/#documentation_13","title":"Documentation","text":""},{"location":"changelog/#tests_12","title":"Tests","text":""},{"location":"changelog/#tools_10","title":"Tools","text":""},{"location":"changelog/#contrib_6","title":"Contrib","text":""},{"location":"changelog/#known-issues_13","title":"Known issues","text":""},{"location":"changelog/#220-2020-06-18","title":"2.2.0 (2020-06-18)","text":"

Dropped support for Python 3.4.

"},{"location":"changelog/#core_14","title":"Core","text":""},{"location":"changelog/#bots_15","title":"Bots","text":""},{"location":"changelog/#collectors_11","title":"Collectors","text":""},{"location":"changelog/#parsers_13","title":"Parsers","text":""},{"location":"changelog/#experts_14","title":"Experts","text":""},{"location":"changelog/#outputs_10","title":"Outputs","text":""},{"location":"changelog/#documentation_14","title":"Documentation","text":""},{"location":"changelog/#packaging_7","title":"Packaging","text":""},{"location":"changelog/#tests_13","title":"Tests","text":""},{"location":"changelog/#tools_11","title":"Tools","text":""},{"location":"changelog/#contrib_7","title":"Contrib","text":""},{"location":"changelog/#known-issues_14","title":"Known issues","text":""},{"location":"changelog/#213-2020-05-26","title":"2.1.3 (2020-05-26)","text":""},{"location":"changelog/#requirements","title":"Requirements","text":""},{"location":"changelog/#core_15","title":"Core","text":""},{"location":"changelog/#harmonization_1","title":"Harmonization","text":""},{"location":"changelog/#bots_16","title":"Bots","text":""},{"location":"changelog/#collectors_12","title":"Collectors","text":""},{"location":"changelog/#parsers_14","title":"Parsers","text":""},{"location":"changelog/#experts_15","title":"Experts","text":""},{"location":"changelog/#outputs_11","title":"Outputs","text":""},{"location":"changelog/#documentation_15","title":"Documentation","text":""},{"location":"changelog/#packaging_8","title":"Packaging","text":""},{"location":"changelog/#tests_14","title":"Tests","text":""},{"location":"changelog/#tools_12","title":"Tools","text":""},{"location":"changelog/#contrib_8","title":"Contrib","text":""},{"location":"changelog/#known-issues_15","title":"Known issues","text":""},{"location":"changelog/#212-2020-01-28","title":"2.1.2 (2020-01-28)","text":""},{"location":"changelog/#core_16","title":"Core","text":""},{"location":"changelog/#bots_17","title":"Bots","text":""},{"location":"changelog/#collectors_13","title":"Collectors","text":""},{"location":"changelog/#parsers_15","title":"Parsers","text":""},{"location":"changelog/#experts_16","title":"Experts","text":""},{"location":"changelog/#outputs_12","title":"Outputs","text":""},{"location":"changelog/#documentation_16","title":"Documentation","text":""},{"location":"changelog/#packaging_9","title":"Packaging","text":""},{"location":"changelog/#tests_15","title":"Tests","text":""},{"location":"changelog/#tools_13","title":"Tools","text":""},{"location":"changelog/#known-issues_16","title":"Known issues","text":""},{"location":"changelog/#211-2019-11-11","title":"2.1.1 (2019-11-11)","text":""},{"location":"changelog/#configuration_6","title":"Configuration","text":""},{"location":"changelog/#core_17","title":"Core","text":""},{"location":"changelog/#bots_18","title":"Bots","text":""},{"location":"changelog/#parsers_16","title":"Parsers","text":""},{"location":"changelog/#experts_17","title":"Experts","text":""},{"location":"changelog/#outputs_13","title":"Outputs","text":""},{"location":"changelog/#documentation_17","title":"Documentation","text":""},{"location":"changelog/#tests_16","title":"Tests","text":""},{"location":"changelog/#tools_14","title":"Tools","text":""},{"location":"changelog/#known-issues_17","title":"Known issues","text":""},{"location":"changelog/#210-2019-10-15","title":"2.1.0 (2019-10-15)","text":""},{"location":"changelog/#core_18","title":"Core","text":""},{"location":"changelog/#harmonization_2","title":"Harmonization","text":""},{"location":"changelog/#bots_19","title":"Bots","text":""},{"location":"changelog/#collectors_14","title":"Collectors","text":""},{"location":"changelog/#parsers_17","title":"Parsers","text":""},{"location":"changelog/#experts_18","title":"Experts","text":""},{"location":"changelog/#outputs_14","title":"Outputs","text":""},{"location":"changelog/#documentation_18","title":"Documentation","text":""},{"location":"changelog/#tests_17","title":"Tests","text":""},{"location":"changelog/#tools_15","title":"Tools","text":""},{"location":"changelog/#contrib_9","title":"Contrib","text":""},{"location":"changelog/#known-issues_18","title":"Known issues","text":""},{"location":"changelog/#202-2019-10-14","title":"2.0.2 (2019-10-14)","text":""},{"location":"changelog/#core_19","title":"Core","text":""},{"location":"changelog/#bots_20","title":"Bots","text":""},{"location":"changelog/#collectors_15","title":"Collectors","text":""},{"location":"changelog/#parsers_18","title":"Parsers","text":""},{"location":"changelog/#experts_19","title":"Experts","text":""},{"location":"changelog/#outputs_15","title":"Outputs","text":""},{"location":"changelog/#packaging_10","title":"Packaging","text":""},{"location":"changelog/#tests_18","title":"Tests","text":""},{"location":"changelog/#tools_16","title":"Tools","text":""},{"location":"changelog/#contrib_10","title":"Contrib","text":""},{"location":"changelog/#known-issues_19","title":"Known issues","text":""},{"location":"changelog/#201-2019-08-23","title":"2.0.1 (2019-08-23)","text":""},{"location":"changelog/#core_20","title":"Core","text":""},{"location":"changelog/#development_7","title":"Development","text":""},{"location":"changelog/#harmonization_3","title":"Harmonization","text":""},{"location":"changelog/#bots_21","title":"Bots","text":""},{"location":"changelog/#collectors_16","title":"Collectors","text":""},{"location":"changelog/#parsers_19","title":"Parsers","text":""},{"location":"changelog/#experts_20","title":"Experts","text":""},{"location":"changelog/#outputs_16","title":"Outputs","text":""},{"location":"changelog/#documentation_19","title":"Documentation","text":""},{"location":"changelog/#packaging_11","title":"Packaging","text":""},{"location":"changelog/#tests_19","title":"Tests","text":""},{"location":"changelog/#tools_17","title":"Tools","text":""},{"location":"changelog/#contrib_11","title":"Contrib","text":""},{"location":"changelog/#known-issues_20","title":"Known issues","text":""},{"location":"changelog/#200-2019-05-22","title":"2.0.0 (2019-05-22)","text":"

See also the changelog for 2.0.0.beta1 below.

"},{"location":"changelog/#configurations","title":"Configurations","text":""},{"location":"changelog/#core_21","title":"Core","text":""},{"location":"changelog/#development_8","title":"Development","text":""},{"location":"changelog/#harmonization_4","title":"Harmonization","text":""},{"location":"changelog/#bots_22","title":"Bots","text":""},{"location":"changelog/#parsers_20","title":"Parsers","text":""},{"location":"changelog/#experts_21","title":"Experts","text":""},{"location":"changelog/#outputs_17","title":"Outputs","text":""},{"location":"changelog/#packaging_12","title":"Packaging","text":""},{"location":"changelog/#tests_20","title":"Tests","text":""},{"location":"changelog/#tools_18","title":"Tools","text":""},{"location":"changelog/#known-issues_21","title":"Known issues","text":""},{"location":"changelog/#200beta1-2019-04-10","title":"2.0.0.beta1 (2019-04-10)","text":"

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

"},{"location":"changelog/#removals-of-deprecated-code","title":"Removals of deprecated code:","text":""},{"location":"changelog/#core_22","title":"Core","text":""},{"location":"changelog/#development_9","title":"Development","text":""},{"location":"changelog/#bots_23","title":"Bots","text":""},{"location":"changelog/#collectors_17","title":"Collectors","text":""},{"location":"changelog/#parsers_21","title":"Parsers","text":""},{"location":"changelog/#experts_22","title":"Experts","text":""},{"location":"changelog/#outputs_18","title":"Outputs","text":""},{"location":"changelog/#documentation_20","title":"Documentation","text":""},{"location":"changelog/#tests_21","title":"Tests","text":""},{"location":"changelog/#tools_19","title":"Tools","text":""},{"location":"changelog/#contrib_12","title":"Contrib","text":""},{"location":"changelog/#known-issues_22","title":"Known issues","text":""},{"location":"changelog/#112-2019-03-25","title":"1.1.2 (2019-03-25)","text":""},{"location":"changelog/#core_23","title":"Core","text":""},{"location":"changelog/#harmonization_5","title":"Harmonization","text":""},{"location":"changelog/#bots_24","title":"Bots","text":""},{"location":"changelog/#collectors_18","title":"Collectors","text":""},{"location":"changelog/#parsers_22","title":"Parsers","text":""},{"location":"changelog/#experts_23","title":"Experts","text":""},{"location":"changelog/#outputs_19","title":"Outputs","text":""},{"location":"changelog/#documentation_21","title":"Documentation","text":""},{"location":"changelog/#packaging_13","title":"Packaging","text":""},{"location":"changelog/#tests_22","title":"Tests","text":""},{"location":"changelog/#tools_20","title":"Tools","text":""},{"location":"changelog/#known-issues_23","title":"Known issues","text":""},{"location":"changelog/#111-2019-01-15","title":"1.1.1 (2019-01-15)","text":""},{"location":"changelog/#core_24","title":"Core","text":""},{"location":"changelog/#default-configuration","title":"Default configuration","text":""},{"location":"changelog/#development_10","title":"Development","text":""},{"location":"changelog/#harmonization_6","title":"Harmonization","text":"

Update allowed classification fields to 2018-09-26 version (#802, #1350, #1380). New values for classification.type are per taxonomy: - Taxonomy 'intrusions': - \"application-compromise\" - \"burglary\" - \"privileged-account-compromise\" - \"unprivileged-account-compromise\" - Taxonomy 'fraud': - \"copyright\" - \"masquerade\" - \"unauthorized-use-of-resources\" - Taxonomy 'information content security': - \"data-loss\" - Taxonomy 'vulnerable': - \"ddos-amplifier\" - \"information-disclosure\" - \"potentially-unwanted-accessible\" - \"vulnerable-system\" - \"weak-crypto\" - Taxonomy 'availability': - \"dos\" - \"outage\" - \"sabotage\" - Taxonomy 'abusive-content': - \"harmful-speech\" - \"violence\" - Taxonomy 'malicious code': - \"malware-distribution\" - Taxonomy 'information-gathering': - \"social-engineering\" - \"sniffing\" - Taxonomy 'information content security': - \"Unauthorised-information-access\" - \"Unauthorised-information-modification\"

"},{"location":"changelog/#bots_25","title":"Bots","text":""},{"location":"changelog/#collectors_19","title":"Collectors","text":""},{"location":"changelog/#parsers_23","title":"Parsers","text":""},{"location":"changelog/#experts_24","title":"Experts","text":""},{"location":"changelog/#outputs_20","title":"Outputs","text":""},{"location":"changelog/#documentation_22","title":"Documentation","text":""},{"location":"changelog/#packaging_14","title":"Packaging","text":""},{"location":"changelog/#tests_23","title":"Tests","text":""},{"location":"changelog/#tools_21","title":"Tools","text":""},{"location":"changelog/#contrib_13","title":"Contrib","text":""},{"location":"changelog/#known-issues_24","title":"Known issues","text":""},{"location":"changelog/#110-2018-09-05","title":"1.1.0 (2018-09-05)","text":""},{"location":"changelog/#tools_22","title":"Tools","text":""},{"location":"changelog/#intelmqctl","title":"intelmqctl","text":""},{"location":"changelog/#contrib_14","title":"Contrib","text":""},{"location":"changelog/#core_25","title":"Core","text":""},{"location":"changelog/#bots_26","title":"Bots","text":""},{"location":"changelog/#collectors_20","title":"Collectors","text":""},{"location":"changelog/#parsers_24","title":"Parsers","text":""},{"location":"changelog/#experts_25","title":"Experts","text":""},{"location":"changelog/#outputs_21","title":"Outputs","text":""},{"location":"changelog/#harmonization_7","title":"Harmonization","text":""},{"location":"changelog/#requirements_1","title":"Requirements","text":""},{"location":"changelog/#documentation_23","title":"Documentation","text":""},{"location":"changelog/#packaging_15","title":"Packaging","text":""},{"location":"changelog/#tests_24","title":"Tests","text":""},{"location":"changelog/#known-bugs","title":"Known bugs","text":""},{"location":"changelog/#106-bugfix-release-2018-08-31","title":"1.0.6 Bugfix release (2018-08-31)","text":""},{"location":"changelog/#bots_27","title":"Bots","text":""},{"location":"changelog/#collectors_21","title":"Collectors","text":""},{"location":"changelog/#parsers_25","title":"Parsers","text":""},{"location":"changelog/#experts_26","title":"Experts","text":""},{"location":"changelog/#outputs_22","title":"Outputs","text":""},{"location":"changelog/#documentation_24","title":"Documentation","text":""},{"location":"changelog/#packaging_16","title":"Packaging","text":""},{"location":"changelog/#tests_25","title":"Tests","text":""},{"location":"changelog/#tools_23","title":"Tools","text":""},{"location":"changelog/#contrib_15","title":"Contrib","text":""},{"location":"changelog/#known-issues_25","title":"Known issues","text":""},{"location":"changelog/#105-bugfix-release-2018-06-21","title":"1.0.5 Bugfix release (2018-06-21)","text":""},{"location":"changelog/#core_26","title":"Core","text":""},{"location":"changelog/#harmonization_8","title":"Harmonization","text":""},{"location":"changelog/#bots_28","title":"Bots","text":""},{"location":"changelog/#collectors_22","title":"Collectors","text":""},{"location":"changelog/#parsers_26","title":"Parsers","text":""},{"location":"changelog/#experts_27","title":"Experts","text":""},{"location":"changelog/#tests_26","title":"Tests","text":""},{"location":"changelog/#tools_24","title":"Tools","text":""},{"location":"changelog/#known-issues_26","title":"Known issues","text":"

no known issues

"},{"location":"changelog/#104-bugfix-release-2018-04-20","title":"1.0.4 Bugfix release (2018-04-20)","text":""},{"location":"changelog/#core_27","title":"Core","text":""},{"location":"changelog/#bots_29","title":"Bots","text":""},{"location":"changelog/#parsers_27","title":"Parsers","text":""},{"location":"changelog/#experts_28","title":"Experts","text":""},{"location":"changelog/#tools_25","title":"Tools","text":""},{"location":"changelog/#tests_27","title":"Tests","text":""},{"location":"changelog/#packaging_17","title":"Packaging","text":""},{"location":"changelog/#known-issues_27","title":"Known issues","text":""},{"location":"changelog/#103-bugfix-release-2018-02-05","title":"1.0.3 Bugfix release (2018-02-05)","text":""},{"location":"changelog/#contrib_16","title":"Contrib","text":""},{"location":"changelog/#core_28","title":"Core","text":""},{"location":"changelog/#harmonization_9","title":"Harmonization","text":""},{"location":"changelog/#bots_30","title":"Bots","text":""},{"location":"changelog/#collectors_23","title":"Collectors","text":""},{"location":"changelog/#parsers_28","title":"Parsers","text":""},{"location":"changelog/#experts_29","title":"Experts","text":""},{"location":"changelog/#outputs_23","title":"Outputs","text":""},{"location":"changelog/#documentation_25","title":"Documentation","text":""},{"location":"changelog/#tools_26","title":"Tools","text":""},{"location":"changelog/#tests_28","title":"Tests","text":""},{"location":"changelog/#packaging_18","title":"Packaging","text":""},{"location":"changelog/#known-issues_28","title":"Known issues","text":""},{"location":"changelog/#102-bugfix-release-2017-11-09","title":"1.0.2 Bugfix release (2017-11-09)","text":""},{"location":"changelog/#core_29","title":"Core","text":""},{"location":"changelog/#bots_31","title":"Bots","text":""},{"location":"changelog/#packaging_19","title":"Packaging","text":""},{"location":"changelog/#documentation_26","title":"Documentation","text":""},{"location":"changelog/#101-bugfix-release-2017-08-30","title":"1.0.1 Bugfix release (2017-08-30)","text":""},{"location":"changelog/#documentation_27","title":"Documentation","text":""},{"location":"changelog/#bots_32","title":"Bots","text":""},{"location":"changelog/#core_30","title":"Core","text":""},{"location":"changelog/#tools_27","title":"Tools","text":""},{"location":"changelog/#100-stable-release-2017-08-04","title":"1.0.0 Stable release (2017-08-04)","text":""},{"location":"changelog/#core_31","title":"Core","text":""},{"location":"changelog/#harmonization_10","title":"Harmonization","text":""},{"location":"changelog/#bots_33","title":"Bots","text":""},{"location":"changelog/#100rc1-release-candidate-2017-07-05","title":"1.0.0.rc1 Release candidate (2017-07-05)","text":""},{"location":"changelog/#core_32","title":"Core","text":""},{"location":"changelog/#development_11","title":"Development","text":""},{"location":"changelog/#documentation_28","title":"Documentation","text":""},{"location":"changelog/#bots_34","title":"Bots","text":""},{"location":"changelog/#collectors_24","title":"Collectors","text":""},{"location":"changelog/#parsers_29","title":"Parsers","text":""},{"location":"changelog/#experts_30","title":"Experts","text":""},{"location":"changelog/#v100dev8-beta-release-2017-06-14","title":"v1.0.0.dev8 Beta release (2017-06-14)","text":""},{"location":"changelog/#general-changes","title":"General changes","text":""},{"location":"changelog/#configuration_7","title":"Configuration","text":""},{"location":"changelog/#documentation_29","title":"Documentation","text":""},{"location":"changelog/#tools_28","title":"Tools","text":""},{"location":"changelog/#core_33","title":"Core","text":""},{"location":"changelog/#bots_35","title":"Bots","text":""},{"location":"changelog/#harmonization_11","title":"Harmonization","text":""},{"location":"changelog/#v100dev7-beta-release-2017-05-09","title":"v1.0.0.dev7 Beta release (2017-05-09)","text":""},{"location":"changelog/#documentation_30","title":"Documentation","text":""},{"location":"changelog/#bots_36","title":"Bots","text":""},{"location":"changelog/#collectors_25","title":"Collectors","text":""},{"location":"changelog/#parsers_30","title":"Parsers","text":""},{"location":"changelog/#experts_31","title":"Experts","text":""},{"location":"changelog/#harmonization_12","title":"Harmonization","text":""},{"location":"changelog/#v100dev6-beta-release-2017-01-11","title":"v1.0.0.dev6 Beta release (2017-01-11)","text":"

Changes between 0.9 and 1.0.0.dev6

"},{"location":"changelog/#general-changes_1","title":"General changes","text":""},{"location":"changelog/#tools_29","title":"Tools","text":""},{"location":"changelog/#bots_37","title":"Bots","text":""},{"location":"changelog/#collectors_26","title":"Collectors","text":""},{"location":"changelog/#parsers_31","title":"Parsers","text":""},{"location":"changelog/#experts_32","title":"Experts","text":""},{"location":"changelog/#outputs_24","title":"Outputs","text":""},{"location":"changelog/#bug-fixes","title":"Bug fixes","text":""},{"location":"changelog/#other-enhancements-and-changes","title":"Other enhancements and changes","text":""},{"location":"changelog/#configuration_8","title":"Configuration","text":""},{"location":"changelog/#harmonization_13","title":"Harmonization","text":""},{"location":"changelog/#most-important-changes","title":"Most important changes:","text":""},{"location":"changelog/#known-issues_29","title":"Known issues","text":""},{"location":"changelog/#contrib_17","title":"Contrib","text":""},{"location":"changelog/#20160618","title":"2016/06/18","text":""},{"location":"changelog/#20150603-aaron","title":"2015/06/03 (aaron)","text":""},{"location":"community/","title":"Community","text":""},{"location":"community/#intelmq-organizational-structure","title":"IntelMQ Organizational Structure","text":"

The central IntelMQ components are maintained by multiple people and organizations in the IntelMQ community. Please note that some components of the IntelMQ Universe can have a different project governance, but all are part of the IntelMQ universe and community.

"},{"location":"community/#intelmq-enhancement-proposals-iep","title":"IntelMQ Enhancement Proposals (IEP)","text":"

Major changes, including architecture, strategy and the internal data format, require so-called IEPs, IntelMQ Enhancement Proposals. Their name is based on the famous \"PEPs\" of Python.

IEPs are collected in the separate IEP Repository.

"},{"location":"community/#code-reviews-and-merging","title":"Code-Reviews and Merging","text":"

Every line of code checked in for the IntelMQ Core, is checked by at least one trusted developer (excluding the author of the changes) of the IntelMQ community. Afterwards, the code can be merged. Currently, these three contributors, have the permission to push and merging code to IntelMQ Core, Manager and API:

Additionally, these people significantly contributed to IntelMQ:

"},{"location":"community/#short-history","title":"Short history","text":"

In 2013 and 2014 Aaron Kaplan (back then working at CERT.at) was researching ways to improve the automation of handling and distributing (IT security) incident reports across a whole country as part of the job of a national CERT. We would get many notifications of vulnerable systems, hacked systems, phishing domains, etc etc. The amount of reports we were getting required an automated solution. Back then, Aaron and a couple of other people looked at a tool called \"Abusehelper\". There was an open source version of Abusehelper, but it was deemed quite complex and complicated at that time.

Frustration with this tool led to discussions amongst multiple CERTs.

The idea and overall concept of an free, truly open source, simple (KISS principle! Keep it simple, stupid) community owned and maintained, extendible software for automated incident handling was born at an meeting of several European CSIRTs in Heraklion, Greece, in 2014. Following the event, Tom\u00e1s Lima \"SYNchroACK\" (working at CERT.pt back then) created IntelMQ from scratch. IntelMQ was born on June 24th, 2014. A major support came from CERT.pt at this early stage. Aaron Kaplan (CERT.at until 2020) engaged in the long-term advancement and from 2015 on, CERT.at took the burden of the maintenance and development (Sebastian Wagner 2015-2021 at CERT.at). From 2016 onward, CERT.at started projects, initiated and lead by Aaron Kaplan, receiving CEFF-funding from the European Union to support IntelMQ's development. IntelMQ became a software component of the EU-funded MeliCERTes framework for CSIRTs. In 2020, IntelMQ's organizational structure and architectural development gained new thrive by the newly founded Board and the start of the IEP process, creating more structure and more transparency in the IntelMQ community's decisions.

"},{"location":"help/","title":"Help","text":""},{"location":"help/#getting-help","title":"Getting help","text":"

In case you are lost, you need assistance or something is not discussed in this guide, you can ask the community for help. To be most efficient in seeking help, please describe your problem or question with all necessary information, for example:

Please report any errors and suggest improvements via issues. Thank you!

"},{"location":"help/#github","title":"GitHub","text":"

GitHub offers a discussion platform where you can ask questions and seek assistance.

To report bugs, GitHub issues are the ideal place to do so. Every IntelMQ component has it's own repository on GitHub, with a separate Issue tracker.

To participate on GitHub, you first need to create an account on the platform.

"},{"location":"help/#mailing-list","title":"Mailing list","text":"

The most traditional way is to ask your question, make a proposal or discuss a topic on the mailing IntelMQ Users mailing list. You need to subscribe to the mailing list before posting, but the archive is publicly available: IntelMQ Users Archive.

"},{"location":"help/#assistance","title":"Assistance","text":"

If your organisation is a member of the CSIRTs Network, you are eligible for support in the MeliCERTes project. You can also ask on for individual support, some members offer support, including, but not limited to:

"},{"location":"overview/","title":"Overview","text":""},{"location":"overview/#overview","title":"Overview","text":"

The complete IntelMQ universe consists of the following components:

"},{"location":"overview/#intelmq","title":"IntelMQ","text":"

This project contains the core functionality.

The Core includes all the components required for processing data feeds. This includes the bots, configuration, pipeline, the internal data format, management tools etc.

\u2192 Repository: IntelMQ

"},{"location":"overview/#intelmq-api","title":"IntelMQ API","text":"

This is an extension of IntelMQ providing hug based REST API for remote management.

\u2192 Repository: IntelMQ API

"},{"location":"overview/#intelmq-manager","title":"IntelMQ Manager","text":"

The Manager is the most known software and can be seen as the face of IntelMQ. It's goal is to provide an intuitive web interface to allow non-programmers to specify the data flow in IntelMQ.

\u2192 Repository: IntelMQ Manager

"},{"location":"overview/#additional-tools","title":"Additional tools","text":"

Here you can find a list of additional tools. If you think something is missing, please let us know!

Unless stated otherwise, the tools are maintained by the IntelMQ community.

"},{"location":"overview/#intelmq-webinput-csv","title":"IntelMQ Webinput CSV","text":"

A web-based interface to ingest CSV data into IntelMQ with on-line validation and live feedback.

This interface allows inserting \"one-shot\" data feeds into IntelMQ without the need to configure bots in IntelMQ.

Developed and maintained by CERT.at.

\u2192 Repository: intelmq-webinput-csv

"},{"location":"overview/#intelmq-mailgen","title":"IntelMQ Mailgen","text":"

A solution allowing an IntelMQ setup with a complex contact database, managed by a web interface and sending out aggregated email reports. In different words: To send grouped notifications to network owners using SMTP.

Developed and maintained by Intevation, initially funded by BSI.

It consists of the following three components, which can also be used on their own.

"},{"location":"overview/#intelmq-certbund-contact","title":"IntelMQ CertBUND Contact","text":"

The certbund-contact consists of two IntelMQ expert bots, which fetch and process the information from the contact database, and scripts to import RIPE data into the contact database. Based on user-defined rules, the experts determine to which contact the event is to be sent to, and which e-mail template and attachment format to use.

\u2192 Repository: intelmq-certbund-contact

"},{"location":"overview/#intelmq-fody","title":"IntelMQ Fody","text":"

Fody is a web based interface for Mailgen. It allows to read and edit contacts, query sent mails (tickets) and call up data from the PostgreSQL database.

It can also be used to just query the database without using Mailgen.

\u2192 Repository: intelmq-fody

\u2192 Repository: intelmq-fody-backend

"},{"location":"overview/#intelmq-mailgen_1","title":"intelmq-mailgen","text":"

Sends emails with grouped event data to the contacts determined by the certbund-contact. Mails can be encrypted with PGP.

\u2192 Repository: intelmq-mailgen

"},{"location":"overview/#constituency-portal-tuency","title":"\"Constituency Portal\" tuency","text":"

A web application helping CERTs to enable members of their constituency to self-administrate how they get warnings related to their network objects (IP addresses, IP ranges, autonomous systems, domains). tuency is developed by Intevation for CERT.at.

If features organizational hierarchies, contact roles, self-administration and network objects per organization (Autonomous systems, network ranges, (sub)domains, RIPE organization handles). A network object claiming and approval process prevents abuse. An hierarchical rule-system on the network objects allow fine-grained settings. The tagging system for contacts and organization complement the contact-management features of the portal. Authentication is based on keycloak, which enables the re-use of the user accounts in the portal. The integrated API enables IntelMQ to query the portal for the right abuse contact and notification settings with the intelmq.bots.experts.tuency.expert expert bot.

\u2192 Repository: tuency

"},{"location":"overview/#constituency-portal-do-portal-deprecated","title":"\"Constituency Portal\" do-portal (deprecated)","text":"

Warning

The do-portal is deprecated and succeeded by tuency.

A contact portal with organizational hierarchies, role functionality and network objects based on RIPE, allows self-administration by the contacts. Can be queried from IntelMQ and integrates the stats-portal.

Originally developed by CERT-EU, then adapted by CERT.at.

\u2192 Repository: do-portal

"},{"location":"overview/#stats-portal","title":"Stats Portal","text":"

A Grafana-based statistics portal for the eventdb{.interpreted-text role=\"doc\"}. Can be integrated into do-portal. It uses aggregated data to serve statistical data quickly.

\u2192 Repository: stats-portal

"},{"location":"overview/#malware-name-mapping","title":"Malware Name Mapping","text":"

A mapping for malware names of different feeds with different names to a common family name.

\u2192 Repository: malware_name_mapping

"},{"location":"overview/#intelmq-docker","title":"IntelMQ-Docker","text":"

A repository with tools for IntelMQ docker instance.

Developed and maintained by CERT.at.

\u2192 Repository: intelmq-docker

"},{"location":"overview/#useful-scripts","title":"Useful scripts","text":"

The list of useful scripts contributed to the IntelMQ universe can be found in the main repository.

\u2192 Repository: intelmq/contrib

"},{"location":"security/","title":"Security","text":""},{"location":"security/#found-a-security-issue","title":"Found a security issue?","text":"

In case you find security-relevant bugs in IntelMQ, please contact team@cert.at. More information including the PGP key can be found on CERT.at's website.

"},{"location":"admin/beta-features/","title":"Beta Features","text":""},{"location":"admin/beta-features/#beta-features","title":"Beta Features","text":""},{"location":"admin/beta-features/#using-supervisor-as-a-process-manager","title":"Using Supervisor as a Process Manager","text":"

Warning

Do not use it in production environments yet! It has not been tested thoroughly yet.

Supervisor is process manager written in Python. The main advantage is that it take care about processes, so if bot process exit with failure (exit code different than 0), supervisor try to run it again. Another advantage is that it not require writing PID files.

This was tested on Ubuntu 18.04.

Install supervisor. supervisor_twiddler is extension for supervisor, that makes possible to create process dynamically. (Ubuntu supervisor package is currently based on Python 2, so supervisor_twiddler must be installed with Python 2 pip.)

apt install supervisor python-pip\npip install supervisor_twiddler\n

Create default config /etc/supervisor/conf.d/intelmq.conf and restart supervisor service:

[rpcinterface:twiddler]\nsupervisor.rpcinterface_factory=supervisor_twiddler.rpcinterface:make_twiddler_rpcinterface\n\n[group:intelmq]\n

Change IntelMQ process manager in the global configuration:

process_manager: supervisor\n

After this it is possible to manage bots like before with intelmqctl command.

"},{"location":"admin/beta-features/#using-amqp-message-broker","title":"Using AMQP Message Broker","text":"

Starting with IntelMQ 1.2 the AMQP protocol is supported as message queue. To use it, install a broker, for example RabbitMQ. The configuration and the differences are outlined here. Keep in mind that it is slower, but has better monitoring capabilities and is more stable. The AMQP support is considered beta, so small problems might occur. So far, only RabbitMQ as broker has been tested.

You can change the broker for single bots (set the parameters in the runtime configuration per bot) or for the whole botnet (using the global configuration).

You need to set the parameter source_pipeline_broker/destination_pipeline_broker to amqp. There are more parameters available:

Bug

This section of the documentation is currently incomplete and will be updated later.

destination_pipeline_broker

(required, string) \"amqp\"

destination_pipeline_host

() (default: '127.0.0.1')

destination_pipeline_port

() (default: 5672)

destination_pipeline_username

()

destination_pipeline_password

()

destination_pipeline_socket_timeout

() (default: no timeout)

destination_pipeline_amqp_exchange

() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: '').

destination_pipeline_amqp_virtual_host

() (default: '/')

source_pipeline_host

() (default: '127.0.0.1')

source_pipeline_port

() (default: 5672)

source_pipeline_username

()

source_pipeline_password

()

source_pipeline_socket_timeout

() (default: no timeout)

source_pipeline_amqp_exchange

() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: ['']).

source_pipeline_amqp_virtual_host

() (default: '/')

intelmqctl_rabbitmq_monitoring_url

() string, see below (default: \"http://{host}:15672\")

For getting the queue sizes, intelmqctl needs to connect to the monitoring interface of RabbitMQ. If the monitoring interface is not available under http://{host}:15672 you can manually set using the parameter intelmqctl_rabbitmq_monitoring_url. In a RabbitMQ's default configuration you might not provide a user account, as by default the administrator (guest:guest) allows full access from localhost. If you create a separate user account, make sure to add the tag \"monitoring\" to it, otherwise IntelMQ can't fetch the queue sizes.

Setting the statistics (and cache) parameters is necessary when the local redis is running under a non-default host/port. If this is the case, you can set them explicitly:

statistics_database

() 3

statistics_host

() \"127.0.0.1\"

statistics_password

() null

statistics_port

() 6379

"},{"location":"admin/beta-features/#multithreading","title":"Multithreading","text":"

First of all: Do not use it in production environments yet! There are a few bugs, see below

Since IntelMQ 2.0 it is possible to provide the following runtime parameter:

instances_threads

Set it to a non-zero integer, then this number of worker threads will be spawn. This is useful if bots often wait for system resources or if network-based lookups are a bottleneck.

However, there are currently a few cavecats:

"},{"location":"admin/common-problems/","title":"Common Problems","text":""},{"location":"admin/common-problems/#common-problems","title":"Common Problems","text":""},{"location":"admin/common-problems/#intelmq","title":"IntelMQ","text":""},{"location":"admin/common-problems/#permission-denied-when-using-redis-unix-socket","title":"Permission denied when using Redis Unix socket","text":"

If you get an error like this:

intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.',)\n

Make sure the intelmq user as sufficient permissions for the socket.

In /etc/redis/redis.conf (or wherever your configuration is), check the permissions and set it for example to group-writeable:

unixsocketperm 770\n

And add the user intelmq to the redis-group:

usermod -aG redis intelmq\n
"},{"location":"admin/common-problems/#my-bots-died-on-startup-with-no-errors-logged","title":"My bot(s) died on startup with no errors logged","text":"

Rather than starting your bot(s) with intelmqctl start, try intelmqctl run [bot]. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.

"},{"location":"admin/common-problems/#orphaned-queues","title":"Orphaned Queues","text":"

This section has been moved to the Management Guide.

"},{"location":"admin/common-problems/#multithreading-is-not-available-for-this-bot","title":"Multithreading is not available for this bot","text":"

Multithreading is not available for some bots and AMQP broker is necessary. Possible reasons why a certain bot or a setup does not support Multithreading include:

If you think this mapping is wrong, please report a bug.

"},{"location":"admin/common-problems/#intelmq-api","title":"IntelMQ API","text":""},{"location":"admin/common-problems/#intelmqctlerror","title":"IntelMQCtlError","text":"

If the command is not configured correctly, you will see exceptions on startup like this:

intelmq_manager.runctl.IntelMQCtlError: <ERROR_MESSAGE>\n

This means the intelmqctl command could not be executed as a subprocess. The <ERROR_MESSAGE> should indicate why.

"},{"location":"admin/common-problems/#access-denied-authentication-required-please-provide-valid-token-verification-credentials","title":"Access Denied / Authentication Required \"Please provide valid Token verification credentials\"","text":"

If you see the IntelMQ Manager interface and menu, but the API calls to the back-end querying configuration and status of IntelMQ fail with \"Access Denied\" or \"Authentication Required: Please provide valid Token verification credentials\" errors, you are maybe not logged in while the API requires authentication.

By default, the API requires authentication. Create user accounts and login with them or - if you have other protection means in place - deactivate the authentication requirement by removing or renaming the session_store parameter in the configuration.

"},{"location":"admin/common-problems/#internal-server-error","title":"Internal Server Error","text":"

There can be various reasons for internal server errors. You need to look at the error log of your web server, for example /var/log/apache2/error.log or /var/log/httpd/error_log for Apache 2. It could be that the sudo-setup is not functional, the configuration file or session database file can not be read or written or other errors in regards to the execution of the API program.

"},{"location":"admin/common-problems/#can-i-just-install-it-from-the-debrpm-packages-while-installing-intelmq-from-a-different-source","title":"Can I just install it from the deb/rpm packages while installing IntelMQ from a different source?","text":"

Yes, you can install the API and the Manager from the deb/rpm repositories, and install your IntelMQ from a somewhere else, e.g. a local repository. However, knowledge about Python and system administration experience is recommended if you do so.

The packages install IntelMQ to /usr/lib/python3*/site-packages/intelmq/. Installing with pip results in /usr/local/lib/python3*/site-packages/intelmq/ (and some other accompaning resources) which overrides the installation in /usr/lib/. You probably need to adapt the configuration parameter intelmq_ctl_cmd to the /usr/local/bin/intelmqctl executable and some other tweaks.

"},{"location":"admin/common-problems/#sqlite3operationalerror-attempt-to-write-a-readonly-database","title":"sqlite3.OperationalError: attempt to write a readonly database","text":"

SQLite does not only need write access to the database itself, but also the folder the database file is located in. Please check that the webserver has write permissions to the folder the session file is located in.

"},{"location":"admin/faq/","title":"FAQ","text":""},{"location":"admin/faq/#frequently-asked-questions","title":"Frequently asked questions","text":""},{"location":"admin/faq/#how-can-i-improve-the-speed","title":"How can I improve the speed?","text":"

In most cases the bottlenecks are look-up experts. In these cases you can easily use the integrated load balancing features.

"},{"location":"admin/faq/#multithreading","title":"Multithreading","text":"

When using the AMQP broker, you can make use of Multi-threading. See the multithreading section.

"},{"location":"admin/faq/#classic-load-balancing-multiprocessing","title":"\"Classic\" load-balancing (Multiprocessing)","text":"

Before Multithreading was available in IntelMQ, and in case you use Redis as broker, the only way to do load balancing involves more work. Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter load_balance to true for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.

True Multiprocessing is not available in IntelMQ. See also this discussion on a possible enhanced load balancing <186>.

"},{"location":"admin/faq/#other-options","title":"Other options","text":"

For any bottleneck based on (online) lookups, optimize the lookup itself and if possible use local databases.

It is also possible to use multiple servers to spread the workload. To get the messages from one system to the other you can either directly connect to the other's pipeline or use a fast exchange mechanism such as the TCP Collector/Output (make sure to secure the network by other means).

"},{"location":"admin/faq/#removing-raw-data-for-higher-performance-and-less-space-usage","title":"Removing raw data for higher performance and less space usage","text":"

If you do not need the raw data, you can safely remove it. For events (after parsers), it keeps the original data, eg. a line of a CSV file. In reports it keeps the actual data to be parsed, so don't delete the raw field in Reports - between collectors and parsers.

The raw data consumes about 50% - 30% of the messages' size. The size of course depends on how many additional data you add to it and how much data the report includes. Dropping it, will improve the speed as less data needs to be transferred and processed at each step.

In a bot

You can do this for example by using the Field Reducer Expert. The configuration could be:

Other solutions are the Modify bot and the Sieve bot. The last one is a good choice if you already use it and you only need to add the command:

remove raw\n

In the database

In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.

To remove the raw data for a events table of a PostgreSQL database, you can use something like:

UPDATE events SET raw = NULL WHERE \"time.source\" < '2018-07-01';\n

If the database is big, make sure only update small parts of the database by using an appropriate WHERE clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The id column can also be used instead of the source's time.

Another way of reducing the raw-data from the database is described in the EventDB documentation: eventdb_raws_table.

"},{"location":"admin/faq/#how-to-uninstall","title":"How to Uninstall","text":"

If you installed intelmq with native packages: Use the package management tool to remove the package intelmq. These tools do not remove configuration by default.

If you installed manually via pip (note that this also deletes all configuration and possibly data):

pip3 uninstall intelmq\nrm -r /opt/intelmq\n
"},{"location":"admin/hardware-requirements/","title":"Hardware Requirements","text":""},{"location":"admin/hardware-requirements/#hardware-requirements","title":"Hardware Requirements","text":"

Do you ask yourself how much RAM do you need to give your new IntelMQ virtual machine?

The honest answer is simple and pointless: It depends ;)

"},{"location":"admin/hardware-requirements/#intelmq-and-the-messaging-queue-broker","title":"IntelMQ and the messaging queue (broker)","text":"

IntelMQ uses a messaging queue to move the messages between the bots. All bot instances can only process one message at a time, therefore all other messages need to wait in the queue. As not all bots are equally fast, the messages will naturally \"queue up\" before the slower ones. Further, parsers produce many events with just one message (the report) as input.

The following estimations assume Redis as messaging broker which is the default for IntelMQ. When RabbitMQ is used, the required resources will differ, and RabbitMQ can handle system overload and therefore a shortage of memory.

As Redis stores all data in memory, the data which is processed at any point in time must fit there, including overheads. Please note that IntelMQ does neither store nor cache any input data. These estimates therefore only relate to the processing step, not the storage.

For a minimal system, these requirements suffice:

Depending on your data input, you will need the twentiethfold of the input data size as memory for processing.

When using Redis persistence, you will additionally need twice as much memory for Redis.

"},{"location":"admin/hardware-requirements/#disk-space","title":"Disk space","text":"

Disk space is only relevant if you save your data to a file, which is not recommended for production setups, and only useful for testing and evaluation.

Do not forget to rotate your logs or use syslog, especially if you use the logging level \"DEBUG\". logrotate is in use by default for all installation with deb/rpm packages. When other means of installation are used (pip, manual), configure log rotation manually. See logging configuration.

"},{"location":"admin/hardware-requirements/#background-on-memory","title":"Background on memory","text":"

For experimentation, we used multiple Shadowserver Poodle reports for demonstration purpose, totaling in 120 MB of data. All numbers are estimates and are rounded. In memory, the report data requires 160 MB. After parsing, the memory usage increases to 850 MB in total, as every data line is stored as JSON, with additional information plus the original data encoded in Base 64. The further processing steps depend on the configuration, but you can estimate that caches (for lookups and deduplication) and other added information cause an additional size increase of about 2x. Once a dataset finished processing in IntelMQ, it is no longer stored in memory. Therefore, the memory is only needed to catch high load.

The above numbers result in a factor of 14 for input data size vs. memory required by Redis. Assuming some overhead and memory for the bots' processes, a factor of 20 seems sensible.

To reduce the amount of required memory and disk size, you can optionally remove the raw data field, see this section in the FAQ.

"},{"location":"admin/hardware-requirements/#additional-components","title":"Additional components","text":"

If some of the optional components are in use, they can add additional hardware requirements.

Those components do not add relevant requirements:

"},{"location":"admin/hardware-requirements/#database","title":"Database","text":"

When storing data in databases (such as MongoDB, PostgreSQL, ElasticSearch), it is recommended to do this on separate machines for operational reasons. Using a different machine results in a separation of stream processing to data storage and allows for a specialized system optimization for both use-cases.

"},{"location":"admin/hardware-requirements/#intelmq-cb-mailgen","title":"IntelMQ cb mailgen","text":"

While the Fody backend and frontend do not have significant requirements, the RIPE import tool of the certbund-contact requires about 8 GB of memory as of March 2021.

"},{"location":"admin/intro/","title":"Intro","text":""},{"location":"admin/intro/#intro","title":"Intro","text":"

This guide provides instructions on how to install, configure and manage IntelMQ and it's components.

IntelMQ uses a message broker such as Redis. This is required for IntelMQ to run.

IntelMQ doesn't handle long term storage of processed Events beyond writing to a file. However it provides connectors (output bots) for writing events to various database systems and log collectors. It is recommended to configure such system to preserve processed events.

"},{"location":"admin/intro/#base-requirements","title":"Base Requirements","text":"

The following instructions assume the following requirements. Python versions >= 3.7 are supported.

Supported and recommended operating systems are:

Other distributions which are (most probably) supported include AlmaLinux, CentOS, Fedora, FreeBSD 12, RHEL and RockyLinux.

A short guide on hardware requirements can be found on the page Hardware Requirements.

"},{"location":"admin/upgrade/","title":"Upgrade","text":""},{"location":"admin/upgrade/#upgrade-instructions","title":"Upgrade instructions","text":"

In order to upgrade your IntelMQ installation it is recommended to follow these five steps:

"},{"location":"admin/upgrade/#1-read-newsmd","title":"1. Read NEWS.md","text":"

Read the NEWS.md file to look for things you need to have a look at.

"},{"location":"admin/upgrade/#2-stop-intelmq-and-create-a-backup","title":"2. Stop IntelMQ and create a backup","text":" 
sudo cp -R /opt/intelmq /opt/intelmq-backup\n
"},{"location":"admin/upgrade/#3-upgrade-intelmq","title":"3. Upgrade IntelMQ","text":"

Before upgrading, check that your setup is clean and there are no events in the queues:

intelmqctl check\nintelmqctl list queues -q\n

The upgrade depends on how you installed IntelMQ.

"},{"location":"admin/upgrade/#linux-packages","title":"Linux Packages","text":"

Use your system's package manager.

"},{"location":"admin/upgrade/#pypi","title":"PyPi","text":"
pip install -U --no-deps intelmq\nsudo intelmqsetup\n

Using --no-deps will not upgrade dependencies, which would probably overwrite the system's libraries. Remove this option to also upgrade dependencies.

"},{"location":"admin/upgrade/#docker","title":"Docker","text":"

You can check out all current versions on our DockerHub.

docker pull certat/intelmq-full:latest\n\ndocker pull certat/intelmq-nginx:latest\n

Alternatively you can use docker-compose:

docker-compose pull\n

You can check the current versions from intelmq & intelmq-manager & intelmq-api via git commit ref.

The Version format for each included item is key=value and they are saparated via ,. I. e. IntelMQ=ab12cd34f,IntelMQ-API=xy65z23.

docker inspect --format '{{ index .Config.Labels \"org.opencontainers.image.version\" }}' intelmq-full:latest\n

Now restart your container, if you're using docker-compose you simply run:

docker-compose down\n

If you don't use docker-compose, you can restart a single container using:

docker ps | grep certat\n\ndocker restart CONTAINER_ID\n
"},{"location":"admin/upgrade/#source-repository","title":"Source repository","text":"

If you have an editable installation, refer to the instructions in the /dev/guide.

Update the repository depending on your setup (e.g. [git pull origin master]).

And run the installation again:

pip install .\nsudo intelmqsetup\n

For editable installations (development only), run [pip install -e .] instead.

"},{"location":"admin/upgrade/#4-upgrade-configuration-and-check-the-installation","title":"4. Upgrade configuration and check the installation","text":"

Go through NEWS.md and apply necessary adaptions to your setup. If you have adapted IntelMQ's code, also read the CHANGELOG.md.

Check your installation and configuration to detect any problems:

intelmqctl upgrade-config\nintelmqctl check\n

intelmqctl upgrade-config supports upgrades from one IntelMQ version to the succeeding. If you skip one or more IntelMQ versions, some automatic upgrades may not work and manual intervention may be necessary.

"},{"location":"admin/upgrade/#5-start-intelmq","title":"5. Start IntelMQ","text":"
intelmqctl start\n
"},{"location":"admin/configuration/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/configuration/intelmq-api/#configuring-intelmq-api","title":"Configuring IntelMQ API","text":"

Depending on your setup you might have to install sudo to make it possible for the intelmq-api to run the intelmq command as the user-account usually used to run intelmq (which is also often called intelmq).

intelmq-api is configured using a configuration file in json format. intelmq-api tries to load the configuration file from /etc/intelmq/api-config.json and ${PREFIX}/etc/intelmq/api-config.json, but you can override the path setting the environment variable INTELMQ_API_CONFIG. (When using Apache, you can do this by modifying the Apache configuration file shipped with intelmq-api, the file contains an example)

When running the API using hug, you can set the environment variable like this:

INTELMQ_API_CONFIG=/etc/intelmq/api-config.json hug -m intelmq_api.serve\n

The default configuration which is shipped with the packages is also listed here for reference:

{\n    \"intelmq_ctl_cmd\": [\"sudo\", \"-u\", \"intelmq\", \"intelmqctl\"],\n    \"allowed_path\": \"/opt/intelmq/var/lib/bots/\",\n    \"session_store\": \"/etc/intelmq/api-session.sqlite\",\n    \"session_duration\": 86400,\n    \"allow_origins\": [\"*\"]\n}\n

On Debian based systems, the default path for the session_store is /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi, because the Debian package uses the Debian packaging tools to manage the database file.

The following configuration options are available:

"},{"location":"admin/configuration/intelmq-api/#permissions","title":"Permissions","text":"

intelmq-api tries to write a couple of configuration files in the ${PREFIX}/etc/intelmq directory - this is only possible if you set the permissions accordingly, given that intelmq-api runs under a different user. The user the API run as also needs write access to the folder the session_store is located in, otherwise there will be an error accessing the session data. If you\\'re using the default Apache 2 setup, you might want to set the group of the files to www-data and give it write permissions (chmod -R g+w <directoryname>). In addition to that, the intelmq-manager tries to store the bot positions via the API into the file ${PREFIX}/etc/intelmq/manager/positions.conf. You should therefore create the folder ${PREFIX}/etc/intelmq/manager and the file positions.conf in it.

"},{"location":"admin/configuration/intelmq-api/#adding-a-user","title":"Adding a user","text":"

If you enable the session_store you will have to create user accounts to be able to access the API functionality. You can do this using intelmq-api-adduser:

intelmq-api-adduser --user <username> --password <password>\n
"},{"location":"admin/configuration/intelmq-api/#a-note-on-selinux","title":"A note on SELinux","text":"

On systems with SELinux enabled, the API will fail to call intelmqctl. Therefore, SELinux needs to be disabled:

setenforce 0\n

We welcome contributions to provide SELinux policies.

"},{"location":"admin/configuration/intelmq-manager/","title":"IntelMQ Manager","text":""},{"location":"admin/configuration/intelmq-manager/#configuring-intelmq-manager","title":"Configuring IntelMQ Manager","text":"

In the file /usr/share/intelmq-manager/html/js/vars.js set ROOT to the URL of your intelmq-api installation - by default that's on the same host as intelmq-manager.

"},{"location":"admin/configuration/intelmq-manager/#configuration-paths","title":"Configuration Paths","text":"

The IntelMQ Manager queries the configuration file paths and directory names from intelmqctl and therefore any global environment variables (if set) are effective in the Manager too. The interface for this query is intelmqctl debug --get-paths, the result is also shown in the /about.html page of your IntelMQ Manager installation.

"},{"location":"admin/configuration/intelmq-manager/#csp-headers","title":"CSP Headers","text":"

It is recommended to set these two headers for all requests:

Content-Security-Policy: script-src 'self'\nX-Content-Security-Policy: script-src 'self'\n
"},{"location":"admin/configuration/intelmq-manager/#security-considerations","title":"Security considerations","text":"

Never ever run intelmq-manager on a public webserver without SSL and proper authentication!

The way the current version is written, anyone can send a POST request and change intelmq's configuration files via sending HTTP POST requests. Intelmq-manager will reject non JSON data but nevertheless, we don't want anyone to be able to reconfigure an intelmq installation.

Therefore you will need authentication and SSL. Authentication can be handled by the intelmq-api. Please refer to its documentation on how to enable authentication and setup accounts.

Never ever allow unencrypted, unauthenticated access to IntelMQ Manager!

"},{"location":"admin/configuration/intelmq-manager/#docker-security-headers","title":"Docker: Security headers","text":"

If you run our docker image in production, we recommend you to set security headers. You can do this by creating a new file called example_config/nginx/security.conf in the cloned intelmq-docker repository.

Write the following inside the configuration file, and change the http(s)://<your-domain> to your domain name.

server_tokens off; # turn off server_token, instead of nginx/13.2 now it will only show nginx\nadd_header X-Frame-Options SAMEORIGIN; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options\nadd_header X-Content-Type-Options nosniff; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options\nadd_header X-XSS-Protection \"1; mode=block\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection\nadd_header Content-Security-Policy \"script-src 'self' 'unsafe-inline' http(s)://<your-domain>; frame-src 'self' http(s)://<your-domain>; object-src 'self' http(s)://<your-domain>\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP\n

After you created the file, edit the docker-compose.yml and mount it to the nginx with

volumes:\n  - ./example_config/nginx/security.conf:/etc/nginx/conf.d/security.conf\n

IMPORTANT Mount the exact name & not the directory, because otherwise you would overwrite the whole directory and the other files would be gone inside the container.

"},{"location":"admin/configuration/intelmq/","title":"IntelMQ","text":""},{"location":"admin/configuration/intelmq/#configuring-intelmq","title":"Configuring IntelMQ","text":""},{"location":"admin/configuration/intelmq/#directories","title":"Directories","text":""},{"location":"admin/configuration/intelmq/#lsb","title":"LSB","text":"

If you installed the packages, standard Linux paths (LSB paths) are used:

Otherwise, the configuration directory is /opt/intelmq/etc/. Using the environment variable INTELMQ_ROOT_DIR allows setting any arbitrary root directory.

You can switch this by setting the environment variables INTELMQ_PATHS_NO_OPT and INTELMQ_PATHS_OPT, respectively.

The environment variable ROOT_DIR is meant to set an alternative root directory instead of /. This is primarily meant for package build environments an analogous to setuptool's --root parameter. Thus it is only used in LSB-mode.

"},{"location":"admin/configuration/intelmq/#environment-variables","title":"Environment Variables","text":"Name Type Description INTELMQ_PATHS_OPT INTELMQ_PATHS_NO_OPT INTELMQ_ROOT_DIR ROOT_DIR"},{"location":"admin/configuration/intelmq/#configuration-files","title":"Configuration Files","text":""},{"location":"admin/configuration/intelmq/#runtimeyaml","title":"runtime.yaml","text":"

This is the main configuration file. It uses YAML format since IntelMQ 3.0. It consists of two parts:

Warning

Comments in YAML are currently not preserved by IntelMQ (known bug #2003).

Example runtime.yaml configuration file is installed by the tool intelmqsetup. If this is not the case, make sure the program was run. It is shipped preconfigured with 4 collectors and parsers, 6 common experts and one output bot. The default collector and the parser handle data from malware domain list, the file output bot writes all data to one of these files (based on your installation):

The runtime.yaml configuration is divided into two sections:

Example configuration snippet:

global: # global configuration section\n  # ...\n  http_timeout_max_tries: 3\n  http_timeout_sec: 30\n  http_user_agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\n  http_verify_cert: true\n\nblocklistde-apache-collector: # individual bot configuration section\n  group: Collector\n  name: Blocklist.de Apache List\n  module: intelmq.bots.collectors.http.collector_http\n  description: Blocklist.de Apache Collector fetches all IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  parameters:\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    name: Blocklist.de Apache\n    rate_limit: 3600\n    http_verify_cert: false # overriding the global configuration for this particular bot\n
"},{"location":"admin/configuration/intelmq/#global-configuration","title":"Global Configuration","text":"

The global configuration parameters apply to all bots, however they can be overridden in the individual bot configuration.

"},{"location":"admin/configuration/intelmq/#logging","title":"Logging","text":"

The logging can be configured with the following parameters:

logging_handler

(required, string) Allowed values are file or syslog.

logging_level

(required, string) Allowed values are CRITICAL, ERROR, WARNING, INFO or DEBUG. Defines the system-wide log level that will be use by all bots and the intelmqctl tool. We recommend logging_level WARNING for production environments and INFO if you want more details. In any case, watch your free disk space!

logging_path

(required, string) When the logging_handler is file this parameter is used to set the logging directory for all the bots as well as the intelmqctl tool. Defaults to /opt/intelmq/var/log/ or /var/log/intelmq/ respectively.

logging_syslog

(required, string) When the logging_handler is syslog. Either a list with hostname and UDP port of syslog service, e.g. [\"localhost\", 514] or a device name/path. Defaults to /var/log.

"},{"location":"admin/configuration/intelmq/#log-rotation","title":"Log Rotation","text":"

To rotate the logs, you can use the standard Linux-tool logrotate. An example logrotate configuration is given in contrib/logrotate/ and delivered with all deb/rpm-packages. When not using logrotate, IntelMQ can rotate the logs itself, which is not enabled by default! You need to set both values.

logging_max_size

(optional, integer) Maximum number of bytes to be stored in one logfile before the file is rotated. Defaults to 0 (log rotation disabled).

logging_max_copies

(optional, integer) Maximum number of logfiles to keep. Compression is not supported. Default is unset.

Some information can as well be found in Python's documentation on the used RotatingFileHandler.

"},{"location":"admin/configuration/intelmq/#error-handling","title":"Error Handling","text":"

error_log_message

(required, boolean) Whether to write the message (Event/Report) to the log file in case of an error.

error_log_exception

(required, boolean) Whether to write an error exception to the log file in case of an error.

error_procedure

(required, string) Allowed values are stop or pass. In case of an error, this option defines the procedure that the bot will adopt. Use the following values:

error_max_retries

(required, integer) In case of an error, the bot will try to re-start processing the current message X times as defined by this option.

error_retry_delay

(required, integer) Defines the number of seconds to wait between subsequent re-tries in case of an error.

error_dump_message

(required, boolean) Specifies if the bot will write queued up messages to its dump file (use intelmqdump to re-insert the message).

If the path _on_error exists for a bot, the message is also sent to this queue, instead of (only) dumping the file if configured to do so.

"},{"location":"admin/configuration/intelmq/#miscellaneous","title":"Miscellaneous","text":"

load_balance

(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:

rate_limit

(required, integer) time interval (in seconds) between messages processing. int value.

ssl_ca_certificate

(optional, string) trusted CA certificate for IMAP connections (supported by some bots).

source_pipeline_broker

(optional, string) Allowed values are redis and amqp. Selects the message broker IntelMQ should use. As this parameter can be overridden by each bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance. Defaults to redis.

destination_pipeline_broker

(required, string) See source_pipeline_broker.

source_pipeline_host

(required, string) Hostname or path to Unix socket that the bot will use to connect and receive messages.

source_pipeline_port

(optional, integer) Broker port that the bot will use to connect and receive messages. Can be empty for Unix socket.

source_pipeline_password

(optional, string) Broker password that the bot will use to connect and receive messages. Can be null for unprotected broker.

source_pipeline_db

(required, integer) broker database that the bot will use to connect and receive messages (requirement from redis broker).

destination_pipeline_host

(optional, string) broker IP, FQDN or Unix socket that the bot will use to connect and send messages.

destination_pipeline_port

(optional, integer) broker port that the bot will use to connect and send messages. Can be empty for Unix socket.

destination_pipeline_password

(optional, string) broker password that the bot will use to connect and send messages. Can be null for unprotected broker.

destination_pipeline_db

(required, integer) broker database that the bot will use to connect and send messages (requirement from redis broker).

http_proxy

(optional, string) Proxy to use for HTTP.

https_proxy

(optional, string) Proxy to use for HTTPS.

http_user_agent

(optional, string) User-Agent to be used for HTTP requests.

http_verify_cert

(optional, boolean) Verify the TLS certificate of the server. Defaults to true.

"},{"location":"admin/configuration/intelmq/#individual-bot-configuration","title":"Individual Bot Configuration","text":"

Info

For the individual bot configuration please see the Bots document in the User Guide.

"},{"location":"admin/configuration/intelmq/#run-mode","title":"Run Mode","text":"

This sections provides more detailed explanation of the two run modes of the bots.

"},{"location":"admin/configuration/intelmq/#continuous","title":"Continuous","text":"

Most of the cases, bots will need to be configured as continuous run mode (the default) in order to have them always running and processing events. Usually, the types of bots that will require the continuous mode will be Parsers, Experts and Outputs. To do this, set run_mode to continuous in the runtime.yaml for the bot. Check the following example:

blocklistde-apache-parser:\n  name: Blocklist.de Parser\n  group: Parser\n  module: intelmq.bots.parsers.blocklistde.parser\n  description: Blocklist.DE Parser is the bot responsible to parse the report and sanitize the information.\n  enabled: false\n  run_mode: continuous\n  parameters: ...\n

You can now start the bot using the following command:

intelmqctl start blocklistde-apache-parser\n

Bots configured as continuous will never exit except if there is an error and the error handling configuration requires the bot to exit. See the Error Handling section for more details.

"},{"location":"admin/configuration/intelmq/#scheduled","title":"Scheduled","text":"

In many cases, it is useful to schedule a bot at a specific time (i.e. via cron(1)), for example to collect information from a website every day at midnight. To do this, set run_mode to scheduled in the runtime.yaml for the bot. Check out the following example:

blocklistde-apache-collector:\n  name: Generic URL Fetcher\n  group: Collector\n  module: intelmq.bots.collectors.http.collector_http\n  description: All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  enabled: false\n  run_mode: scheduled\n  parameters:\n    feed: Blocklist.de Apache\n    provider: Blocklist.de\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    ssl_client_certificate: null\n

You can schedule the bot with a crontab-entry like this:

0 0 * * * intelmqctl start blocklistde-apache-collector\n

Bots configured as scheduled will exit after the first successful run. Setting enabled to false will cause the bot to not start with intelmqctl start, but only with an explicit start, in this example intelmqctl start blocklistde-apache-collector.

"},{"location":"admin/configuration/intelmq/#additional-runtime-parameters","title":"Additional Runtime Parameters","text":"

Some of the parameters are deliberately skipped from the User Guide because they are configured via graphical user interface provided by the IntelMQ Manager. These parameters have to do with configuring the pipeline: defining how the data is exchanged between the bots. Using the IntelMQ Manager for this have many benefits as it guarantees that the configuration is correct upon saving.

However as an administrator you should be also familiar with the manual (and somewhat tedious) configuration. For each bot there are two parameters that need to be set:

source_queue

(optional, string) The name of the source queue from which the bot is going to processing data. Each bot has maximum one source queue (collector bots don't have any source queue as they fetch data from elsewhere). Defaults to the bot id appended with the string -queue.

Example: a bot with id example-bot will have a default source queue named example-bot-queue.

destination_queues

(optional, object) Bots can have multiple destination queues. Destination queues can also be grouped into named paths. There are two special path names _default and _on_error. The path _default is used if the path is not is specified by the bot itself (which is the most common case). In case of an error during the processing, the message will be sent to the _on_error path if specified (optional).

Only few of the bots (mostly expert bots with filtering capabilities) can take advantage of arbitrarily named paths. Some expert bots are capable of sending messages to paths, this feature is explained in their documentation, e.g. the Filter expert and the Sieve expert.

Example:

blocklistde-apache-collector:\n  # ...\n  parameters:\n    # ...\n    destination_queues:\n      _default:\n        - <first destination pipeline name>\n        - <second destination pipeline name>\n      _on_error:\n        - <optional first destination pipeline name in case of errors>\n        - <optional second destination pipeline name in case of errors>\n      other-path:\n        - <second destination pipeline name>\n        - <third destination pipeline name>\n
"},{"location":"admin/configuration/intelmq/#harmonizationconf","title":"harmonization.conf","text":"

This configuration is used to specify the fields for all message types. The harmonization library will load this configuration to check, during the message processing, if the values are compliant to the configured harmonization format. Usually, this configuration doesn't need any change. It is mostly maintained by the IntelMQ maintainers.

Template:

{\n  \"<message type>\": {\n    \"<field 1>\": {\n      \"description\": \"<field 1 description>\",\n      \"type\": \"<field value type>\"\n    },\n    \"<field 2>\": {\n      \"description\": \"<field 2 description>\",\n      \"type\": \"<field value type>\"\n    }\n  }\n}\n

Example:

{\n  \"event\": {\n    \"destination.asn\": {\n      \"description\": \"The autonomous system number from which originated the connection.\",\n      \"type\": \"Integer\"\n    },\n    \"destination.geolocation.cc\": {\n      \"description\": \"Country-Code according to ISO3166-1 alpha-2 for the destination IP.\",\n      \"regex\": \"^[a-zA-Z0-9]{2}$\",\n      \"type\": \"String\"\n    }\n  }\n}\n
"},{"location":"admin/database/elasticsearch/","title":"Elasticsearch","text":""},{"location":"admin/database/elasticsearch/#using-elasticsearch-as-a-database-for-intelmq","title":"Using Elasticsearch as a database for IntelMQ","text":"

If you wish to run IntelMQ with Elasticsearch or full ELK stack (Elasticsearch, Logstash, Kibana) it is entirely possible. This guide assumes the reader is familiar with basic configuration of ELK and does not aim to cover using ELK in general. It is based on the version 6.8.0 (ELK is a fast moving train therefore things might change). Assuming you have IntelMQ (and Redis) installation in place, lets dive in.

"},{"location":"admin/database/elasticsearch/#configuration-without-logstash","title":"Configuration without Logstash","text":"

This case involves two steps:

  1. Configure IntelMQ to output data directly into Elasticsearch.

  2. Configure Elasticsearch for ingesting the inserted data.

Bug

This section of the documentation is currently incomplete and will be updated later.

"},{"location":"admin/database/elasticsearch/#configuration-with-logstash","title":"Configuration with Logstash","text":"

This case involves three steps:

  1. Configuring IntelMQ to output data to Redis.

  2. Configure Logstash to collect data from Redis and insert them into Elasticsearch.

  3. Configure Elasticsearch for ingesting the inserted data.

Each step is described in detail in the following sections.

"},{"location":"admin/database/elasticsearch/#configuring-intelmq","title":"Configuring IntelMQ","text":"

In order to pass IntelMQ events to Logstash we will utilize already installed Redis. Add a new Redis Output Bot to your pipeline. As the minimum fill in the following parameters: bot-id, redis_server_ip (can be hostname) , redis_server_port, redis_password (if required, else set to empty!), redis_queue (name for the queue). It is recommended to use a different redis_db parameter than used by the IntelMQ (specified as source_pipeline_db , destination_pipeline_db and statistics_database).

Example values:

bot-id: redis-output\nredis_server_ip: 10.10.10.10\nredis_server_port: 6379\nredis_db: 4\nredis_queue: logstash-queue\n

Warning

You will not be able to monitor this redis queue via IntelMQ Manager.

"},{"location":"admin/database/elasticsearch/#configuring-logstash","title":"Configuring Logstash","text":"

Logstash defines pipelines as well. In the pipeline configuration of Logstash you need to specify where it should look for IntelMQ events, what to do with them and where to pass them.

"},{"location":"admin/database/elasticsearch/#input","title":"Input","text":"

This part describes how to receive data from Redis queue. See the example configuration and comments below:

input {\n  redis {\n    host => \"10.10.10.10\"\n    port => 6379\n    db => 4 \n    data_type => \"list\"\n    key => \"logstash-queue\"\n  }\n}\n

Tip

You can use environment variables for the Logstash configuration, for example host => \"${REDIS_HOST:10.10.10.10}\". The value will be taken from the environment variable $REDIS_HOST. If the environment variable is not set then the default value of 10.10.10.10 will be used instead.

"},{"location":"admin/database/elasticsearch/#filter-optional","title":"Filter (optional)","text":"

Before passing the data to the database you can apply certain changes. This is done with filters. See an example:

filter {\n  mutate {\n    lowercase => [\"source.geolocation.city\", \"classification.identifier\"]\n    remove_field => [\"__type\", \"@version\"]\n  }\n  date {\n    match => [\"time.observation\", \"ISO8601\"]\n  }\n}\n

Tip

It is recommended to use the date filter: generally we have two timestamp fields - time.source (provided by the feed source this can be understood as when the event happened; however it is not always present) and time.observation (when IntelMQ collected this event). Logstash also adds another field @timestamp with time of processing by Logstash. While it can be useful for debugging, I recommend to set the @timestamp to the same value as time.observation.

Warning

It is not recommended to apply any modifications to the data (within the mutate key) outside of the IntelMQ. All necessary modifications should be done only by appropriate IntelMQ bots. This example only demonstrates the possibility.

"},{"location":"admin/database/elasticsearch/#output","title":"Output","text":"

The pipeline also needs output, where we define our database (Elasticsearch). The simplest way of doing so is defining an output like this:

output {\n  elasticsearch {\n    hosts => [\"http://10.10.10.11:9200\", \"http://10.10.10.12:9200\"]\n    index => \"intelmq-%{+YYYY.MM}\"\n  }\n}\n

Tip

Authors experience, hardware equipment and the amount of events collected led to having a separate index for each month. This might not necessarily suit your needs, but it is a suggested option.

Warning

By default the ELK stack uses insecure HTTP. It is possible to setup Security for secure connections and basic user management. This is possible with the Basic (free) licence since versions 6.8.0 and 7.1.0.

"},{"location":"admin/database/elasticsearch/#configuring-elasticsearch","title":"Configuring Elasticsearch","text":"

Configuring Elasticsearch is entirely up to you and should be consulted with the official documentation. What you will most likely need is something called index template mappings. IntelMQ provides a tool for generating such mappings. See ElasticMapper Tool.

Danger

Default installation of Elasticsearch database allows anyone with cURL and connection capability to have administrative access to the database. Make sure you secure your toys!

"},{"location":"admin/database/mssql/","title":"MSSQL","text":""},{"location":"admin/database/mssql/#mssql","title":"MSSQL","text":"

For MSSQL support, the library pymssql>=2.2 is required.

To output data to MSSQL use SQL Output Bot with parameter engine set to mssql.

For more information see SQL Output Bot documentation page.

"},{"location":"admin/database/postgresql/","title":"PostgreSQL","text":""},{"location":"admin/database/postgresql/#using-postgresql-as-a-database-for-intelmq","title":"Using PostgreSQL as a database for IntelMQ","text":"

The EventDB is a database (usually PostgreSQL) that gets filled with with data from IntelMQ using the SQL Output Bot.

"},{"location":"admin/database/postgresql/#intelmq_psql_initdb","title":"intelmq_psql_initdb","text":"

IntelMQ comes with the intelmq_psql_initdb command line tool designed to help with creating the EventDB. It creates in the first line:

Having an events table as outlined in the SQL file, IntelMQ's SQL Output Bot can write all received events into this database table.

In addition, the script supports some additional features supporting use cases described later in this document:

For a full list of supported parameters, call the script help using -h parameter.

All elements of the generated SQL file can be adapted and extended before running the SQL file against a database, especially the indexes. Please review the generated script before applying.

Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions, refer to PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>.

"},{"location":"admin/database/postgresql/#eventdb-utilities","title":"EventDB Utilities","text":"

Some scripts related to the EventDB are located in the contrib/eventdb folder in the IntelMQ git repository.

"},{"location":"admin/database/postgresql/#apply-malware-name-mapping","title":"Apply Malware Name Mapping","text":"

The apply_mapping_eventdb.py script applies the malware name mapping to the EventDB. Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand. It queries the database for all distinct malware names with the taxonomy \"malicious-code\" and sets another column to the malware family name.

"},{"location":"admin/database/postgresql/#apply-domain-suffix","title":"Apply Domain Suffix","text":"

The apply_domain_suffix.py script writes the public domain suffix to the source.domain_suffix / destination.domain_suffix columns, extracted from source.fqdn / destination.fqdn.

"},{"location":"admin/database/postgresql/#usage","title":"Usage","text":"

The Python scripts can connect to a PostgreSQL server with an eventdb database and an events table. The command line arguments interface for both scripts are the same. See --help for more information:

apply_mapping_eventdb.py -h\napply_domain_suffix.py -h\n
"},{"location":"admin/database/postgresql/#postgresql-trigger","title":"PostgreSQL trigger","text":"

PostgreSQL trigger is a trigger keeping track of the oldest inserted/updated \"time.source\" data. This can be useful to (re-)generate statistics or aggregation data.

The SQL script can be executed in the database directly.

"},{"location":"admin/database/postgresql/#eventdb-statistics","title":"EventDB Statistics","text":"

The EventDB provides a great base for statistical analysis of the data.

The eventdb-stats repository contains a Python script that generates an HTML file and includes the Plotly JavaScript Open Source Graphing Library. By modifying the configuration file it is possible to configure various queries that are then displayed using graphs:

"},{"location":"admin/database/postgresql/#using-eventdb-with-timescale-db","title":"Using EventDB with Timescale DB","text":"

Timescale DB is a PostgreSQL extension to add time-series support, which is quite handy as you don't have to learn other syntaxes as you already know. You can use the SQL Queries as before, the extension will handle the rest. To see all limitations, please check the Timescale DB Documentation.

"},{"location":"admin/database/postgresql/#what-is-time-series","title":"What is time-series?","text":"

Time-series has been invented as traditional database design like relational or nosql are not made for time-based data. A big benefit of time-series instead of other database designs over a time-based search pattern is the performance. As IntelMQ uses data based upon time, this design is awesome & will give you a performance boost.

"},{"location":"admin/database/postgresql/#how-to-choose-the-time-column","title":"How to choose the time column?","text":"

To utilize the time-series, choose a column containing the right time. This is then used by you for manual queries and graphs, and also by the database itself for organizing the data.

An Event has two fields that can be used for this: time.source or time.observation. Depending on your needs (tracking when the event occurred or when it was detected, if different), choose one of them.

You can use the :ref:intelmq_psql_initdb tool to generate SQL schema valid for TimescaleDB by passing the partitioning key:

intelmq_psql_initdb --partition-key \"time.source\"\n
"},{"location":"admin/database/postgresql/#how-to-setup","title":"How to setup","text":"

Thanks to TimescaleDB its very easy to setup.

  1. Choose your preferred Timescale DB environment & follow the installation instructions. 2. Now lets create a hypertable, which is the timescale DB time-series structure. SELECT create_hypertable('', 'time.source');. 3. Now our hypertable is setup & timescaleDB takes care of the rest. You can perform queries as usual, for further information please check Timescale DB Documentation.
"},{"location":"admin/database/postgresql/#how-to-upgrade-from-my-existing-database","title":"How to upgrade from my existing database?","text":"

To update your existing database to use this awesome time-series feature, just follow the How to setup instruction. You can perform the hypertable command even on already existing databases. BUT there are some limitations from timescaleDB.

"},{"location":"admin/database/postgresql/#separating-raw-values-in-postgresql-using-view-and-trigger","title":"Separating raw values in PostgreSQL using view and trigger","text":"

In order to reduce the row size in the events table, the raw column's data can be separated from the other columns. While the raw-data is about 30-50% of the data row's size, it is not used in most database queries, as it serves only a backup functionality. Other possibilities to reduce or getting rid of this field are described in the FAQ, section faq-remove-raw-data.

The steps described here are best performed before the events table is filled with data, but can as well be done with existing data.

The approach requires four steps:

  1. An existing events table, see the first section of this document.
  2. Deleting or renaming the raw column of the events table.
  3. Creating a table raws which holds only the raw field of the events and linking both tables using the event_id.
  4. Creating the view v_events which joins the tables events and raws.
  5. Creating the function process_v_events_insert and INSERT trigger tr_events.

The last steps brings us several advantages:

The complete SQL script can be generated using the intelmq_psql_initdb. It does not cover step 2 to avoid accidental data loss - you need to do this step manually.

"},{"location":"admin/database/postgresql/#other-docs","title":"Other docs","text":"

You have two basic choices to run PostgreSQL:

  1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
  2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right connection parameters to each psql or client call.

Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.

PostgreSQL Version

Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [1].

If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone formatting string \"OF\" for date-times and the GiST index for the CIDR type. This may be useful depending on how you plan to use the events that this bot writes into the database.

How to install

Use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql

You need a PostgreSQL database-user to own the result database. The recommendation is to use the name intelmq . There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from setting up the expert/certbund_contact bot.)

Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system user postgres:

createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq\n

Create the new database:

createdb --encoding='utf-8' --owner=intelmq intelmq-events\n

(The encoding parameter should ensure the right encoding on platform where this is not the default.)

Now initialize it as database-user intelmq (in this example a network connection to localhost is used, so you would get to test if the user intelmq can authenticate):

psql -h localhost intelmq-events intelmq </tmp/initdb.sql\n

PostgreSQL and null characters

While null characters (0, not SQL \"NULL\") in TEXT and JSON/JSONB fields are valid, data containing null characters can cause troubles in some combinations of clients, servers and each settings. To prevent unhandled errors and data which can't be inserted into the database, all null characters are escaped (u0000) before insertion.

"},{"location":"admin/database/splunk/","title":"Splunk","text":""},{"location":"admin/database/splunk/#sending-intelmq-events-to-splunk","title":"Sending IntelMQ events to Splunk","text":"
  1. Go to Splunk and configure in order to be able to receive logs (intelmq events) to a TCP port
  2. Use TCP output bot and configure accordingly to the Splunk configuration that you applied.
"},{"location":"admin/database/sqlite/","title":"SQLite","text":""},{"location":"admin/database/sqlite/#sqlite","title":"SQLite","text":"

Similarly to PostgreSQL, you can use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql.

Create the new database (you can ignore all errors since SQLite doesn't know all SQL features generated for PostgreSQL):

sqlite3 your-db.db\nsqlite> .read /tmp/initdb.sql\n

Then, set the database parameter to the your-db.db file path.

To output data to SQLite use SQL Output Bot with parameter engine set to sqlite. For more information see SQL Output Bot documentation page.

"},{"location":"admin/installation/dockerhub/","title":"DockerHub","text":""},{"location":"admin/installation/dockerhub/#installation-from-dockerhub","title":"Installation from DockerHub","text":"

This guide provides instruction on how to install IntelMQ and it's components using Docker.

Warning

Docker installation is currently in Beta state and things might break. Consider this if you plan to use IntelMQ as a production level system.

Warning

Currently you can't manage your botnet via intelmqctl command line tool. You need to use IntelMQ-Manager currently!

The latest IntelMQ image is hosted on Docker Hub and the image build instructions are in our intelmq-docker repository.

Follow Docker Install and Docker-Compose Install instructions.

Before you start using docker-compose or any docker related tools, make sure docker is running:

# To start the docker daemon\nsystemctl start docker.service\n# To enable the docker daemon for the future\nsystemctl enable docker.service\n
"},{"location":"admin/installation/dockerhub/#docker-with-docker-compose","title":"Docker with docker-compose","text":"

Now we can download IntelMQ and start the containers. Navigate to your preferred installation directory and run the following commands:

git clone https://github.com/certat/intelmq-docker.git --recursive\ncd intelmq-docker\nsudo docker-compose pull\nsudo docker-compose up\n

Your installation should be successful now. You're now able to visit http://127.0.0.1:1337/ to access the intelmq-manager. You have to login with the username intelmq and the password intelmq, if you want to change the username or password, you can do this by adding the environment variables INTELMQ_API_USER for the username and INTELMQ_API_PASS for the password.

Note

If you get an Permission denied error, you should run chown -R $USER:$USER example_config

"},{"location":"admin/installation/dockerhub/#docker-without-docker-compose","title":"Docker without docker-compose","text":"

If not already installed, please install Docker.

Navigate to your preferred installation directory and run git clone https://github.com/certat/intelmq-docker.git --recursive.

You need to prepare some volumes & configs. Edit the left-side after -v, to change paths.

Change redis_host to a running redis-instance. Docker will resolve it automatically. All containers are connected using Docker Networks.

In order to work with your current infrastructure, you need to specify some environment variables

sudo docker pull redis:latest\n\nsudo docker pull certat/intelmq-full:latest\n\nsudo docker pull certat/intelmq-nginx:latest\n\nsudo docker network create intelmq-internal\n\nsudo docker run -v ~/intelmq/example_config/redis/redis.conf:/redis.conf \\\n                --network intelmq-internal \\\n                --name redis \\\n                redis:latest\n\nsudo docker run --network intelmq-internal \\\n                --name nginx \\\n                certat/intelmq-nginx:latest\n\nsudo docker run -e INTELMQ_IS_DOCKER=\"true\" \\\n                -e INTELMQ_SOURCE_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_DESTIONATION_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_HOST: redis \\\n                -e INTELMQ_SOURCE_PIPELINE_HOST: redis \\\n                -e INTELMQ_DESTINATION_PIPELINE_HOST: redis \\\n                -e INTELMQ_REDIS_CACHE_HOST: redis \\\n                -v $(pwd)/example_config/intelmq/etc/:/etc/intelmq/etc/ \\\n                -v $(pwd)/example_config/intelmq-api/config.json:/etc/intelmq/api-config.json \\\n                -v $(pwd)/intelmq_logs:/etc/intelmq/var/log \\\n                -v $(pwd)/intelmq_output:/etc/intelmq/var/lib/bots \\\n                -v ~/intelmq/lib:/etc/intelmq/var/lib \\\n                --network intelmq-internal \\\n                --name intelmq \\\n                certat/intelmq-full:latest\n

If you want to use another username and password for the intelmq-manager / api login, additionally add two new environment variables.

-e INTELMQ_API_USER: \"your username\"\n-e INTELMQ_API_PASS: \"your password\"\n
"},{"location":"admin/installation/linux-packages/","title":"Linux Package","text":""},{"location":"admin/installation/linux-packages/#installation-as-linux-package","title":"Installation as Linux package","text":"

This guide provides instructions on how to install IntelMQ and it's components from Linux distribution's package repository.

Note

Some bots may have additional dependencies which are mentioned in their own documentation.

"},{"location":"admin/installation/linux-packages/#supported-os","title":"Supported OS","text":"

Native packages are currently provided for the following Linux distributions:

"},{"location":"admin/installation/linux-packages/#debian-11-and-12","title":"Debian 11 and 12","text":"

Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):

echo \"deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_$(lsb_release -rs)/ /\" | sudo tee /etc/apt/sources.list.d/intelmq\ncurl -fsSL \"https://download.opensuse.org/repositories/home:sebix:intelmq/Debian_$(lsb_release -rs)/Release.key\" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null\nsudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n
"},{"location":"admin/installation/linux-packages/#opensuse-tumbleweed","title":"openSUSE Tumbleweed","text":"

Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):

zypper addrepo https://download.opensuse.org/repositories/home:sebix:intelmq/openSUSE_Tumbleweed/home:sebix:intelmq.repo\nzypper refresh\nzypper install intelmq intelmq-api intelmq-manager\n
"},{"location":"admin/installation/linux-packages/#ubuntu-2004-and-2204","title":"Ubuntu 20.04 and 22.04","text":"

For Ubuntu you must enable the Universe repository which provides community-maintained free and open-source software.

Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):

  1. Open the file /etc/apt/sources.list in an editor of your choice. Use sudo or the root user.

  2. Append universe to this line: 

    deb http://[...].archive.ubuntu.com/ubuntu/ focal main universe\n

  3. Update the list of available packages and install IntelMQ: 

    sudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n

"},{"location":"admin/installation/pypi/","title":"PyPI","text":""},{"location":"admin/installation/pypi/#installation-from-pypi","title":"Installation from PyPI","text":"

This guide provides instruction on how to install IntelMQ and it's components using the Python Package Index (PyPI) repository.

Note

Some bots may have additional dependencies which are mentioned in their own documentation.

"},{"location":"admin/installation/pypi/#installing-intelmq","title":"Installing IntelMQ","text":""},{"location":"admin/installation/pypi/#requirements","title":"Requirements","text":""},{"location":"admin/installation/pypi/#ubuntu-debian","title":"Ubuntu / Debian","text":"
apt install python3-pip python3-dnspython python3-psutil python3-redis python3-requests python3-termstyle python3-tz python3-dateutil redis-server bash-completion jq\n# optional dependencies\napt install python3-pymongo python3-psycopg2\n
"},{"location":"admin/installation/pypi/#opensuse","title":"openSUSE:","text":"
zypper install python3-dateutil python3-dnspython python3-psutil python3-redis python3-requests python3-python-termstyle redis bash-completion jq\n# optional dependencies\nzypper in python3-psycopg2 python3-pymongo\n
"},{"location":"admin/installation/pypi/#centos-8","title":"CentOS 8:","text":"
dnf install epel-release\ndnf install python3-dateutil python3-dns python3-pip python3-psutil python3-redis python3-requests redis bash-completion jq\n# optional dependencies\ndnf install python3-psycopg2 python3-pymongo\n
"},{"location":"admin/installation/pypi/#centos-7-rhel-7","title":"CentOS 7 / RHEL 7:","text":"

Warning

We no longer support already end-of-life Python 3.6, which is the last Python version officially packaged for CentOS 7. You can either use alternative Python source, or stay on the IntelMQ 3.0.2.

yum install epel-release\nyum install python36 python36-dns python36-requests python3-setuptools redis bash-completion jq\nyum install gcc gcc-c++ python36-devel\n# optional dependencies\nyum install python3-psycopg2\n
"},{"location":"admin/installation/pypi/#installation","title":"Installation","text":"

The default installation directory is /opt/intelmq/.

If you prefer to use Linux Standard Base (LSB) paths, set the following environment variable:

export INTELMQ_PATHS_NO_OPT=1\n

If you want to use custom installation directory, set the following environment variable:

export INTELMQ_ROOT_DIR=/my-installation-directory-path\n

Run the following commands to install IntelMQ. The provided tool intelmqsetup will create all the necessary directories and installs a default configuration for new setups. If you are using the LSB paths installation, change the --home-dir parameter to /var/lib/intelmq

sudo --preserve-env=INTELMQ_PATHS_NO_OPT,INTELMQ_ROOT_DIR -i\npip3 install intelmq\n[[ ! -z \"$INTELMQ_PATHS_NO_OPT\" ]] && export HOME_DIR=/var/lib/intelmq || export HOME_DIR=${INTELMQ_ROOT_DIR:-/opt/intelmq}\nuseradd --system --user-group --home-dir $HOME_DIR --shell /bin/bash intelmq\nintelmqsetup\n
"},{"location":"admin/installation/pypi/#installation-to-python-virtual-environment","title":"Installation to Python virtual environment","text":"
sudo mkdir -m 755 /opt/intelmq\nsudo useradd --system --user-group --home-dir /opt/intelmq --shell /bin/bash intelmq\nsudo chown intelmq:intelmq /opt/intelmq/\nsudo -u intelmq python3 -m venv /opt/intelmq/venv\nsudo -u intelmq /opt/intelmq/venv/bin/pip install intelmq intelmq-api intelmq-manager\nsudo /opt/intelmq/venv/bin/intelmqsetup\n
"},{"location":"admin/installation/pypi/#installing-intelmq-api-optional","title":"Installing IntelMQ API (optional)","text":"

The intelmq-api packages ships:

The value of ${PREFIX} depends on your environment and is something like /usr/local/lib/pythonX.Y/dist-packages/ (where X.Y is your Python version).

The virtualhost configuration file needs to be placed in the correct directory for your Apache 2 installation.

Don't forget to reload your webserver afterwards.

The api configuration file and the positions configuration file need to be placed in one of the following directories (based on your IntelMQ installation directory):

The sudoers configuration file should be placed in the /etc/sudoers.d/ directory and adapt the webserver username in this file. Set the file permissions to 0o440.

Afterwards continue with the section Permissions below.

IntelMQ 2.3.1 comes with a tool intelmqsetup which performs these set-up steps automatically. Please note that the tool is very new and may not detect all situations correctly. Please report us any bugs you are observing. The tools is idempotent, you can execute it multiple times.

"},{"location":"admin/installation/pypi/#installing-intelmq-manager-optional","title":"Installing IntelMQ Manager (optional)","text":"

To use the IntelMQ Manager web interface, it is required to have a working IntelMQ and IntelMQ API installation.

For installation via pip, the situation is more complex. The intelmq-manager package does not contain ready-to-use files, they need to be built locally. First, lets install the Manager itself:

pip3 install intelmq-manager\n

If your system uses wheel-packages, not the source distribution, you can use the intelmqsetup tool. intelmqsetup which performs these set-up steps automatically but it may not detect all situations correctly. If it finds intelmq-manager installed, calls its build routine is called. The files are placed in /usr/share/intelmq_manager/html, where the default Apache configuration expect it.

If your system used the dist-package or if you are using a local source, the tool may not do all required steps. To call the build routine manually, use intelmq-manager-build --output-dir your/preferred/output/directory/.

intelmq-manager ships with a default configuration for the Apache webserver (manager-apache.conf):

Alias /intelmq-manager /usr/share/intelmq_manager/html/\n\n<Directory /usr/share/intelmq_manager/html>\n    <IfModule mod_headers.c>\n    Header set Content-Security-Policy \"script-src 'self'\"\n    Header set X-Content-Security-Policy \"script-src 'self'\"\n    </IfModule>\n</Directory>\n

This file needs to be placed in the correct place for your Apache 2 installation.

"},{"location":"admin/integrations/cifv3/","title":"CIFv3","text":""},{"location":"admin/integrations/cifv3/#cifv3-integrations-in-intelmq","title":"CIFv3 integrations in IntelMQ","text":"

CIF creates an accessible indicator store. A REST API is exposed to interact with the store and quickly process/share indicators. CIFv3 can correlate indicators via the UUID attribute.

"},{"location":"admin/integrations/cifv3/#cif3-api-output","title":"CIF3 API Output","text":"

Can be used to submit indicators to a CIFv3 instance by using the CIFv3 API.

Look at the CIFv3 API Output Bot for more information.

"},{"location":"admin/integrations/misp/","title":"MISP","text":""},{"location":"admin/integrations/misp/#misp-integrations-in-intelmq","title":"MISP integrations in IntelMQ","text":"

While MISP and IntelMQ seem to solve similar problems in the first hindsight, their intentions and strengths differ significantly.

In a nutshell, MISP stores manually curated indicators (called attributes) grouped in events. An event can have an arbitrary number of attributes. MISP correlates these indicators with each other and can synchronize the data between multiple MISP instances.

On the other side, IntelMQ in it's essence (not considering the EventDB <eventdb>) has no state or database, but is stream-oriented. IntelMQ acts as a toolbox which can be configured as needed to automate processes of mass data with little or no human interaction At the end of the processing the data may land in some database or be sent to other systems.

Both systems do not intend to replace each other or do compete. They integrate seamless and combine each other enabling more use-cases and

"},{"location":"admin/integrations/misp/#misp-api-collector","title":"MISP API Collector","text":"

The MISP API Collector fetches data from MISP via the MISP API .

Look at the Bots documentation page for more information.

"},{"location":"admin/integrations/misp/#misp-expert","title":"MISP Expert","text":"

The MISP Expert searches MISP by using the MISP API for attributes/events matching the source.ip of the event. The MISP Attribute UUID and MISP Event ID of the newest attribute are added to the event.

Look at the Bots documentation page for more information.

"},{"location":"admin/integrations/misp/#misp-feed-output","title":"MISP Feed Output","text":"

This bot creates a complete MISP feed ready to be configured in MISP as incoming data source.

Look at the Bots documentation page for more information.

"},{"location":"admin/integrations/misp/#misp-api-output","title":"MISP API Output","text":"

Can be used to directly create MISP events in a MISP instance by using the MISP API.

Look at the Bots documentation page for more information.

"},{"location":"admin/integrations/n6/","title":"N6","text":""},{"location":"admin/integrations/n6/#intelmq-n6-integration","title":"IntelMQ - n6 Integration","text":"

n6 is an Open Source Tool with very similar aims as IntelMQ: processing and distributing IoC data. The use-cases, architecture and features differ and both tools have non-overlapping strengths. n6 is maintained and developed by CERT.pl.

Information about n6 can be found here:

"},{"location":"admin/integrations/n6/#data-format","title":"Data format","text":"

The internal data representation differs between IntelMQ and n6, so any data exchange between the systems requires a format conversion. For example, in n6 one message can contain multiple IP addresses, but IntelMQ is intentionally restricted to one IP address per message. Therefore, one n6 event results in one or more IntelMQ events. Because of this, and some other naming differences and ambiguities, the format conversion is not bidirectional.

"},{"location":"admin/integrations/n6/#data-exchange-interface","title":"Data exchange interface","text":"

n6 offers a STOMP interface via the RabbitMQ broker, which can be used for both sending and receiving data. IntelMQ offers both a STOMP collector bot for receiving data from n6, as well as a STOMP output bot for sending data to n6 instances.

"},{"location":"admin/integrations/n6/#data-conversion","title":"Data conversion","text":"

IntelMQ can parse n6 data using the n6 parser and n6 can parse IntelMQ data using the Intelmq n6 parser.

"},{"location":"admin/integrations/n6/#complete-example","title":"Complete example","text":""},{"location":"admin/integrations/n6/#data-flow-n6-to-intelmq","title":"Data flow n6 to IntelMQ","text":""},{"location":"admin/integrations/n6/#data-flow-intelmq-to-n6","title":"Data flow IntelMQ to n6","text":""},{"location":"admin/integrations/n6/#certpl-data-feed","title":"CERT.pl Data feed","text":"

CERT.pl offers data feed available to their partners through the STOMP interface. Our feeds documentation contains details how it can be enabled in IntelMQ: CERT.pl n6 STOMP stream

"},{"location":"admin/integrations/n6/#webinput-csv","title":"Webinput CSV","text":"

The IntelMQ Webinput CSV software can also be used together with n6. The documentation on this component can be found in the software's repository: https://github.com/certat/intelmq-webinput-csv/blob/master/docs/webinput-n6.md

"},{"location":"admin/management/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#managing-intelmq-api","title":"Managing IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#running","title":"Running","text":"

For development purposes and testing you can run directly using hug:

hug -m intelmq_api.serve\n
"},{"location":"admin/management/intelmq/","title":"IntelMQ","text":""},{"location":"admin/management/intelmq/#managing-intelmq","title":"Managing IntelMQ","text":""},{"location":"admin/management/intelmq/#required-services","title":"Required services","text":"

You need to enable and start Redis if not already done. Using systemd it can be done with:

systemctl enable redis.service\nsystemctl start redis.service\n
"},{"location":"admin/management/intelmq/#introduction","title":"Introduction","text":"

intelmqctl is the main tool to handle a intelmq installation. It handles the bots themselves and has some tools to handle the installation.

Should you get lost any time, just use the --help after any argument for further explanation.

> intelmqctl run file-output --help\n
"},{"location":"admin/management/intelmq/#manage-the-botnet","title":"Manage the botnet","text":"

In IntelMQ, the botnet is the set of all currently configured and enabled bots. All configured bots have their configuration in runtime.yaml. By default, all bots are enabled.

If no bot id is given, the command applies to all bots / the botnet. All commands except the start action are applied to all bots. But only enabled bots are started.

In the examples below, a very minimal botnet is used.

"},{"location":"admin/management/intelmq/#start","title":"start","text":"

The start action applies to all bots which are enabled.

> intelmqctl start\nStarting abusech-domain-parser...\nabusech-domain-parser is running.\nStarting abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is running.\nStarting deduplicator-expert...\ndeduplicator-expert is running.\nfile-output is disabled.\nBotnet is running.\n

As we can file-output is disabled and thus has not been started. You can always explicitly start disabled bots.

"},{"location":"admin/management/intelmq/#stop","title":"stop","text":"

The stop action applies to all bots. Assume that all bots have been running:

> intelmqctl stop\nStopping Botnet...\nStopping abusech-domain-parser...\nabusech-domain-parser is stopped.\nStopping abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is stopped.\nStopping deduplicator-expert...\ndeduplicator-expert is stopped.\nStopping file-output...\nfile-output is stopped.\nBotnet is stopped.\n
"},{"location":"admin/management/intelmq/#status","title":"status","text":"

With this command we can see the status of all configured bots. Here, the botnet was started beforehand:

> intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is disabled.\n

And if the disabled bot has also been started:

> intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is running.\n

If the botnet is stopped, the output looks like this:

> intelmqctl status\nabusech-domain-parser is stopped.\nabusech-feodo-domains-collector is stopped.\ndeduplicator-expert is stopped.\nfile-output is disabled.\n
"},{"location":"admin/management/intelmq/#restart","title":"restart","text":"

The same as start and stop consecutively.

"},{"location":"admin/management/intelmq/#reload","title":"reload","text":"

The same as reload of every bot.

"},{"location":"admin/management/intelmq/#enable-disable","title":"enable / disable","text":"

The sub commands enable and disable set the corresponding flags in runtime.yaml.

> intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl disable file-output\n> intelmqctl status\nfile-output is disabled.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl enable file-output\n> intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n
"},{"location":"admin/management/intelmq/#manage-individual-bots","title":"Manage individual bots","text":"

As all init systems, intelmqctl has the methods start, stop, restart, reload and status.

"},{"location":"admin/management/intelmq/#start_1","title":"start","text":"

This will start the bot with the ID file-output. A file with it's PID will be created in /opt/intelmq/var/run/[bot-id].pid.

> intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n

If the bot is already running, it won't be started again:

> intelmqctl start file-output\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#stop_1","title":"stop","text":"

If the PID file does exist, a SIGINT will be sent to the process. After 0.25s we check if the process is running. If not, the PID file will be removed.

> intelmqctl stop file-output\nStopping file-output...\nfile-output is stopped.\n

If there's no running bot, there's nothing to do.

> intelmqctl stop file-output\nfile-output was NOT RUNNING.\n

If the bot did not stop in 0.25s, intelmqctl will say it's still running:

> intelmqctl stop file-output\nfile-output is still running\n
"},{"location":"admin/management/intelmq/#status_1","title":"status","text":"

Checks for the PID file and if the process with the given PID is alive. If the PID file exists, but the process does not exist, it will be removed.

> intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n> intelmqctl status file-output\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#restart_1","title":"restart","text":"

The same as stop and start consecutively.

> intelmqctl restart file-output\nStopping file-output...\nfile-output is stopped.\nStarting file-output...\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#reload_1","title":"reload","text":"

Sends a SIGHUP to the bot, which will then reload the configuration.

> intelmqctl reload file-output\nReloading file-output ...\nfile-output is running.\n

If the bot is not running, we can't reload it:

> intelmqctl reload file-output\nfile-output was NOT RUNNING.\n
"},{"location":"admin/management/intelmq/#run","title":"run","text":"

This command is used for debugging purposes.

If launched with no arguments, the bot will call its init method and start processing messages as usual -- but you see everything happens.

> intelmqctl run file-output\nfile-output: RestAPIOutputBot initialized with id file-output and version 3.5.2 as process 12345.\nfile-output: Bot is starting.\nfile-output: Loading source pipeline and queue 'file-output-queue'.\nfile-output: Connected to source queue.\nfile-output: No destination queues to load.\nfile-output: Bot initialization completed.\nfile-output: Waiting for incoming message.\n

Note that if another instance of the bot is running, only warning will be displayed.

> intelmqctl run file-output\nMain instance of the bot is running in the background. You may want to launch: intelmqctl stop file-output\n

You can set the log level with the -l flag, e.g. -l DEBUG. For the 'console' subcommand, 'DEBUG' is the default.

"},{"location":"admin/management/intelmq/#console","title":"console","text":"

This command is used for debugging purposes.

If launched with console argument, you get a pdb live console; or ipdb or pudb consoles if they were previously installed (I.E. pip3 install ipdb --user).

> intelmqctl run file-output console\n*** Using console ipdb. Please use 'self' to access to the bot instance properties. ***\nipdb> self. ...\n

You may specify the desired console in the next argument.

> intelmqctl run file-output console pudb\n
"},{"location":"admin/management/intelmq/#message","title":"message","text":"

Operate directly with the input / output pipelines.

If get is the parameter, you see the message that waits in the input (source or internal) queue. If the argument is pop, the message gets popped as well.

> intelmqctl run file-output message get\nfile-output: Waiting for a message to get...\n{\n    \"classification.type\": \"c&c\",\n    \"feed.url\": \"https://example.com\",\n    \"raw\": \"1233\",\n    \"source.ip\": \"1.2.3.4\",\n    \"time.observation\": \"2017-05-17T22:00:33+00:00\",\n    \"time.source\": \"2017-05-17T22:00:32+00:00\"\n}\n

To send directly to the bot's output queue, just as it was sent by self.send_message() in bot's process() method, use the send argument. In our case of file-output, it has no destination queue so that nothing happens.

> intelmqctl run file-output message send '{\"time.observation\": \"2017-05-17T22:00:33+00:00\", \"time.source\": \"2017-05-17T22:00:32+00:00\"}'\nfile-output: Bot has no destination queues.\n

Note, if you would like to know possible parameters of the message, put a wrong one -- you will be prompted if you want to list all the current bot harmonization.

"},{"location":"admin/management/intelmq/#process","title":"process","text":"

With no other arguments, bot's process() method will be run one time.

> intelmqctl run file-output process\nfile-output: Bot is starting.\nfile-output: Bot initialization completed.\nfile-output: Processing...\nfile-output: Waiting for incoming message.\nfile-output: Received message {'raw': '1234'}.\n

If run with --dryrun|-d flag, the message gets never really popped out from the source or internal pipeline, nor sent to the output pipeline. Plus, you receive a note about the exact moment the message would get sent, or acknowledged. If the message would be sent to a non-default path, the name of this path is printed on the console.

> intelmqctl run file-output process -d\nfile-output:  * Dryrun only, no message will be really sent through.\n...\nfile-output: DRYRUN: Message would be acknowledged now!\n

You may trick the bot to process a JSON instead of the Message in its pipeline with --msg|-m flag.

> intelmqctl run file-output process -m '{\"source.ip\":\"1.2.3.4\"}'\nfile-output:  * Message from cli will be used when processing.\n...\n

If you wish to display the processed message as well, you the --show-sent|-s flag. Then, if sent through (either with --dryrun or without), the message gets displayed as well.

"},{"location":"admin/management/intelmq/#disable","title":"disable","text":"

Sets the enabled flag in the runtime configuration of the bot to false. By default, all bots are enabled.

Example output:

> intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl disable file-output\n> intelmqctl status file-output\nfile-output is disabled.\n
"},{"location":"admin/management/intelmq/#enable","title":"enable","text":"

Sets the enabled flag in the runtime configuration of the bot to true.

Example output:

> intelmqctl status file-output\nfile-output is disabled.\n> intelmqctl enable file-output\n> intelmqctl status file-output\nfile-output is stopped.\n
"},{"location":"admin/management/intelmq/#list-bots","title":"List bots","text":"

intelmqctl list bots does list all configured bots and their description.

"},{"location":"admin/management/intelmq/#list-queues","title":"List queues","text":"

intelmqctl list queues shows all queues which are currently in use according to the configuration and how much events are in it:

> intelmqctl list queues\nabusech-domain-parser-queue - 0\nabusech-domain-parser-queue-internal - 0\ndeduplicator-expert-queue - 0\ndeduplicator-expert-queue-internal - 0\nfile-output-queue - 234\nfile-output-queue-internal - 0\n

Use the -q or --quiet flag to only show non-empty queues:

> intelmqctl list queues -q\nfile-output-queue - 234\n

The --sum or --count flag will show the sum of events on all queues:

> intelmqctl list queues --sum\n42\n
"},{"location":"admin/management/intelmq/#logging","title":"Logging","text":"

intelmqctl can show the last log lines for a bot, filtered by the log level.

Logs are stored in /opt/intelmq/var/log/ or /var/log/intelmq/ directory. In case of failures, messages are dumped to the same directory with the file extension .dump.

See the help page for more information.

"},{"location":"admin/management/intelmq/#check","title":"Check","text":"

This command will do various sanity checks on the installation and especially the configuration.

"},{"location":"admin/management/intelmq/#orphaned-queues","title":"Orphaned Queues","text":"

The intelmqctl check tool can search for orphaned queues. \"Orphaned queues\" are queues that have been used in the past and are no longer in use. For example you had a bot which you removed or renamed afterwards, but there were still messages in it's source queue. The source queue won't be renamed automatically and is now disconnected. As this queue is no longer configured, it won't show up in the list of IntelMQ's queues too. In case you are using redis as message broker, you can use the redis-cli tool to examine or remove these queues:

redis-cli -n 2\nkeys * # lists all existing non-empty queues\nllen [queue-name] # shows the length of the queue [queue-name]\nlindex [queue-name] [index] # show the [index]'s message of the queue [queue-name]\ndel [queue-name] # remove the queue [queue-name]\n

To ignore certain queues in this check, you can set the parameter intelmqctl_check_orphaned_queues_ignore in the defaults configuration file. For example:

\"intelmqctl_check_orphaned_queues_ignore\": [\"Taichung-Parser\"]\n
"},{"location":"admin/management/intelmq/#configuration-upgrade","title":"Configuration upgrade","text":"

The intelmqctl upgrade-config function upgrade, upgrade the configuration from previous versions to the current one. It keeps track of previously installed versions and the result of all \"upgrade functions\" in the \"state file\", locate in the $var_state_path/state.json /opt/intelmq/var/lib/state.json or /var/lib/intelmq/state.json).

This function has been introduced in version 2.0.1.

It makes backups itself for all changed files before every run. Backups are overridden if they already exists. So make sure to always have a backup of your configuration just in case.

"},{"location":"admin/management/intelmq/#output-type","title":"Output type","text":"

intelmqctl can be used as command line tool, as library and as tool by other programs. If called directly, it will print all output to the console (stderr). If used as python library, the python types themselves are returned. The third option is to use machine-readable JSON as output (used by other managing tools).

"},{"location":"admin/management/intelmq/#exit-code","title":"Exit code","text":"

In case of errors, unsuccessful operations, the exit code is higher than 0. For example, when running intelmqctl start and one enabled bot is not running, the exit code is 1. The same is valid for e.g. intelmqctl status, which can be used for monitoring, and all other operations.

"},{"location":"admin/management/intelmq/#error-handling","title":"Error Handling","text":"

When bots are failing due to bad input data or programming errors, they can dump the problematic message to a file along with a traceback, if configured accordingly. These dumps are saved at in the logging directory as [botid].dump as JSON files. IntelMQ comes with an inspection and reinjection tool, called intelmqdump. It is an interactive tool to show all dumped files and the number of dumps per file. Choose a file by bot-id or listed numeric id. You can then choose to delete single entries from the file with e 1,3,4, show a message in more readable format with s 1 (prints the raw-message, can be long!), recover some messages and put them back in the pipeline for the bot by a or r 0,4,5. Or delete the file with all dumped messages using d.

intelmqdump -h\nusage:\n    intelmqdump [botid]\n    intelmqdump [-h|--help]\n\nintelmqdump can inspect dumped messages, show, delete or reinject them into\nthe pipeline. It's an interactive tool, directly start it to get a list of\navailable dumps or call it with a known bot id as parameter.\n\npositional arguments:\n  botid       botid to inspect dumps of\n\noptional arguments:\n  -h, --help  show this help message and exit\n  --truncate TRUNCATE, -t TRUNCATE\n                        Truncate raw-data with more characters than given. 0 for no truncating. Default: 1000.\n\nInteractive actions after a file has been selected:\n- r, Recover by IDs\n  > r id{,id} [queue name]\n  > r 3,4,6\n  > r 3,7,90 modify-expert-queue\n  The messages identified by a consecutive numbering will be stored in the\n  original queue or the given one and removed from the file.\n- a, Recover all\n  > a [queue name]\n  > a\n  > a modify-expert-queue\n  All messages in the opened file will be recovered to the stored or given\n  queue and removed from the file.\n- d, Delete entries by IDs\n  > d id{,id}\n  > d 3,5\n  The entries will be deleted from the dump file.\n- d, Delete file\n  > d\n  Delete the opened file as a whole.\n- s, Show by IDs\n  > s id{,id}\n  > s 0,4,5\n  Show the selected IP in a readable format. It's still a raw format from\n  repr, but with newlines for message and traceback.\n- e, Edit by ID\n  > e id\n  > e 0\n  > e 1,2\n  Opens an editor (by calling `sensible-editor`) on the message. The modified message is then saved in the dump.\n- q, Quit\n  > q\n\n$ intelmqdump\n id: name (bot id)                    content\n  0: alienvault-otx-parser            1 dumps\n  1: cymru-whois-expert               8 dumps\n  2: deduplicator-expert              2 dumps\n  3: dragon-research-group-ssh-parser 2 dumps\n  4: file-output2                     1 dumps\n  5: fraunhofer-dga-parser            1 dumps\n  6: spamhaus-cert-parser             4 dumps\n  7: test-bot                         2 dumps\nWhich dump file to process (id or name)? 3\nProcessing dragon-research-group-ssh-parser: 2 dumps\n  0: 2015-09-03T13:13:22.159014 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n  1: 2015-09-01T14:40:20.973743 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n(r)ecover by ids, recover (a)ll, delete (e)ntries, (d)elete file, (s)how by ids, (q)uit, edit id (v)? d\nDeleted file /opt/intelmq/var/log/dragon-research-group-ssh-parser.dump\n

Bots and the intelmqdump tool use file locks to prevent writing to already opened files. Bots are trying to lock the file for up to 60 seconds if the dump file is locked already by another process (intelmqdump) and then give up. Intelmqdump does not wait and instead only shows an error message.

By default, the show command truncates the raw field of messages at 1000 characters to change this limit or disable truncating at all (value 0), use the --truncate parameter.

"},{"location":"admin/management/intelmq/#known-issues","title":"Known issues","text":"

The currently implemented process managing using PID files is very erroneous.

"},{"location":"admin/utilities/bash-completion/","title":"Bash Completion","text":""},{"location":"admin/utilities/bash-completion/#bash-completion","title":"Bash Completion","text":"

To enable bash completion on intelmqctl and intelmqdump in order to help you run the commands in an easy manner, follow the installation process here.

Bug

This section of the documentation is currently incomplete and will be added later.

"},{"location":"dev/adding-feeds/","title":"Adding Feeds","text":""},{"location":"dev/adding-feeds/#adding-feeds","title":"Adding Feeds","text":"

Adding a feed doesn't necessarily require any programming experience. There are several collector and parser bots intended for general use. Depending on the data source you are trying to add as a feed, it might be only a matter of creating a working combination of collector bot (such as URL Fetcher) configuration and a parser bot (such as CSV parser) configuration. When you are satisfied with the configurations, add it to the intelmq/etc/feeds.yaml file using the following template and open a pull request!

<NAME OF THE FEED PROVIDER>:\n    <NAME OF THE FEED>:\n      description: <DESCRIPTION OF WHAT KIND OF DATA THE FEED PROVIDES>\n      additional_information: <ANY ADDITIONAL INFORMATION>\n      documentation: <FEED HOMEPAGE/DOCUMENTATION URL>\n      revision: <DATE WHEN YOU ADDED THIS FEED>\n      public: <TRUE/FALSE IF THE DATA SOURCE IS PUBLICLY AVAILABLE>\n      bots:\n        collector:\n          module: <MODULE USED FOR THE COLLECTOR BOT>\n          parameters:\n            name: __FEED__ # KEEP AS IT IS\n            provider: __PROVIDER__  # KEEP AS IT IS\n            <ADDITIONAL COLLECTOR BOT PARAMETERS>\n        parser:\n          module: <MODULE USED FOR THE PARSER BOT>\n          parameters:\n            <ADDITIONAL PARSER BOT PARAMETERS>\n

If the data source utilizes some unusual way of distribution or uses a custom format for the data it might be necessary to develop specialized bot(s) for this particular data source. Always try to use existing bots before you start developing your own. Please also consider extending an existing bot if your use-case is close enough to it's features. If you are unsure which way to take, start an issue and you will receive guidance.

"},{"location":"dev/adding-feeds/#feeds-wishlist","title":"Feeds Wishlist","text":"

This is a list with potentially interesting data sources, which are either currently not supported or the usage is not clearly documented in IntelMQ. If you want to contribute new feeds to IntelMQ, this is a great place to start!

Note

Some of the following data sources might better serve as an expert bot for enriching processed events.

"},{"location":"dev/bot-development/","title":"Bot Development","text":""},{"location":"dev/bot-development/#bot-development","title":"Bot Development","text":"

Here you should find everything you need to develop a new bot.

"},{"location":"dev/bot-development/#steps","title":"Steps","text":"
  1. Create appropriately placed and named python file.
  2. Use correct parent class.
  3. Code the functionality you want (with mixins, inheritance, etc).
  4. Create appropriately placed test file.
  5. Prepare code for testing your bot.
  6. Add documentation for your bot.
  7. Add changelog and news info.
"},{"location":"dev/bot-development/#layout-rules","title":"Layout Rules","text":"
intelmq/\n  lib/\n    bot.py\n    cache.py\n    message.py\n    pipeline.py\n    utils.py\n  bots/\n    collector/\n      <bot name>/\n            collector.py\n    parser/\n      <bot name>/\n            parser.py\n    expert/\n      <bot name>/\n            expert.py\n    output/\n      <bot name>/\n            output.py\n  etc/\n    runtime.yaml\n

Assuming you want to create a bot for a new 'Abuse.ch' feed. It turns out that here it is necessary to create different parsers for the respective kind of events (e.g. malicious URLs). Therefore, the usual hierarchy intelmq/bots/parser/<FEED>/parser.py would not be suitable because it is necessary to have more parsers for each Abuse.ch Feed. The solution is to use the same hierarchy with an additional \"description\" in the file name, separated by underscore. Also see the section Directories and Files naming.

Example (including the current ones):

/intelmq/bots/parser/abusech/parser_domain.py\n/intelmq/bots/parser/abusech/parser_ip.py\n/intelmq/bots/parser/abusech/parser_ransomware.py\n/intelmq/bots/parser/abusech/parser_malicious_url.py\n
"},{"location":"dev/bot-development/#directories-hierarchy-on-default-installation","title":"Directories Hierarchy on Default Installation","text":""},{"location":"dev/bot-development/#directories-and-files-naming","title":"Directories and Files naming","text":"

Any directory and file of IntelMQ has to follow the Directories and Files naming. Any file name or folder name has to:

In the bot directories name, the name must correspond to the feed provider. If necessary and applicable the feed name can and should be used as postfix for the filename.

Examples:

intelmq/bots/parser/taichung/parser.py\nintelmq/bots/parser/cymru/parser_full_bogons.py\nintelmq/bots/parser/abusech/parser_ransomware.py\n
"},{"location":"dev/bot-development/#guide","title":"Guide","text":""},{"location":"dev/bot-development/#naming-your-bot-class","title":"Naming your bot class","text":"

Class name of the bot (ex: PhishTank Parser) must correspond to the type of the bot (ex: Parser) e.g. PhishTankParserBot

"},{"location":"dev/bot-development/#choosing-the-parent-class","title":"Choosing the parent class","text":"

Please use the correct bot type as parent class for your bot. The intelmq.lib.bot module contains the following classes:

"},{"location":"dev/bot-development/#template","title":"Template","text":"

Please adjust the doc strings accordingly and remove the in-line comments (#).

\"\"\"\nSPDX-FileCopyrightText: 2021 Your Name\nSPDX-License-Identifier: AGPL-3.0-or-later\n\nParse data from example.com, be a nice ExampleParserBot.\n\nDocument possible necessary configurations.\n\"\"\"\nimport sys\n\n# imports for additional libraries and intelmq\nfrom intelmq.lib.bot import ParserBot\n\n\nclass ExampleParserBot(ParserBot):\n    option1: str = \"defaultvalue\"\n    option2: bool = False\n\n    def process(self):\n        report = self.receive_message()\n\n        event = self.new_event(report)  # copies feed.name, time.observation\n        ...  # implement the logic here\n        event.add('source.ip', '127.0.0.1')\n        event.add('extra', {\"os.name\": \"Linux\"})\n        if self.option2:\n            event.add('extra', {\"customvalue\": self.option1})\n\n        self.send_message(event)\n        self.acknowledge_message()\n\n\nBOT = ExampleParserBot\n

Any attributes of the bot that are not private can be set by the user using the IntelMQ configuration settings.

There are some names with special meaning. These can be used i.e. called:

These can be defined:

All other names can be used freely.

"},{"location":"dev/bot-development/#mixins","title":"Mixins","text":"

For common settings and methods you can use mixins from intelmq.lib.mixins. To use the mixins, just let your bot inherit from the Mixin class (in addition to the inheritance from the Bot class). For example:

class HTTPCollectorBot(CollectorBot, HttpMixin):\n

The following mixins are available:

The HttpMixin provides the HTTP attributes described in common-parameters and the following methods:

The SqlMixin provides methods to connect to SQL servers. Inherit this Mixin so that it handles DB connection for you. You do not have to bother:

The CacheMixin provides methods to cache values for bots in a Redis database. It uses the following attributes:

and provides the methods:

"},{"location":"dev/bot-development/#pipeline-interactions","title":"Pipeline Interactions","text":"

We can call three methods related to the pipeline:

"},{"location":"dev/bot-development/#logging","title":"Logging","text":""},{"location":"dev/bot-development/#log-messages-format","title":"Log Messages Format","text":"

Log messages have to be clear and well formatted. The format is the following:

Format:

<timestamp> - <bot id> - <log level> - <log message>\n

Rules:

When the logger instance is created, the bot id must be given as parameter anyway. The function call defines the log level, see below.

"},{"location":"dev/bot-development/#log-levels","title":"Log Levels","text":""},{"location":"dev/bot-development/#what-to-log","title":"What to Log","text":""},{"location":"dev/bot-development/#how-to-log","title":"How to Log","text":"

The Bot class creates a logger with that should be used by bots. Other components won't log anyway currently. Examples:

self.logger.info('Bot start processing.')\nself.logger.error('Pipeline failed.')\nself.logger.exception('Pipeline failed.')\n

The exception method automatically appends an exception traceback. The logger instance writes by default to the file /opt/intelmq/var/log/[bot-id].log and to stderr.

"},{"location":"dev/bot-development/#string-formatting-in-logs","title":"String formatting in Logs","text":"

Parameters for string formatting are better passed as argument to the log function, see https://docs.python.org/3/library/logging.html#logging.Logger.debug In case of formatting problems, the error messages will be better. For example:

self.logger.debug('Connecting to %r.', host)\n
"},{"location":"dev/bot-development/#error-handling","title":"Error handling","text":"

The bot class itself has error handling implemented. The bot itself is allowed to throw exceptions and intended to fail! The bot should fail in case of malicious messages, and in case of unavailable but necessary resources. The bot class handles the exception and will restart until the maximum number of tries is reached and fail then. Additionally, the message in question is dumped to the file /opt/intelmq/var/log/[bot-id].dump and removed from the queue.

"},{"location":"dev/bot-development/#initialization","title":"Initialization","text":"

Maybe it is necessary so setup a Cache instance or load a file into memory. Use the init function for this purpose:

class ExampleParserBot(Bot):\n    def init(self):\n        try:\n            self.database = pyasn.pyasn(self.database)\n        except IOError:\n            self.logger.error(\"pyasn data file does not exist or could not be \"\n                              \"accessed in '%s'.\" % self.database)\n            self.logger.error(\"Read 'bots/experts/asn_lookup/README.md' and \"\n                              \"follow the procedure.\")\n            self.stop()\n
"},{"location":"dev/bot-development/#custom-configuration-checks","title":"Custom configuration checks","text":"

Every bot can define a static method check(parameters) which will be called by intelmqctl check. For example the check function of the ASNLookupExpert:

@staticmethod\ndef check(parameters):\n    if not os.path.exists(parameters.get('database', '')):\n        return [[\"error\", \"File given as parameter 'database' does not exist.\"]]\n    try:\n        pyasn.pyasn(parameters['database'])\n    except Exception as exc:\n        return [[\"error\", \"Error reading database: %r.\" % exc]]\n
"},{"location":"dev/bot-development/#running","title":"Running","text":"

You can always start any bot directly from command line by calling the executable. The executable will be created during installation a directory for binaries. After adding new bots to the code, install IntelMQ to get the files created. Don't forget to give an bot id as first argument. Also, running bots with other users than intelmq will raise permission errors.

$ sudo -i intelmq\n$ intelmqctl run file-output  # if configured\n$ intelmq.bots.outputs.file.output file-output\n

You will get all logging outputs directly on stderr as well as in the log file.

"},{"location":"dev/bot-development/#examples","title":"Examples","text":""},{"location":"dev/bot-development/#parsers","title":"Parsers","text":"

Parsers can use a different, specialized Bot-class. It allows to work on individual elements of a report, splitting the functionality of the parser into multiple functions:

For common cases, like CSV, existing function can be used, reducing the amount of code to implement. In the best case, only parse_line needs to be coded, as only this part interprets the data.

You can have a look at the implementation intelmq/lib/bot.py or at examples, e.g. the DummyBot in intelmq/tests/lib/test_parser_bot.py. This is a stub for creating a new Parser, showing the parameters and possible code:

class MyParserBot(ParserBot):\n\n    def parse(self, report):\n        \"\"\"A generator yielding the single elements of the data.\n\n        Comments, headers etc. can be processed here. Data needed by\n        `self.parse_line` can be saved in `self.tempdata` (list).\n\n        Default parser yields stripped lines.\n        Override for your use or use an existing parser, e.g.:\n            parse = ParserBot.parse_csv\n        \"\"\"\n        for line in utils.base64_decode(report.get(\"raw\")).splitlines():\n            yield line.strip()\n\n    def parse_line(self, line, report):\n        \"\"\"A generator which can yield one or more messages contained in line.\n\n        Report has the full message, thus you can access some metadata.\n        Override for your use.\n        \"\"\"\n        raise NotImplementedError\n\n    def process(self):\n        self.tempdata = []  # temporary data for parse, parse_line and recover_line\n        self.__failed = []\n        report = self.receive_message()\n\n        for line in self.parse(report):\n            if not line:\n                continue\n            try:\n                # filter out None\n                events = list(filter(bool, self.parse_line(line, report)))\n            except Exception as exc:\n                self.logger.exception('Failed to parse line.')\n                self.__failed.append((exc, line))\n            else:\n                self.send_message(*events)\n\n        for exc, line in self.__failed:\n            self._dump_message(exc, self.recover_line(line))\n\n        self.acknowledge_message()\n\n    def recover_line(self, line):\n        \"\"\"Reverse of parse for single lines.\n\n        Recovers a fully functional report with only the problematic line.\n        \"\"\"\n        return 'n'.join(self.tempdata + [line])\n\n\nBOT = MyParserBot\n
"},{"location":"dev/bot-development/#parse_line","title":"parse_line","text":"

One line can lead to multiple events, thus parse_line can't just return one Event. Thus, this function is a generator, which allows to easily return multiple values. Use yield event for valid Events and return in case of a void result (not parsable line, invalid data etc.).

"},{"location":"dev/bot-development/#tests","title":"Tests","text":"

In order to do automated tests on the bot, it is necessary to write tests including sample data. Have a look at some existing tests:

Ideally an example contains not only the ideal case which should succeed, but also a case where should fail instead. (TODO: Implement assertEventNotEqual or assertEventNotcontainsSubset or similar) Most existing bots are only tested with one message. For newly written test it is appreciable to have tests including more then one message, e.g. a parser fed with an report consisting of multiple events.

import unittest\n\nimport intelmq.lib.test as test\nfrom intelmq.bots.parsers.exampleparser.parser import ExampleParserBot  # adjust bot class name and module\n\n\nclass TestExampleParserBot(test.BotTestCase, unittest.TestCase):  # adjust test class name\n    \"\"\"A TestCase for ExampleParserBot.\"\"\"\n\n    @classmethod\n    def set_bot(cls):\n        cls.bot_reference = ExampleParserBot  # adjust bot class name\n        cls.default_input_message = EXAMPLE_EVENT  # adjust source of the example event (dict), by default an empty event or report (depending on bot type)\n\n    # This is an example how to test the log output\n    def test_log_test_line(self):\n        \"\"\"Test if bot does log example message.\"\"\"\n        self.run_bot()\n        self.assertRegexpMatches(self.loglines_buffer,\n                                 \"INFO - Lorem ipsum dolor sit amet\")\n\n    def test_event(self):\n        \"\"\"Test if correct Event has been produced.\"\"\"\n        self.run_bot()\n        self.assertMessageEqual(0, EXAMPLE_REPORT)\n\n\nif __name__ == '__main__':  # pragma: no cover\n    unittest.main()\n

When calling the file directly, only the tests in this file for the bot will be expected. Some default tests are always executed (via the test.BotTestCase class), such as pipeline and message checks, logging, bot naming or empty message handling.

See the testing section about how to run the tests.

"},{"location":"dev/bot-development/#cache","title":"Cache","text":"

Bots can use a Redis database as cache instance. Use the intelmq.lib.utils.Cache class to set this up and/or look at existing bots, like the cymru_whois expert how the cache can be used. Bots must set a TTL for all keys that are cached to avoid caches growing endless over time. Bots must use the Redis databases >= 10, but not those already used by other bots. Look at find intelmq -type f -name '*.py' -exec grep -r 'redis_cache_db' {} + to see which databases are already used.

The databases < 10 are reserved for the IntelMQ core:

"},{"location":"dev/bot-development/#documentation","title":"Documentation","text":"

Please document your added/modified code.

For doc strings, we are using the sphinx-napoleon-google-type-annotation.

Additionally, Python's type hints/annotations are used, see PEP484.

"},{"location":"dev/bot-development/#testing-pre-releases","title":"Testing Pre-releases","text":""},{"location":"dev/bot-development/#installation","title":"Installation","text":"

The installation procedures need to be adapted only a little bit.

For native packages, you can find the unstable packages of the next version here: Installation Unstable Native Packages . The unstable only has a limited set of packages, so enabling the stable repository can be activated in parallel. For CentOS 8 unstable, the stable repository is required.

For the installation with pip, use the --pre parameter as shown here following command:

pip3 install --pre intelmq\n

All other steps are not different. Please report any issues you find in our Issue Tracker.

"},{"location":"dev/data-format/","title":"Data Format","text":""},{"location":"dev/data-format/#data-format","title":"Data Format","text":"

Data passed between bots is called a Message. There are two types of Messages: Report and Event. Report is produced by collector bots and consists of collected raw data (CSV, JSON, HTML, etc) and feed metadata. It is passed to a parser bot which parses Report into a single or multiple Events. Expert bots and output bots handle only Events.

All Messages (Reports and Events) are Python dictionaries (or JSONs). The key names and according types are defined by the IntelMQ Data Format.

The source code for the Data Format can be found in the Python module intelmq.lib.harmonization and the configuration is present inside the harmonization.conf file. (The term Harmonization is used for historical reasons.)

"},{"location":"dev/data-format/#rules-for-keys","title":"Rules for keys","text":"

The keys are grouped together in sub-fields, e.g. source.ip or source.geolocation.latitude.

Only the lower-case alphabet, numbers and the underscore are allowed. Further, the field name must not begin with a number. Thus, keys must match ^[a-z_][a-z_0-9]+(\\.[a-z_0-9]+)*$. These rules also apply for the otherwise unregulated extra. namespace.

"},{"location":"dev/data-format/#data-types","title":"Data Types","text":"

This document describes the IntelMQ data types used for individual events with a description of each allowed field.

"},{"location":"dev/data-format/#asn","title":"ASN","text":"

ASN type. Derived from Integer with forbidden values.

Only valid are: 0 < ASN <= 4294967295

See https://en.wikipedia.org/wiki/Autonomous_system_(Internet)

The first and last ASNs of the original 16-bit integers, namely 0 and 65,535, and the last ASN of the 32-bit numbers, namely 4,294,967,295 are reserved and should not be used by operators.

"},{"location":"dev/data-format/#accuracy","title":"Accuracy","text":"

Accuracy type. A Float between 0 and 100.

"},{"location":"dev/data-format/#base64","title":"Base64","text":"

Base64 type. Always gives unicode strings.

Sanitation encodes to base64 and accepts binary and unicode strings.

"},{"location":"dev/data-format/#boolean","title":"Boolean","text":"

Boolean type. Without sanitation only python bool is accepted.

Sanitation accepts string 'true' and 'false' and integers 0 and 1.

"},{"location":"dev/data-format/#classificationtaxonomy","title":"ClassificationTaxonomy","text":"

classification.taxonomy type.

The mapping follows Reference Security Incident Taxonomy Working Group \u2013 RSIT WG: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/

These old values are automatically mapped to the new ones:

Allowed values are:

"},{"location":"dev/data-format/#classificationtype","title":"ClassificationType","text":"

classification.type type.

The mapping extends Reference Security Incident Taxonomy Working Group \u2013 RSIT WG:

https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/

These old values are automatically mapped to the new ones:

These values changed their taxonomy: 'malware': In terms of the taxonomy 'malicious-code' they can be either 'infected-system' or 'malware-distribution' but in terms of malware actually, it is now taxonomy 'other'

Allowed values are:

"},{"location":"dev/data-format/#datetime","title":"DateTime","text":"

Date and time type for timestamps.

Valid values are timestamps with time zone and in the format '%Y-%m-%dT%H:%M:%S+00:00'. Invalid are missing times and missing timezone information (UTC). Microseconds are also allowed.

Sanitation normalizes the timezone to UTC, which is the only allowed timezone.

The following additional conversions are available with the convert function:

"},{"location":"dev/data-format/#fqdn","title":"FQDN","text":"

Fully qualified domain name type.

All valid lowercase domains are accepted, no IP addresses or URLs. Trailing dot is not allowed.

To prevent values like '10.0.0.1:8080' (#1235), we check for the non-existence of ':'.

"},{"location":"dev/data-format/#float","title":"Float","text":"

Float type. Without sanitation only python float/integer/long is accepted. Boolean is explicitly denied.

Sanitation accepts strings and everything float() accepts.

"},{"location":"dev/data-format/#ipaddress","title":"IPAddress","text":"

Type for IP addresses, all families. Uses the ipaddress module.

Sanitation accepts integers, strings and objects of ipaddress.IPv4Address and ipaddress.IPv6Address.

Valid values are only strings. 0.0.0.0 is explicitly not allowed.

"},{"location":"dev/data-format/#ipnetwork","title":"IPNetwork","text":"

Type for IP networks, all families. Uses the ipaddress module.

Sanitation accepts strings and objects of ipaddress.IPv4Network and ipaddress.IPv6Network. If host bits in strings are set, they will be ignored (e.g 127.0.0.1/32).

Valid values are only strings.

"},{"location":"dev/data-format/#integer","title":"Integer","text":"

Integer type. Without sanitation only python integer/long is accepted. Bool is explicitly denied.

Sanitation accepts strings and everything int() accepts.

"},{"location":"dev/data-format/#json","title":"JSON","text":"

JSON type.

Sanitation accepts any valid JSON objects.

Valid values are only unicode strings with JSON objects.

"},{"location":"dev/data-format/#jsondict","title":"JSONDict","text":"

JSONDict type.

Sanitation accepts pythons dictionaries and JSON strings.

Valid values are only unicode strings with JSON dictionaries.

"},{"location":"dev/data-format/#lowercasestring","title":"LowercaseString","text":"

Like string, but only allows lower case characters.

Sanitation lowers all characters.

"},{"location":"dev/data-format/#registry","title":"Registry","text":"

Registry type. Derived from UppercaseString.

Only valid values: AFRINIC, APNIC, ARIN, LACNIC, RIPE. RIPE-NCC and RIPENCC are normalized to RIPE.

"},{"location":"dev/data-format/#string","title":"String","text":"

Any non-empty string without leading or trailing whitespace.

"},{"location":"dev/data-format/#tlp","title":"TLP","text":"

TLP level type. Derived from UppercaseString.

Only valid values: WHITE, GREEN, AMBER, RED.

Accepted for sanitation are different cases and the prefix 'tlp:'.

"},{"location":"dev/data-format/#url","title":"URL","text":"

URI type. Local and remote.

Sanitation converts hxxp and hxxps to http and https. For local URIs (file) a missing host is replaced by localhost.

Valid values must have the host (network location part).

"},{"location":"dev/data-format/#uppercasestring","title":"UppercaseString","text":"

Like string, but only allows upper case characters.

Sanitation uppers all characters.

"},{"location":"dev/documentation/","title":"Documentation","text":""},{"location":"dev/documentation/#documentation","title":"Documentation","text":"

The documentation is automatically published to https://docs.intelmq.org at every push to the develop branch of the repository.

To build the documentation you need additional packages:

pip3 install .[development]\n

Then use the Makefile to build the documentation using mkdocs:

make docs\n

Some parts of the documentation are automatically generated using dedicated scripts. You can find them in the Makefile.

"},{"location":"dev/environment/","title":"Environment","text":""},{"location":"dev/environment/#development-environment","title":"Development Environment","text":""},{"location":"dev/environment/#directories","title":"Directories","text":"

For development purposes, you need two directories:

The default root directory of the IntelMQ installation is /opt/intelmq. This directory is used for configurations (/opt/intelmq/etc), local states (/opt/intelmq/var/lib) and logs (/opt/intelmq/var/log). If you want to change it, please set the INTELMQ_ROOT_DIR environment variable with a desired location.

For repository directory, you can use any path that is accessible by users you use to run IntelMQ. For globally installed IntelMQ, the directory has to be readable by other unprivileged users (e.g. home directories on Fedora can't be read by other users by default).

To keep commands in the guide universal, we will use environmental variables for repository and installation paths. You can set them with following commands:

# Adjust paths if you want to use non-standard directories\nexport INTELMQ_REPO=/opt/dev_intelmq\nexport INTELMQ_ROOT_DIR=/opt/intelmq\n

Note

If using non-default installation directory, remember to keep the root directory variable set for every run of IntelMQ commands. If you don't, then the default location /opt/intelmq will be used.

"},{"location":"dev/environment/#installation","title":"Installation","text":"

Developers can create a fork repository of IntelMQ in order to commit the new code to this repository and then be able to do pull requests to the main repository. Otherwise you can just use the 'certtools' as username below.

The following instructions will use pip3 -e, which gives you a so called editable installation. No code is copied in the libraries directories, there's just a link to your code. However, configuration files still required to be moved to /opt/intelmq as the instructions show.

The traditional way to work with IntelMQ is to install it globally and have a separated user for running it. If you wish to separate your machine Python's libraries, e.g. for development purposes, you could alternatively use a Python virtual environment and your local user to run IntelMQ. Please use your preferred way from instructions below.

"},{"location":"dev/environment/#using-globally-installed-intelmq","title":"Using globally installed IntelMQ","text":"
sudo -s\n\ngit clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npip3 install -e .\n\nuseradd -d $INTELMQ_ROOT_DIR -U -s /bin/bash intelmq\n\nintelmqsetup\n
"},{"location":"dev/environment/#using-virtual-environment","title":"Using virtual environment","text":"
git clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npython -m venv .venv\nsource .venv/bin/activate\n\npip install -e .\n\n# If you use a non-local directory as INTELMQ_ROOT_DIR, use following\n# command to create it and change the ownership.\nsudo install -g `whoami` -o `whoami` -d $INTELMQ_ROOT_DIR\n# For local directory, just create it with mkdir:\nmkdir $INTELMQ_ROOT_DIR\n\nintelmqsetup --skip-ownership\n

Note

Please do not forget that configuration files, log files will be available on $INTELMQ_ROOT_DIR. However, if your development is somehow related to any shipped configuration file, you need to apply the changes in your repository $INTELMQ_REPO/intelmq/etc/.

"},{"location":"dev/environment/#additional-services","title":"Additional services","text":"

Some features require additional services, like message queue or database. The commonly used services are gained for development purposes in the Docker Compose file in contrib/development-tools/docker-compose-common-services.yaml in the repository. You can use them to run services on your machine in a docker containers, or decide to configure them in an another way. To run them using Docker Compose, use following command from the main repository directory:

# For older Docker versions, you may need to use `docker-compose` command\ndocker compose -f contrib/development-tools/docker-compose-common-services.yaml up -d\n

This will start in the background containers with Redis, RabbitMQ, PostgreSQL and MongoDB.

"},{"location":"dev/environment/#how-to-develop","title":"How to develop","text":"

After you successfully setup your IntelMQ development environment, you can perform any development on any .py file on $INTELMQ_REPO. After you change, you can use the normal procedure to run the bots:

su - intelmq # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start spamhaus-drop-collector\n\ntail -f $INTELMQ_ROOT_DIR/var/log/spamhaus-drop-collector.log\n

You can also add new bots, creating the new .py file on the proper directory inside cd $INTELMQ_REPO/intelmq. However, your IntelMQ installation with pip3 needs to be updated. Please check the following section.

"},{"location":"dev/environment/#update","title":"Update","text":"

In case you developed a new bot, you need to update your current development installation. In order to do that, please follow this procedure:

  1. Make sure that you have your new bot in the right place.

  2. Update pip metadata and new executables: 

    sudo -s # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\ncd /opt/dev_intelmq\npip3 install -e .\n

  3. If you're using the global installation, an additional step of changing permissions and ownership is necessary: 

    find $INTELMQ_ROOT_DIR/ -type d -exec chmod 0770 {} \\+\nfind $INTELMQ_ROOT_DIR/ -type f -exec chmod 0660 {} \\+\nchown -R intelmq.intelmq $INTELMQ_ROOT_DIR\n## if you use the intelmq manager (adapt the webservers' group if needed):\nchown intelmq.www-data $INTELMQ_ROOT_DIR/etc/*.conf\n

Now you can test run your new bot following this procedure:

su - intelmq              # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start <bot_id>\n
"},{"location":"dev/extensions-packages/","title":"Extensions Packages","text":""},{"location":"dev/extensions-packages/#creating-extensions-packages","title":"Creating extensions packages","text":"

IntelMQ supports adding additional bots using your own independent packages. You can use this to add a new integration that is special to you, or cannot be integrated into the main IntelMQ repository for some reason.

"},{"location":"dev/extensions-packages/#building-an-extension-package","title":"Building an extension package","text":"

A simple example of the package can be found in contrib/example-extension-package. To make your custom bots work with IntelMQ, you need to ensure that

Apart from these requirements, your package can use any of the usual package features. We strongly recommend following the same principles and main guidelines as the official bots. This will ensure the same experience when using official and additional bots.

"},{"location":"dev/extensions-packages/#naming-convention","title":"Naming convention","text":"

Building your own extensions gives you a lot of freedom, but it's important to know that if your bot's entry point uses the same name as another bot, it may not be possible to use it, or to determine which one is being used. For this reason, we recommend that you start the name of your bot with an with an organization identifier and then the bot name.

For example, if I create a collector bot for feed source Special and run it on behalf of the organization Awesome, the suggested entry point might be intelmq.bots.collectors.awesome.special. Note that the structure of your package doesn't matter, as long as it can be imported properly.

For example, I could create a package called awesome-bots with the following file structure

   awesome_bots\n   \u251c\u2500\u2500 pyproject.toml\n   \u2514\u2500\u2500 awesome_bots\n        \u251c\u2500\u2500 __init__.py\n        \u2514\u2500\u2500 special.py\n

The pyproject.toml file would then have the following section:

   [project.scripts]\n   intelmq.bots.collectors.awesome.special = \"awesome_bots.special:BOT.run\"\n

Once you have installed your package, you can run intelmqctl list bots to check if your bot was properly registered.

"},{"location":"dev/guidelines/","title":"Guidelines","text":""},{"location":"dev/guidelines/#development-guidelines","title":"Development Guidelines","text":""},{"location":"dev/guidelines/#coding-rules","title":"Coding-Rules","text":"

Most important: KEEP IT SIMPLE! This can not be over-estimated. Feature creep can destroy any good software project. But if new folks can not understand what you wrote in 10-15 minutes, it is not good. It's not about the performance, etc. It's about readability.

In general, we follow PEP8. We recommend reading it before committing code.

There are some exceptions: sometimes it does not make sense to check for every PEP8 error (such as whitespace indentation when you want to make a dict=() assignment look pretty. Therefore, we do have some exceptions defined in the setup.cfg file.

We support Python 3 only.

"},{"location":"dev/guidelines/#unicode","title":"Unicode","text":""},{"location":"dev/guidelines/#back-end-independence-and-compatibility","title":"Back-end independence and Compatibility","text":"

Any component of the IntelMQ MUST be independent of the message queue technology (Redis, RabbitMQ, etc...).

"},{"location":"dev/guidelines/#license-header","title":"License Header","text":"

Please add a license and copyright header to your bots. There is a Github action that tests for reuse compliance of your code files.

"},{"location":"dev/guidelines/#intelmq-data-format-rules","title":"IntelMQ Data Format Rules","text":"

Any component of IntelMQ MUST respect the IntelMQ Data Format.

"},{"location":"dev/guidelines/#code-submission-rules","title":"Code Submission Rules","text":""},{"location":"dev/guidelines/#releases-repositories-and-branches","title":"Releases, Repositories and Branches","text":""},{"location":"dev/guidelines/#branching-model","title":"Branching model","text":""},{"location":"dev/guidelines/#how-to-contribute","title":"How to Contribute","text":""},{"location":"dev/guidelines/#workflow","title":"Workflow","text":"

We assume here, that origin is your own fork. We first add the upstream repository:

 git remote add upstream https://github.com/certtools/intelmq.git\n

Syncing develop:

 git checkout develop\n git pull upstream develop\n git push origin develop\n

You can do the same with the branches master and maintenance.

Create a separate feature-branch to work on, sync develop with upstream. Create working branch from develop:

 git checkout develop\n git checkout -b bugfix\n# your work\n git commit\n

Or, for bugfixes create a separate bugfix-branch to work on, sync maintenance with upstream. Create working branch from maintenance:

git checkout maintenance\ngit checkout -b new-feature\n# your work\ngit commit\n

Getting upstream's changes for master or any other branch:

git checkout develop\ngit pull upstream develop\ngit push origin develop\n

There are 2 possibilities to get upstream's commits into your branch. Rebasing and Merging. Using rebasing, your history is rewritten, putting your changes on top of all other commits. You can use this if your changes are not published yet (or only in your fork).

git checkout bugfix\ngit rebase develop\n

Using the -i flag for rebase enables interactive rebasing. You can then remove, reorder and squash commits, rewrite commit messages, beginning with the given branch, e.g. develop.

Or using merging. This doesn't break the history. It's considered more , but also pollutes the history with merge commits.

git checkout bugfix\ngit merge develop\n

You can then create a PR with your branch bugfix to our upstream repository, using GitHub's web interface.

"},{"location":"dev/guidelines/#commit-messages","title":"Commit Messages","text":"

If it fixes an existing issue, please use GitHub syntax, e.g.: fixes certtools/intelmq#<IssueID>

"},{"location":"dev/guidelines/#prepare-for-discussion-in-github","title":"Prepare for Discussion in GitHub","text":"

If we don't discuss it, it's probably not tested.

"},{"location":"dev/guidelines/#license-and-author-files","title":"License and Author files","text":"

License and Authors files can be found at the root of repository.

License and authors must be only listed in an external file but not inside the code files.

"},{"location":"dev/intro/","title":"Intro","text":""},{"location":"dev/intro/#intro","title":"Intro","text":"

This guide is for developers of IntelMQ. It explains the code architecture, coding guidelines as well as ways you can contribute code or documentation. If you have not done so, please read the User Guide and the Administrator Guide first. Once you feel comfortable running IntelMQ with open source bots and you feel adventurous enough to contribute to the project, this guide is for you. It does not matter if you are an experienced Python programmer or just a beginner. There is a lot of examples to help you out.

However, before we go into the details, it is important to observe and internalize some overall project goals.

"},{"location":"dev/intro/#goals","title":"Goals","text":"

It is important, that all developers agree and stick to these meta-guidelines. IntelMQ tries to:

The main take away point from the list above is: things MUST stay intuitive and easy. How do you ultimately test if things are still easy? Let them new programmers test-drive your features and if it is not understandable in 15 minutes, go back to the drawing board.

Similarly, if code does not get accepted upstream by the main developers, it is usually only because of the ease-of-use argument. Do not give up, go back to the drawing board, and re-submit again.

"},{"location":"dev/intro/#mailing-list","title":"Mailing list","text":"

There is a separate mailing list for developers to discuss development topics: The IntelMQ-DevArchive is public as well.

"},{"location":"dev/intro/#github","title":"GitHub","text":"

The ideal way to propose changes and additions to IntelMQ is to open a Pull Request on GitHub.

"},{"location":"dev/library/","title":"Use as Library","text":""},{"location":"dev/library/#running-intelmq-as-library","title":"Running IntelMQ as Library","text":""},{"location":"dev/library/#introduction","title":"Introduction","text":"

The feature is specified in IEP007.

"},{"location":"dev/library/#quickstart","title":"Quickstart","text":"

First, import the Python module and a helper. More about the BotLibSettings later.

from intelmq.lib.bot import BotLibSettings\nfrom intelmq.bots.experts.domain_suffix.expert import DomainSuffixExpertBot\n

Then we need to initialize the bot's instance. We pass two parameters:

domain_suffix = DomainSuffixExpertBot('domain-suffix',  # bot id\nsettings=BotLibSettings | {\n'field': 'fqdn',\n'suffix_file': '/usr/share/publicsuffix/public_suffix_list.dat'}\n

As the bot is not fully initialized, we can process messages now. Inserting a message as dictionary: 

queues = domain_suffix.process_message({'source.fqdn': 'www.example.com'})\n

The return value is a dictionary of queues, e.g. the output queue and the error queue. More details below.

The methods accepts multiple messages as positional argument:

domain_suffix.process_message(\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n)\ndomain_suffix.process_message(*[\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n])\n

Select the output queue (as defined in destination_queues), first message, access the field source.domain_suffix: 

>>> output['output'][0]['source.domain_suffix']\n'com'\n

"},{"location":"dev/library/#configuration","title":"Configuration","text":"

Configuration files are not required to run IntelMQ as library. Contrary to IntelMQ normal behavior, if the files runtime.yaml and harmonization.conf do not exist, IntelMQ won't raise any errors. For the harmonization configuration, internal defaults are loaded.

"},{"location":"dev/release/","title":"Release","text":""},{"location":"dev/release/#release-procedure","title":"Release procedure","text":"

General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.

"},{"location":"dev/release/#check-before","title":"Check before","text":""},{"location":"dev/release/#documentation","title":"Documentation","text":"

These apply to all projects:

"},{"location":"dev/release/#intelmq","title":"IntelMQ","text":"

Eventually adapt the default log levels if necessary. Should be INFO for stable releases.

"},{"location":"dev/release/#intelmq-api","title":"IntelMQ API","text":""},{"location":"dev/release/#intelmq-manager","title":"IntelMQ Manager","text":""},{"location":"dev/release/#commit-push-review-and-merge","title":"Commit, push, review and merge","text":"

Commit your changes in a separate branch, the final commit message should start with REL:. Push and create a pull request to the develop branch. Someone else should review the changes. Eventually fix them, make sure the REL: is the last commit, you can also push that one at last, after the reviews.

Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.

"},{"location":"dev/release/#tag-and-release","title":"Tag and release","text":"

Tag the commit with git tag -s version HEAD, merge it into develop, push the branches and the tag. The tag is just a.b.c, not prefixed with v (that was necessary only with SVN a long time ago...).

Go to https://github.com/certtools/intelmq/tags and enter the release notes (from the CHANGELOG) for the new tag, then it's considered a release by GitHub.

"},{"location":"dev/release/#tarballs-and-pypi","title":"Tarballs and PyPI","text":" 
rm -r build/\npython3 setup.py sdist bdist_wheel\n
"},{"location":"dev/release/#documentation_1","title":"Documentation","text":"

Since using mkdocs (see https://docs.intelmq.org) nothing needs to be done anymore.

"},{"location":"dev/release/#packages","title":"Packages","text":"

We are currently using the public Open Build Service instance of openSUSE: http://build.opensuse.org/project/show/home:sebix:intelmq

First, test all the steps first with the unstable-repository and check that at least installations succeed.

"},{"location":"dev/release/#docker-image","title":"Docker Image","text":"

Releasing a new Docker image is very easy.

"},{"location":"dev/release/#announcements","title":"Announcements","text":"

Announce the new version at the mailinglists intelmq-users, intelmq-dev. For bigger releases, probably also at IHAP, Twitter, etc. Ask your favorite social media consultant.

"},{"location":"dev/release/#prepare-new-version","title":"Prepare new version","text":"

Increase the version in intelmq/version.py and declare it as alpha version. Add the new version in intelmq/lib/upgrades.py. Add a new entry in debian/changelog with dch -v [version] -c debian/changelog.

Add new entries to CHANGELOG.md and NEWS.md.

"},{"location":"dev/release/#intelmq_1","title":"IntelMQ","text":"

For CHANGELOG.md:

### Configuration\n\n### Core\n\n### Development\n\n### Data Format\n\n### Bots\n#### Collectors\n\n#### Parsers\n\n#### Experts\n\n#### Outputs\n\n### Documentation\n\n### Packaging\n\n### Tests\n\n### Tools\n\n### Contrib\n\n### Known issues\n

And for NEWS.md:

### Requirements\n\n### Tools\n\n### Data Format\n\n### Configuration\n\n### Libraries\n\n### Postgres databases\n
"},{"location":"dev/release/#intelmq-api_1","title":"IntelMQ API","text":"

An empty section of CHANGELOG.rst.

"},{"location":"dev/release/#intelmq-manager_1","title":"IntelMQ Manager","text":"

For CHANGELOG.md:

### Pages\n\n#### Landing page\n\n#### Configuration\n\n#### Management\n\n#### Monitor\n\n#### Check\n\n### Documentation\n\n### Third-party libraries\n\n### Packaging\n\n### Known issues\n

And an empty section in the NEWS.md file.

"},{"location":"dev/structure/","title":"Structure","text":""},{"location":"dev/structure/#system-overview","title":"System Overview","text":"

In the intelmq/lib/ directory you can find some libraries:

"},{"location":"dev/structure/#code-architecture","title":"Code Architecture","text":""},{"location":"dev/testing/","title":"Testing","text":""},{"location":"dev/testing/#testing","title":"Testing","text":""},{"location":"dev/testing/#additional-test-requirements","title":"Additional test requirements","text":"

Libraries required for tests are listed in the setup.py file. You can install them with pip:

pip3 install -e .[development]\n

or the package management of your operating system.

"},{"location":"dev/testing/#run-the-tests","title":"Run the tests","text":"

All changes have to be tested and new contributions should be accompanied by according unit tests. Please do not run the tests as root just like any other IntelMQ component for security reasons. Any other unprivileged user is possible.

You can run the tests by changing to the directory with IntelMQ repository and running either unittest or pytest. For virtual environment installation, please activate it and omit the sudo -u from examples below:

cd $INTELMQ_REPO\nsudo -u intelmq python3 -m unittest {discover|filename}  # or\nsudo -u intelmq pytest [filename]\nsudo -u intelmq python3 setup.py test  # uses a build environment (no external dependencies)\n

Some bots need local databases to succeed. If you only want to test one explicit test file, give the file path as argument.

There are multiple GitHub Action Workflows setup for automatic testing, which are triggered on pull requests. You can also easily activate them for your forks.

"},{"location":"dev/testing/#environment-variables","title":"Environment variables","text":"

There are a bunch of environment variables which switch on/off some tests:

Environment\u00a0Variable\u00a0Name Description INTELMQ_TEST_DATABASES databases such as postgres, elasticsearch, mongodb are not tested by default. Set this environment variable to 1 to test those bots. These tests need preparation, e.g. running databases with users and certain passwords etc. Have a look at the .github/workflows/unittests.yml and the corresponding .github/workflows/scripts/setup-full.sh in IntelMQ's repository for steps to set databases up. INTELMQ_SKIP_INTERNET tests requiring internet connection will be skipped if this is set to 1. INTELMQ_SKIP_REDIS redis-related tests are ran by default, set this to 1 to skip those. INTELMQ_TEST_EXOTIC some bots and tests require libraries which may not be available, those are skipped by default. To run them, set this to 1. INTELMQ_TEST_REDIS_PASSWORD Set this value to the password for the local redis database if needed. INTELMQ_LOOKYLOO_TEST Set this value to run the lookyloo tests. Public lookyloo instance will be used as default. INTELMQ_TEST_INSTALLATION Set this value to run tests which require a local IntelMQ installation, such as for testing the command lines tools relying on configuration files, dump files etc.

For example, to run all tests you can use:

INTELMQ_TEST_DATABASES=1 INTELMQ_TEST_EXOTIC=1 pytest intelmq/tests/\n
"},{"location":"dev/testing/#configuration-test-files","title":"Configuration test files","text":"

The tests use the configuration files in your working directory, not those installed in /opt/intelmq/etc/ or /etc/. You can run the tests for a locally changed intelmq without affecting an installation or requiring root to run them.

"},{"location":"tutorials/intelmq-manager/","title":"Using IntelMQ Manager","text":""},{"location":"tutorials/intelmq-manager/#tutorial-on-using-intelmq-manager","title":"Tutorial on using IntelMQ Manager","text":"

Bug

This section of the documentation is currently incomplete and will be updated later.

"},{"location":"unsorted/botnet-concept/","title":"Botnet concept","text":""},{"location":"unsorted/botnet-concept/#botnet-concept","title":"Botnet Concept","text":"

The \\\"botnet\\\" represents all currently configured bots which are explicitly enabled. It is, in essence, the graph of the bots which are connected together via their input source queues and destination queues.

To get an overview which bots are running, use intelmqctl status or use the IntelMQ Manager. Set \"enabled\": true in the runtime configuration to add a bot to the botnet. By default, bots will be configured as \"enabled\": true. See bots{.interpreted-text role=\"doc\"} for more details on configuration.

Disabled bots can still be started explicitly using intelmqctl start <bot_id>, but will remain in the state disabled if stopped (and not be implicitly enabled by the start command). They are not started by intelmqctl start in analogy to the behavior of widely used initialization systems.

"},{"location":"unsorted/intelmq-3.0-architecture/","title":"Intelmq 3.0 architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#idea-list-and-architecture-of-intelmq-30","title":"Idea list and architecture of IntelMQ 3.0","text":"

Authors: Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at

"},{"location":"unsorted/intelmq-3.0-architecture/#use-cases","title":"Use-cases","text":"

XXX fill in a complete list of use cases XXX

"},{"location":"unsorted/intelmq-3.0-architecture/#certs","title":"CERTs","text":"

No direct access to networks in constituency.

"},{"location":"unsorted/intelmq-3.0-architecture/#data-collection","title":"Data collection","text":""},{"location":"unsorted/intelmq-3.0-architecture/#distribution-of-information","title":"Distribution of information","text":""},{"location":"unsorted/intelmq-3.0-architecture/#national-cert","title":"National CERT","text":"

Work is based heavily on Geolocation

"},{"location":"unsorted/intelmq-3.0-architecture/#sector-cert","title":"Sector CERT","text":"

Work is based on known constituents, sector information, lists of IP address ranges and domains, company & organisation names.

"},{"location":"unsorted/intelmq-3.0-architecture/#socs-and-nocs","title":"SOCs and NOCs","text":"

Goal is the protection of internal known networks only. Direct access to the networks.

Involves collecting information from internal infrastructure, matching IoCs to internal infrastructure, using IoCs for active protection.

"},{"location":"unsorted/intelmq-3.0-architecture/#data-science-and-research","title":"Data science and research","text":""},{"location":"unsorted/intelmq-3.0-architecture/#users","title":"Users","text":"

XXX fill in a complete list of use cases XXX

"},{"location":"unsorted/intelmq-3.0-architecture/#restful-api","title":"RESTful API","text":"

For automation purposes, we will need a typical RESTful API to manage, control, monitor the IntelMQ \"botnet\" and read and set configs. See #1424

"},{"location":"unsorted/intelmq-3.0-architecture/#ux","title":"UX","text":""},{"location":"unsorted/intelmq-3.0-architecture/#devops-sysadmin-perspective","title":"Devops/ Sysadmin perspective","text":""},{"location":"unsorted/intelmq-3.0-architecture/#docker","title":"Docker","text":"

Task: create a setup where each bot MAY run in a docker container

Background: It might make sense to be able to run each bot in a docker container since it fits with a lot of new paradigms in orchestration. With a proper template, each bot running in a docker container could send its logs to some central logger (for example splunk or similar) and the sysadmin/devops teams which are already using these systems for monitoring alerts can properly fit the IntelMQ logs and alerts to their regular daily routine. Docker also allows the sysadmin/devops folks to centrally manage the system.

Think about: how do we integrate the pipeline graph?

Category: this feature should be OPTIONAL.

"},{"location":"unsorted/intelmq-3.0-architecture/#tutorials-and-vms-dockers","title":"Tutorials and VMs / dockers","text":"

Task: create tutorials with VMs/docker images.

Background: We are missing good tutorials (\"playbooks\") on how to run certain workflows via IntelMQ. Ideally, we would offer ready-made VMs/docker images where people who want to try out IntelMQ (and consequently adapt the setup to their own needs). This also helps teachers/presenters who want to demo IntelMQ.

Specifically we would like to have: * how to process shadowserver feeds * how to process shodan data * how to process n6 data

Think about: shadowserver already created some training material. Build on this.

Category: OPTIONAL component, but highly needed.

"},{"location":"unsorted/intelmq-3.0-architecture/#architecture","title":"Architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#message-queue","title":"Message queue","text":"

Task: Create a Kafka MQ backend: add Kafka as a replaceable MQ for IntelMQ 3.0

Background: IntelMQ 2.0 supports AMQP (RabbitMQ) next to redis as a message queue. Many organisations use Kafka internally. Support connecting to their other work flows.

Think about: Using Apache Pulsar

Category: SHOULD

"},{"location":"unsorted/intelmq-3.0-architecture/#notification-settings","title":"Notification settings","text":"

Task: Keep notification settings per event: Where to (destination mail/host address), how (protocol, authentication (SSL client certificate), etc), how often/time information (intervals etc.)

Background: CERTs (and potentially other groups of users) need to specify where the events should be sent to, how often etc. Currently only destination email addresses can be saved (source.abuse_contact), which is not enough for most use-cases. There exist some custom solutions (e.g. notify boolean at cert.at (to be changed), extra.processing dictionary at BSI), but no least common denominator.

See also https://github.com/certtools/intelmq/issues/758

Category: this feature should be OPTIONAL but is NEEDED by several users.

"},{"location":"unsorted/intelmq-3.0-architecture/#configuration-parameter-handling-in-bots-and-a-bots-unified-documentation","title":"Configuration parameter handling in Bots and a bot's unified documentation","text":"

Task: Handle bots' configuration parameters by the core, providing type sanitation, checks, default values and documentation.

Background: Currently every bot needs to handle these issues itself, but many of these checks could be done centrally in a generic way. At upgrades, new configuration might get introduced and the bots need to provide defaults values although they are available in BOTS. Error handling on parameters must be done for every bot on itself. Documentation is not available to the Bots, not available in BOTS and the Manager. There are 3 places for parameters where the available information is spread: BOTS, Bots.md and the bots' code.

"},{"location":"unsorted/intelmq-3.0-architecture/#automatic-monitoring-management-handling-full-load-situations","title":"Automatic Monitoring & Management: Handling full load situations","text":"

Task: Create a solution to prevent system over-loading (only for Redis).

Background: If too much data is ingested, collected or enriched, the system can easily run out of memory. This quickly causes major operation troubles and data loss, needing manual intervention.

See also: https://github.com/certtools/intelmq/issues/709

"},{"location":"unsorted/intelmq-3.0-architecture/#making-intelmq-plug-able-and-getting-rid-of-bots","title":"Making intelmq plug-able and getting rid of BOTS","text":"

Task: Allow installation of IntelMQ bots, meaning the deprecation of the centralized BOTS file and a generated documentation.

Background: Adapting IntelMQ to specific needs also means the development of specific bots which might not part of the public repository. Adding them to an existing IntelMQ installation is currently only possible by cloning the repository and adding the code there, not by just providing/installing the required code (because of BOTS and central documentation).

See also https://github.com/certtools/intelmq/issues/972

"},{"location":"unsorted/intelmq-3.0-architecture/#exposing-a-plug-in-or-hooking-api","title":"Exposing a plug-in or hooking API","text":"

Task: Provide an hooking API for the core classes.

Background: Adapting IntelMQ to specific can require adaptions in the Core classes' code. Instead of making the changes/extensions in the core itself, we can provide a hook system allowing to call (or replace?) functions at specific steps. For example custom monitoring.

"},{"location":"unsorted/intelmq-3.0-architecture/#grouping-of-events","title":"Grouping of events","text":"

Task: Provide possibilities to assign an event to a group of events.

Background: Several IoCs part of one MISP Event. Grouping of similar events to one group for outputs (e.g. one CSV file per Network).

See also: https://github.com/certtools/intelmq/issues/751

"},{"location":"unsorted/intelmq-3.0-architecture/#data-format-multiple-values","title":"Data Format: Multiple values","text":"

Task: Allow multiple values for (some) fields in the data format.

Background: In some cases one value per field is not enough, for example for Domain -> IP address lookups. Other formats like IDEA and n6 support this.

See also: https://github.com/certtools/intelmq/issues/543 https://github.com/certtools/intelmq/issues/373

"},{"location":"unsorted/intelmqctl-more/","title":"Intelmqctl more","text":""},{"location":"unsorted/intelmqctl-more/#command-line-interface-intelmqctl","title":"Command-line interface: intelmqctl","text":"

Syntax see intelmqctl -h

"},{"location":"unsorted/intelmqctl-more/#reloading","title":"Reloading","text":"

Whilst restart is a mere stop & start, performing intelmqctl reload <bot_id> will not stop the bot, permitting it to keep the state: the same common behavior as for ( Linux) daemons. It will initialize again (including reading all configuration again) after the current action is finished. Also, the rate limit/sleep is continued (with the new time) and not interrupted like with the restart command. So if you have a collector with a rate limit of 24 h, the reload does not trigger a new fetching of the source at the time of the reload, but just 24 h after the last run -- with the new configuration. Which state the bots are keeping depends on the bots of course.

"},{"location":"unsorted/intelmqctl-more/#forcing-reset-pipeline-and-cache-be-careful","title":"Forcing reset pipeline and cache (be careful)","text":"

If you are using the default broker (Redis), in some test situations you may need to quickly clear all pipelines and caches. Use the following procedure:

redis-cli FLUSHDB\nredis-cli FLUSHALL\n
"},{"location":"unsorted/intelmqctl-more/#management","title":"Management","text":"

IntelMQ has a modular structure consisting of bots. There are four types of bots:

Each bot has one source queue (except collectors) and can have multiple destination queues (except outputs). But multiple bots can write to the same pipeline (queue), resulting in multiple inputs for the next bot.

Every bot runs in a separate process. A bot is identifiable by a bot id.

Currently only one instance (i.e. with the same bot id) of a bot can run at the same time. Concepts for multiprocessing are being discussed, see this issue: Multiprocessing per queue is not supported #186 <186>. Currently you can run multiple processes of the same bot ( with different bot ids) in parallel.

Example: multiple gethostbyname bots (with different bot ids) may run in parallel, with the same input queue and sending to the same output queue. Note that the bot providing the input queue must have the load_balance option set to true.

"},{"location":"user/abuse-contacts/","title":"Abuse Contacts","text":""},{"location":"user/abuse-contacts/#abuse-contact-look-ups","title":"Abuse-contact look-ups","text":"

The right decision whom to contact about a specific incident is vital to get the incident resolved as quick as possible. Different types of events may required different abuse-contact to be selected. For example, issues about a device, e.g. a vulnerability in the operating system or an application, is better sent to the hoster which can inform the server administrator. For website-related issues, like defacements or phishing, the domain owner (maintaining the content of the website) could be the better and more direct contact. Additionally, different CERT's have different approaches and different contact databases. Multiple information sources have different information, and some sources are more accurate than others. IntelMQ can query multiple sources of abuse-contacts and combine them. Internal databases, like a Constituency Portal provide high-quality and first-hand contact information. The RIPE document Sources of Abuse Contact Information for Abuse Handlers contains a good summary of the complex of themes.

"},{"location":"user/abuse-contacts/#sources-for-abuse-contacts","title":"Sources for abuse-contacts","text":"

All these bots add the queried contacts to the IntelMQ events in the field source.abuse_contact if not state otherwise in the documentation.

"},{"location":"user/abuse-contacts/#sources-for-domain-based-abuse-contacts","title":"Sources for domain-based abuse-contacts","text":"

These bots are suitable for domain-based abuse-contact look-ups.

"},{"location":"user/abuse-contacts/#sources-for-ip-address-based-abuse-contacts","title":"Sources for IP address-based abuse-contacts","text":"

These bots are suitable for IP address and ASN based abuse-contact look-ups.

"},{"location":"user/abuse-contacts/#generic-sources-for-abuse-contacts","title":"Generic sources for abuse-contacts","text":""},{"location":"user/abuse-contacts/#helpful-other-bots-for-pre-processing","title":"Helpful other bots for pre-processing","text":""},{"location":"user/abuse-contacts/#combining-the-lookup-approaches","title":"Combining the lookup approaches","text":"

In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. IntelMQ's modularity provides methods to arrange and configure the bots as needed. Among others, the following bots can help in getting the best result:

"},{"location":"user/api/","title":"API","text":""},{"location":"user/api/#using-intelmq-api","title":"Using IntelMQ API","text":"

Bug

This section of the documentation is currently incomplete and will be added later.

"},{"location":"user/api/#usage-from-programs","title":"Usage from programs","text":"

The IntelMQ API can also be used from programs, not just browsers. To do so, first send a POST-Request with JSON-formatted data to http://localhost/intelmq/v1/api/login/

{\n    \"username\": \"$your_username\",\n    \"password\": \"$your_password\"\n}\n

With valid credentials, the JSON-formatted response contains the login_token. This token can be used like an API key in the Authorization header for the next API calls:

Authorization: $login_token\n

Here is a full example using curl:

  1. Authentication step: 

    curl --location --request POST \"http://localhost/intelmq/v1/api/login/\" \\\n     --header \"Content-Type: application/x-www-form-urlencoded\" \\\n     --data-urlencode \"username=$username\"\\\n     --data-urlencode \"password=$password\"\n
    {\"login_token\":\"68b329da9893e34099c7d8ad5cb9c940\",\"username\":\"$username\"}\n

  2. Using the login token to fetch data: 

    curl --location \"http://localhost/intelmq/v1/api/version\" \\\n     --header \"Authorization: 68b329da9893e34099c7d8ad5cb9c940\"\n
    {\"intelmq\":\"3.0.0rc1\",\"intelmq-manager\":\"2.3.1\"}\n

The same approach also works for Ansible, as you can see here:

  1. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/00_registerauth.yml#L1-L9
  2. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/02_queuestatus.yml#L1-L5
"},{"location":"user/bots/","title":"Bots","text":""},{"location":"user/bots/#bots-inventory","title":"Bots Inventory","text":"

This document contains complete reference of bots implemented by IntelMQ and how to configure them from the users perspective (meaning via IntelMQ Manager). Some of the bots are intended for general use and some of them are for processing particular data sources.

"},{"location":"user/bots/#individual-bot-configuration","title":"Individual Bot Configuration","text":"

Each bot has it's own configuration. The configuration consists of two types of parameters:

"},{"location":"user/bots/#generic-parameters","title":"Generic Parameters","text":"

These parameters must be set for each bot (at least the required ones).

"},{"location":"user/bots/#id","title":"id","text":"

(required, string) This must be a unique identifier. Commonly it looks something like this: abusech-feodo-tracker-collector. It is safer to avoid using spaces.

"},{"location":"user/bots/#name","title":"name","text":"

(required, string) Human readable name of the bot.

"},{"location":"user/bots/#description","title":"description","text":"

(required, string) The description of the bot.

"},{"location":"user/bots/#module","title":"module","text":"

(required, string) The executable (should be in PATH environment variable) which will be started.

"},{"location":"user/bots/#group","title":"group","text":"

(optional, string) The group of the bot. Can be Collector, Parser, Expert or Output. Only used for visualization by other tools.

"},{"location":"user/bots/#enabled","title":"enabled","text":"

(optional, boolean) Whether the bot will start when the whole botnet is started. You can still start a disabled bot explicitly. Defaults to true.

"},{"location":"user/bots/#run_mode","title":"run_mode","text":"

(optional, string) There are two run modes, continuous or scheduled. In the first case, the bot will be running forever until stopped or exits because of errors (depending on the configuration). In the latter case, the bot will stop after one successful run. This is especially useful when scheduling bots via cron or systemd. Check Configuration section for more details. Defaults to continuous.

"},{"location":"user/bots/#http-parameters","title":"HTTP Parameters","text":"

Common HTTP runtime parameters used in multiple bots.

"},{"location":"user/bots/#http_timeout_sec","title":"http_timeout_sec","text":"

(optional, float) A tuple of floats or only one float describing the timeout (seconds) of the HTTP connection. Can be a tuple of two floats (read and connect timeout) or just one float (applies for both timeouts). See also https://requests.readthedocs.io/en/master/user/advanced/#timeouts. Defaults to 30.

"},{"location":"user/bots/#http_timeout_max_tries","title":"http_timeout_max_tries","text":"

(optional, integer) An integer depicting how many times a connection is retried, when a timeout occurred. Defaults to 3.

"},{"location":"user/bots/#http_username","title":"http_username","text":"

(optional, string) Username for basic HTTP authentication.

"},{"location":"user/bots/#http_password","title":"http_password","text":"

(optional, string) Password for basic HTTP authentication.

"},{"location":"user/bots/#http_proxy","title":"http_proxy","text":"

(optional, string) Proxy to use for HTTP.

"},{"location":"user/bots/#https_proxy","title":"https_proxy","text":"

(optional, string) Proxy to use for HTTPS.

"},{"location":"user/bots/#http_user_agent","title":"http_user_agent","text":"

(optional, string) User-Agent to be used for HTTP requests.

"},{"location":"user/bots/#http_verify_cert","title":"http_verify_cert","text":"

(optional, boolean/string) Path to trusted CA bundle or directory, false to ignore verifying SSL certificates, or true to verify SSL certificates. Defaults to true.

"},{"location":"user/bots/#ssl_client_certificate","title":"ssl_client_certificate","text":"

(optional, string) Path to client certificate to use for TLS connections.

"},{"location":"user/bots/#ssl_ca_certificate","title":"ssl_ca_certificate","text":"

(optional, string) Path to trusted CA certificate. Only used by some bots.

"},{"location":"user/bots/#cache-parameters","title":"Cache Parameters","text":"

Common Redis cache runtime parameters used in multiple bots (mainly lookup experts).

"},{"location":"user/bots/#redis_cache_host","title":"redis_cache_host","text":"

(required, string) Hostname of the Redis database.

"},{"location":"user/bots/#redis_cache_port","title":"redis_cache_port","text":"

(required, string) Port of the Redis database.

"},{"location":"user/bots/#redis_cache_db","title":"redis_cache_db","text":"

(required, integer) Database number.

"},{"location":"user/bots/#redis_cache_ttl","title":"redis_cache_ttl","text":"

(required, integer) TTL used for caching.

"},{"location":"user/bots/#redis_cache_password","title":"redis_cache_password","text":"

(optional, string) Password for the Redis database.

"},{"location":"user/bots/#collector-bots","title":"Collector Bots","text":"

Multihreading is disabled for all Collectors, as this would lead to duplicated data.

"},{"location":"user/bots/#feed-parameters","title":"Feed Parameters","text":"

These runtime parameters must be set for each collector bot (at least the required ones).

"},{"location":"user/bots/#name_1","title":"name","text":"

(required, string) Name of the feed (feed.name).

"},{"location":"user/bots/#accuracy","title":"accuracy","text":"

(optional, float) Accuracy of the data from the feed (feed.accuracy).

"},{"location":"user/bots/#code","title":"code","text":"

(optional, string) Code for the feed (feed.code).

"},{"location":"user/bots/#documentation","title":"documentation","text":"

(optional, string) Link to documentation for the feed (feed.documentation).

"},{"location":"user/bots/#provider","title":"provider","text":"

(optional, string) Name of the provider of the feed (feed.provider).

"},{"location":"user/bots/#rate_limit","title":"rate_limit","text":"

(optional, integer) Time interval (in seconds) between fetching data if applicable. Defaults to 0.

"},{"location":"user/bots/#alien-vault-otx","title":"Alien Vault OTX","text":"

Collects report messages from Alien Vault OTX.

Module: intelmq.bots.collectors.alienvault_otx.collector

Requirements

Install the library from GitHub, as there is no package in PyPi:

pip3 install -r intelmq/bots/collectors/alienvault_otx/REQUIREMENTS.txt\n

Parameters (also expects feed parameters):

api_key

(required, string) API Key

modified_pulses_only

(optional, boolean) Whether to get only modified pulses instead of all. Defaults to false.

interval

(optional, integer) When modified_pulses_only is set, define the time in hours (integer value) to get modified pulses since then. Defaults to 24 (hours).

"},{"location":"user/bots/#amqp","title":"AMQP","text":"

This bot collects data from (remote) AMQP servers, for both IntelMQ as well as external data. Currently only fetching from a queue is supported can be extended in the future. Messages will be acknowledge at AMQP after it is sent to the pipeline. Requires the pika library, minimum version 1.0.0.

Module: intelmq.bots.collectors.amqp.collector_amqp

Parameters (also expects feed parameters):

connection_host

(optional, string) Hostname of the AMQP server. Defaults to 127.0.0.1.

connection_port

(optional, integer) Port of the AMQP server. Defaults to 5672.

connection_attempts

(optional, integer) The number of connection attempts to the defined server. Defaults to 3.

connection_heartbeat

(optional, integer) Heartbeat to server (seconds). Defaults to 3600.

connection_vhost

(optional, string) Virtual host to connect, on an HTTP(S) connection would be .

expect_intelmq_message

(optional, boolean) This parameter denotes whether the the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false.

queue_name

(optional, string) The name of the queue to fetch the data from.

username

(optional, string) Username for authentication to the AMQP server.

password

(optional, string) Password for authentication to the AMQP server.

use_ssl

(optional, boolean) Use of TLS for the connection. Make sure to also set the correct port. Defaults to false.

"},{"location":"user/bots/#api","title":"API","text":"

This bot collects data from HTTP or Socket REST API. The API is available at /intelmq/push when the HTTP interface is used. Requires the tornado library.

Module: intelmq.bots.collectors.api.collector

Parameters (also expects feed parameters):

port

(optional, integer) The local port at which the API is available. Defaults to 5000.

use_socket

(optional, boolean) If true, the socket will be opened at the location given with socket_path. Defaults to false.

socket_path

(optional, string) Location of the socket. Defaults to /tmp/imq_api_default_socket.

socket_perms

(optional, octal integer) Unix permissions to grant to the socket file. Default: 600

socket_group

(optional, string) Name of group to change group ownership of socket file to.

"},{"location":"user/bots/#generic-url-fetcher","title":"Generic URL Fetcher","text":"

This bot collects data from remote hosts using HTTP protocol. If the HTTP response' status code is not 2xx, this is treated as error. In Debug logging level, the request's and response's headers and body are logged for further inspection.

Module: intelmq.bots.collectors.http.collector_http

Parameters (also expects feed parameters and HTTP parameters):

http_url

(required, string) Location of the resource to download.

http_url_formatting

(optional, boolean/object) When true, {time[format]} will be replaced by the current time in local timezone formatted by the given format. E.g. if the URL is http://localhost/{time[%Y]}, then the resulting URL is http://localhost/2019 for the year 2019. ( Python's Format Specification Mini-Language is used for this.). You may use a JSON specifying time-delta parameters to shift the current time accordingly. For example use days: -1 for the yesterday's date; the URL http://localhost/{time[%Y-%m-%d]} will get translated to http://localhost/2018-12-31 for the 1st Jan of 2019. Defaults to false.

extract_files

(optional, boolean/array of strings) If true, the retrieved (compressed) file or archived will be uncompressed/unpacked and the files are extracted. If the parameter is a list of strings, only the files matching the filenames are extracted. Extraction handles gzipped files and both compressed and uncompressed tar-archives as well as zip archives. For extracted files, every extracted file is sent in it's own report. Every report has a field named extra.file_name with the file name in the archive the content was extracted from. Defaults to false.

verify_pgp_signatures

(optional, boolean) When true, signature file is downloaded and report file is checked. On error (missing signature, mismatch, ...), the error is logged and the report is not processed. Public key has to be imported in local keyring. This requires the python-gnupg library. Defaults to false.

signature_url

(optional, string) Location of the signature file for the downloaded content.

signature_url_formatting

(optional, boolean/object) Same as http_url_formatting. Defaults to false.

gpg_keyring

(optional, string) If specified, the string represents path to keyring file. Otherwise the PGP keyring file of the current intelmq user is used.

"},{"location":"user/bots/#generic-url-stream-fetcher","title":"Generic URL Stream Fetcher","text":"

Opens a streaming connection to the URL and collects the received lines.

If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.

Module: intelmq.bots.collectors.http.collector_http_stream

Parameters (also expects feed parameters and HTTP parameters):

Uses the same parameters as Generic URL Fetcher. The parameter http_timeout_max_tries is of no use in this collector.

strip_lines

(optional, boolean) Whether the single lines should be stripped (removing whitespace from the beginning and the end of the line) or not. Defaults to true.

"},{"location":"user/bots/#generic-mail-url-fetcher","title":"Generic Mail URL Fetcher","text":"

Extracts URLs from e-mail messages and downloads the content from the URLs.

The resulting reports contain the following special fields:

Chunking

For line-based inputs the bot can split up large reports into smaller chunks. This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB. To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot. Specifically, to configure a large file input to work around Redis size limitation set chunk_size to something like 384000000 (~384 MB).

Module: intelmq.bots.collectors.mail.collector_mail_url

Parameters (also expects feed parameters and HTTP parameters):

mail_host

(required, string) Hostname of the mail server.

mail_port

(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.

mail_user

(required, string) Username of the email account.

mail_password

(required, string) Password associated with the user account.

mail_ssl

(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.

folder

(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.

subject_regex

(optional, string) Regular expression to look for in the e-mail subject.

url_regex

(optional, string) Regular expression of the feed URL to look for in the e-mail body.

sent_from

(optional, string) Filter messages by the sender.

sent_to

(optional, string) Filter messages by the recipient.

ssl_ca_certificate

(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.

"},{"location":"user/bots/#generic-mail-attachment-fetcher","title":"Generic Mail Attachment Fetcher","text":"

This bot collects messages from mailboxes and downloads the attachments.

The resulting reports contains the following special fields:

Module: intelmq.bots.collectors.mail.collector_mail_attach

Parameters (also expects feed parameters):

mail_host

(required, string) Hostname of the mail server.

mail_port

(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.

mail_user

(required, string) Username of the email account.

mail_password

(required, string) Password associated with the user account.

mail_ssl

(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.

folder

(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.

subject_regex

(optional, string) Regular expression to look for in the e-mail subject.

attach_regex

(optional, string) Regular expression of the name of the attachment. Defaults to csv.zip.

extract_files

(optional, boolean) Whether to extract compress files from the attachment. Defaults to true.

sent_from

(optional, string) Only process messages sent from this address. Defaults to null (any sender).

sent_to

(optional, string) Only process messages sent to this address. Defaults to null (any recipient).

ssl_ca_certificate

(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. By default, no certificate is used.

"},{"location":"user/bots/#generic-mail-body-fetcher","title":"Generic Mail Body Fetcher","text":"

This bot collect messages from mailboxes, forwards the bodies as reports. Each non-empty body with the matching content type is sent as individual report.

The resulting reports contains the following special fields:

Module: intelmq.bots.collectors.mail.collector_mail_body

Parameters (also expects feed parameters):

mail_host

(required, string) Hostname of the mail server.

mail_port

(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.

mail_user

(required, string) Username of the email account.

mail_password

(required, string) Password associated with the user account.

mail_ssl

(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.

folder

(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.

subject_regex

(optional, string) Regular expression to look for in the e-mail subject.

url_regex

(optional, string) Regular expression of the feed URL to look for in the e-mail body.

sent_from

(optional, string) Filter messages by the sender.

sent_to

(optional, string) Filter messages by the recipient.

ssl_ca_certificate

(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.

content_types

(optional, boolean/array of strings) Which bodies to use based on the content_type. Defaults to true (same as ['html', 'plain']) for all:

"},{"location":"user/bots/#github-api","title":"Github API","text":"

Collects files matched by regular expression from GitHub repository via the GitHub API. Optionally with GitHub credentials, which are used as the Basic HTTP authentication.

Workflow

The optional authentication parameters provide a high limit of the GitHub API requests. With the git hub user authentication, the requests are rate limited to 5000 per hour, otherwise to 60 requests per hour.

The collector recursively searches for regex-defined files in the provided repository. Additionally it adds extra file metadata defined by the extra_fields.

The bot always sets the url, from which downloaded the file, as feed.url.

Module: intelmq.bots.collectors.github_api.collector_github_contents_api

Parameters (also expects feed parameters):

personal_access_token

(required, string) GitHub account personal access token GitHub documentation: Creating a personal access token

repository

(required, string) GitHub target repository (<USER>/<REPOSITORY>)

regex

(optional, string) Valid regular expression of target files within the repository. Defaults to .*.json.

extra_fields

(optional, array of strings) Comma-separated list of extra fields from GitHub contents API.

"},{"location":"user/bots/#file","title":"File","text":"

This bot is capable of reading files from the local file-system. This is handy for testing purposes, or when you need to react to spontaneous events. In combination with the Generic CSV parser this should work great.

The resulting reports contains the following special fields:

Chunking

Additionally, for line-based inputs the bot can split up large reports into smaller chunks.

This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB.

To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot.

Specifically, to configure a large file input to work around Redis' size limitation set chunk_size to something like 384000, i.e., ~384 MB.

Workflow

The bot loops over all files in path and tests if their file name matches *postfix, e.g. *.csv. If yes, the file will be read and inserted into the queue.

If delete_file is set, the file will be deleted after processing. If deletion is not possible, the bot will stop.

To prevent data loss, the bot also stops when no postfix is set and delete_file was set. This cannot be overridden.

The bot always sets the file name as feed.url.

Module: intelmq.bots.collectors.file.collector_file

Parameters (also expects feed parameters):

path

(required, string) Path to file.

postfix

(required, string) The postfix (file ending) of the files to look for. For example [.csv].

delete_file

(optional, boolean) Whether to delete the file after reading. Defaults to false.

"},{"location":"user/bots/#fireeye","title":"FireEye","text":"

This bot is capable of collecting hashes and URLs from a FireEye MAS appliance.

The Python library xmltodict is required to run this bot.

Workflow

The bot collects all alerts which occurred during specified duration. After this we make a second call and check if there is additional information like domains and hashes available. After collecting the openioc data we send this information to the Fireeye parser.

Module: intelmq.bots.collectors.fireeye.collector_fireeye

Parameters (also expects feed parameters):

host

(required, string) DNS name of the target appliance.

request_duration

(required, string) Allowed values: 24_hours or 48_hours. Length of the query in past eg. collect alerts from last 24hours/48hours.

http_username

(required, string) Password for authentication.

http_password

(required, string) Username for authentication.

"},{"location":"user/bots/#kafka","title":"Kafka","text":"

Requires the kafka python library.

Module: intelmq.bots.collectors.kafka.collector

Parameters (also expects feed parameters):

topic

(required, string) Kafka topic the collector should get messages from.

bootstrap_servers

(required, string) Kafka server(s) and port the collector should connect to. Defaults to localhost:9092

ssl_check_hostname

(optional, boolean) Whether to verify TLS certificates. Defaults to true.

ssl_client_certificate

(optional, string) Path to client certificate to use for TLS connections.

ssl_ca_certificate

(optional, string) Path to trusted CA certificate.

"},{"location":"user/bots/#misp-generic","title":"MISP Generic","text":"

Collects messages from MISP, a malware information sharing platform server.

Workflow

This collector will search for events on a MISP server that have a [to_process] tag attached to them (see the [misp_tag_to_process] parameter) and collect them for processing by IntelMQ. Once the MISP event has been processed the [to_process] tag is removed from the MISP event and a [processed] tag is then attached (see the [misp_tag_processed] parameter).

NB. The MISP tags must be configured to be 'exportable' otherwise they will not be retrieved by the collector.

Module: intelmq.bots.collectors.misp.collector

Parameters (also expects feed parameters):

misp_url

(required, string) URL of MISP server (with trailing '/').

misp_key

(required, string) MISP Authkey.

misp_tag_to_process

(required, string) MISP tag for events to be processed.

misp_tag_processed

(optional, string) MISP tag for processed events.

http_verify_cert

(optional, boolean) Verify the TLS certificate of the server. Defaults to true.

"},{"location":"user/bots/#request-tracker","title":"Request Tracker","text":"

Request Tracker Collector fetches attachments from an RTIR instance.

This rt bot will connect to RT and inspect the given search_queue for tickets matching all criteria in search_*, Any matches will be inspected. For each match, all (RT-) attachments of the matching RT tickets are iterated over and within this loop, the first matching filename in the attachment is processed. If none of the filename matches apply, the contents of the first (RT-) \"history\" item is matched against the regular expression for the URL (url_regex).

The parameter http_timeout_max_tries is of no use in this collector.

Search

The parameters prefixed with search_ allow configuring the ticket search.

Empty strings and null as value for search parameters are ignored.

File downloads

Attachments can be optionally unzipped, remote files are downloaded with the http_* settings applied.

If url_regex or attachment_regex are empty strings, false or null, they are ignored.

Ticket processing

Optionally, the RT bot can \"take\" RT tickets (i.e. the user is assigned this ticket now) and/or the status can be changed (leave set_status empty in case you don't want to change the status). Please note however that you MUST do one of the following: either \"take\" the ticket or set the status (set_status). Otherwise, the search will find the ticket every time and get stuck in an endless loop.

In case a resource needs to be fetched and this resource is permanently not available (status code is 4xx), the ticket status will be set according to the configuration to avoid processing the ticket over and over. For temporary failures the status is not modified, instead the ticket will be skipped in this run.

Time search

To find only tickets newer than a given absolute or relative time, you can use the search_not_older_than parameter. Absolute time specification can be anything parseable by dateutil, best use a ISO format.

Relative must be in this format: [NUMBER] [TIMESPAN]s, e.g. 3 days. Timespan can be hour, day, week, month or year. Trailing 's' is supported for all timespans. Relative times are subtracted from the current time directly before the search is performed.

The resulting reports contains the following special fields:

Requirements

You need the rt-library >= 1.9 and < 3.0 from from nic.cz, available via pypi: pip3 install rt<3

Module: intelmq.bots.collectors.rt.collector_rt

Parameters (also expects feed parameters and HTTP parameters):

extract_attachment

(optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.

extract_download

(optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.

uri

(optional, string) URL of the REST interface of the RT. Defaults to http://localhost/rt/REST/1.0.

user

(optional, string) RT username. Defaults to intelmq.

password

(optional, string) RT password. Defaults to password.

search_not_older_than

(optional, string) Absolute time (use ISO format) or relative time, e.g. 3 days.

search_owner

(optional, string) Owner of the ticket to search for. Defaults to nobody.

search_queue

(optional, string) Queue of the ticket to search for. Defaults to Incident Reports.

search_requestor

(optional, string) E-mail address of the requestor.

search_status

(optional, string) Status of the ticket to search for. Defaults to new.

search_subject_like

(optional, string/array of strings) Part of the subject of the ticket to search for. Defaults to \"Report\".

search_subject_notlike

(optional, string/array of strings) Exclude subject containing given value, use list for multiple excluding values.

set_status

(optional, string) Status to set the ticket to after processing. Use false or null to keep current status. Defaults to open.

take_ticket

(optional, boolean) Whether to take the ticket. Defaults to true.

url_regex

(optional, string) Regular expression of an URL to search for in the ticket. Defaults to https://dl.shadowserver.org/[a-zA-Z0-9?_-]*.

attachment_regex

(optional, string) Eegular expression of an attachment in the ticket. Defaults to \\.csv\\.zip$.

"},{"location":"user/bots/#rsync","title":"Rsync","text":"

This bot downloads a file via rsync and then load data from downloaded file. Downloaded file is located in var/lib/bots/rsync_collector.

Requires the rsync executable.

Module: intelmq.bots.collectors.rsync.collector_rsync

Parameters (also expects feed parameters):

file

(required, string) The filename to process, combined with rsync_path.

rsync_path

(required, string) Path to the directory of the file. Allowed values are local directory (such as /home/username/) or remote directory (such as <username@remote_host>:/home/username/directory).

rsync_file_path_formatting

(optional, boolean) Whether the file and rsync_path should be formatted by the given format. E.g. if the path is /path/to_file/{time[%Y]}, then the resulting path is /path/to/file/2023 for the year 2023. (Python's Format Specification Mini-Language <https://docs.python.org/3/library/string.html#formatspec> is used for this.). You may use a JSON specifying time-delta <https://docs.python.org/3/library/datetime.html#datetime.timedelta> parameters to shift the current time accordingly. For example use {\"days\": -1} for the yesterday's date; the path /path/to/file/{time[%Y-%m-%d]} will get translated to \"/path/to/file/2018-12-31\" for the 1st Jan of 2023. Defaults to false.

extra_params

(optional, array of strings) A list of extra parameters to pass to rsync.

private_key

(optional, string) Private key to use for rsync authentication.

private_key_path

(optional, string) Path to private key to use for rsync authentication. Use private_key or private_key_path, not both.

strict_host_key_checking

(optional, boolean) Whether the host key should be checked. Defaults to false.

temp_directory

(optional, string) The temporary directory for rsync to use for collected files. Defaults to /opt/intelmq/var/run/{BOT-ID} or /var/run/intelmq/{BOT-ID}.

"},{"location":"user/bots/#shadowserver-reports-api","title":"Shadowserver Reports API","text":"

Connects to the Shadowserver API, requests a list of all the reports for a specific country and processes the ones that are new.

The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).

The resulting reports contain the following special field:

Module: intelmq.bots.collectors.shadowserver.collector_reports_api

Parameters (also expects feed parameters and cache parameters):

apikey

(required, string) Your Shadowserver API key.

secret

(required, string) Your Shadowserver API secret.

reports

(required, string/array of strings) An array of strings (or a list of comma-separated values) of the mailing lists you want to process.

types

(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names defined the the schema. Please see the Supported Reports of the Shadowserver parser for details.

Sample configuration

  shadowserver-collector:\n    description: Our bot responsible for getting reports from Shadowserver\n    enabled: true\n    group: Collector\n    module: intelmq.bots.collectors.shadowserver.collector_reports_api\n    name: Shadowserver_Collector\n    parameters:\n      destination_queues:\n        _default: [shadowserver-parser-queue]\n      file_format: csv\n      api_key: \"$API_KEY_received_from_the_shadowserver_foundation\"\n      secret: \"$SECRET_received_from_the_shadowserver_foundation\"\n    run_mode: continuous\n
"},{"location":"user/bots/#shodan-stream","title":"Shodan Stream","text":"

Queries the Shodan Streaming API.

Requires the shodan library to be installed:

Module: intelmq.bots.collectors.shodan.collector_stream

Parameters (also expects feed parameters and HTTP parameters):

Only the proxy is used (requires shodan-python > 1.8.1). Certificate is always verified.

countries

() A list of countries to query for. If it is a string, it will be spit by ,.

If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.

"},{"location":"user/bots/#tcp","title":"TCP","text":"

TCP is the bot responsible to receive events on a TCP port (ex: from TCP Output of another IntelMQ instance). Might not be working on Python 3.4.6.

Response

TCP collector just sends an \"OK\" message after every received message, this should not pose a problem for an arbitrary input. If you intend to link two IntelMQ instance via TCP, have a look at the TCP output bot documentation.

Module: intelmq.bots.collectors.tcp.collector

Parameters (also expects feed parameters):

ip

(required, string) IP of the destination server.

port

(required, integer) Port of destination server.

"},{"location":"user/bots/#blueliv-crimeserver","title":"Blueliv Crimeserver","text":"

Collects report messages from Blueliv API.

For more information visit https://github.com/Blueliv/api-python-sdk

Module: intelmq.bots.collectors.blueliv.collector_crimeserver

Requirements

Install the required library:

pip3 install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt\n

Parameters (also expects feed parameters):

api_key

(required, string) location of information resource, see https://map.blueliv.com/?redirect=get-started#signup

api_url

(optional, string) The optional API endpoint. Defaults to https://freeapi.blueliv.com.

"},{"location":"user/bots/#calidog-certstream","title":"Calidog Certstream","text":"

A Bot to collect data from the Certificate Transparency Log (CTL). This bot works based on certstream library (https://github.com/CaliDog/certstream-python)

Module: intelmq.bots.collectors.calidog.collector_certstream

Parameters (also expects feed parameters):

"},{"location":"user/bots/#eset-eti","title":"ESET ETI","text":"

Collects data from ESET ETI TAXII server.

For more information visit https://www.eset.com/int/business/services/threat-intelligence/.

Module: intelmq.bots.collectors.eset.collector

Requirements

Install the required cabby library:

pip3 install -r intelmq/bots/collectors/eset/REQUIREMENTS.txt\n

Parameters (also expects feed parameters):

username

(required, string) Your username.

password

(required, string) Your password.

endpoint

(optional, string) Defaults to eti.eset.com.

time_delta

(optional, integer) The time (in seconds) span to look back. Default to 3600.

collection

(required, string) The collection to fetch.

"},{"location":"user/bots/#mcafee-opendxl","title":"McAfee openDXL","text":"

Collects messages via McAfee openDXL.

Module: intelmq.bots.collectors.opendxl.collector

Parameters (also expects feed parameters):

dxl_config_file

(required, string) Path to the the configuration file containing required information to connect.

dxl_topic

(optional, string) Name of the DXL topic to subscribe to. Defaults to /mcafee/event/atd/file/report.

"},{"location":"user/bots/#microsoft-azure","title":"Microsoft Azure","text":"

Collects blobs from Microsoft Azure using their library.

Iterates over all blobs in all containers in an Azure storage. The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).

This bot significantly changed in a backwards-incompatible way in IntelMQ Version 2.2.0 to support current versions of the Microsoft Azure Python libraries. azure-storage-blob>=12.0.0 is required.

Module: intelmq.bots.collectors.microsoft.collector_azure

Parameters (also expects feed parameters and cache parameters):

connection_string

(required, string) Connection string as given by Microsoft.

container_name

(required, string) Name of the container to connect to.

"},{"location":"user/bots/#microsoft-interflow","title":"Microsoft Interflow","text":"

This bot collects files from Microsoft Interflow API.

Iterates over all files available by this API. Make sure to limit the files to be downloaded with the parameters, otherwise you will get a lot of data! The cache is used to remember which files have already been downloaded. Make sure the TTL is high enough, higher than not_older_than.

Module: intelmq.bots.collectors.microsoft.collector_interflow

Parameters (also expects feed parameters):

api_key

(required, string) API generated in their portal.

file_match

(optional, string) Regular expression to match file names.

not_older_than

(optional, integer/datetime) an optional relative (minutes) or absolute time (UTC is assumed) expression to determine the oldest time of a file to be downloaded.

redis_cache_* and especially redis_cache_ttl

Settings for the cache where file names of downloaded files are saved. The cache's TTL must always be bigger than not_older_than.

Additional functionalities

Files are automatically ungzipped if the filename ends with .gz.

"},{"location":"user/bots/#stomp","title":"STOMP","text":"

Collects messages from a STOMP server.

Module: intelmq.bots.collectors.stomp.collector

Requirements

Install the stomp.py library from PyPI:

pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt\n

Alternatively, you may want to install it using your OS's native packaging tools, e.g.:

apt install python3-stomp\n

Apart from that, depending on what STOMP server you connect to, you may need to obtain, from the organization or company owning the server, one or more of the following security/authentication-related resources:

Also, you will need to know an appropriate STOMP destination (aka exchange point), e.g. /exchange/my.example.org/*.*.*.*.

Parameters (also expects feed parameters):

server

(required, string) STOMP server's hostname or IP, e.g. \"n6stream.cert.pl\" (which is default)

port

(optional, integer) STOMP server's port number (default: 61614)

exchange

(required, string) STOMP destination to subscribe to, e.g. \"/exchange/my.org/*.*.*.*\"

heartbeat

(optional, integer) default: 6000

ssl_ca_certificate

(optional, string) Path to CA file, or empty string to load system's default CA certificates

auth_by_ssl_client_certificate

(optional, boolean) Default: true (note: false is needed for new n6 auth)

ssl_client_certificate

(optional, string) Path to client certificate to use for TLS connections.

ssl_client_certificate_key

(optional, string) Path to client private key to use for TLS connections.

username

(optional, string) Username to use.

password

(optional, string) Password to use.

"},{"location":"user/bots/#twitter-remove","title":"Twitter (REMOVE?)","text":"

Collects tweets.

Collects tweets from target_timelines. Up to tweet_count tweets from each user and up to timelimit back in time. The tweet text is sent separately and if allowed, links to pastebin are followed and the text sent in a separate report

Module: intelmq.bots.collectors.twitter.collector_twitter

Parameters (also expects feed parameters):

target_timelines

() screen_names of twitter accounts to be followed

tweet_count

() number of tweets to be taken from each account

timelimit

() maximum age of the tweets collected in seconds

follow_urls

() list of screen_names for which URLs will be followed

exclude_replies

() exclude replies of the followed screen_names

include_rts

() whether to include retweets by given screen_name

consumer_key

() Twitter API login data

consumer_secret

() Twitter API login data

access_token_key

() Twitter API login data

access_token_secret

() Twitter API login data

"},{"location":"user/bots/#parser-bots","title":"Parser Bots","text":""},{"location":"user/bots/#common-parameters","title":"Common parameters","text":""},{"location":"user/bots/#default_fields","title":"default_fields","text":"

(optional, object) Map of statically added fields to each event (only applied if parsing the event doesn't set the value).

example usage:

defaults_fields:\n  classification.type: c2-server\n  protocol.transport: tcp\n
"},{"location":"user/bots/#abusech-feodo-tracker","title":"Abuse.ch Feodo Tracker","text":"

Parses data from Abuse.ch Feodo Tracker (JSON format).

Module: intelmq.bots.parsers.abusech.parser_feodotracker

No additional parameters.

"},{"location":"user/bots/#alienvault-api","title":"AlienVault API","text":"

Parses data from AlienVault API.

Module: intelmq.bots.parsers.alienvault.parser

No additional parameters.

"},{"location":"user/bots/#alienvault-otx","title":"AlienVault OTX","text":"

Parses data from AlientVault Open Threat Exchange (OTX).

Module: intelmq.bots.parsers.alienvault.parser_otx

No additional parameters.

"},{"location":"user/bots/#anubisnetworks-cyberfeed-stream","title":"AnubisNetworks Cyberfeed Stream","text":"

Parses data from AnubisNetworks Cyberfeed Stream.

The feed format changes over time. The parser supports at least data from 2016 and 2020.

Events with the Malware \"TestSinkholingLoss\" are ignored, as they are for the feed provider's internal purpose only and should not be processed at all.

Module: intelmq.bots.parsers.anubisnetworks.parser

Parameters:

use_malware_family_as_classification_identifier

(optional, boolean) Use the malw.family field as classification.type. If false, check if the same as malw.variant. If it is the same, it is ignored. Otherwise saved as extra.malware.family. Defaults to true.

"},{"location":"user/bots/#bambenek","title":"Bambenek","text":"

Parses data from Bambenek DGA, Domain, and IP feeds.

Module: intelmq.bots.parsers.bambenek.parser

No additional parameters.

"},{"location":"user/bots/#blocklistde","title":"Blocklist.de","text":"

Parses data from Blocklist.de feeds.

Module: intelmq.bots.parsers.blocklistde.parser

No additional parameters.

"},{"location":"user/bots/#blueliv-crimeserver_1","title":"Blueliv Crimeserver","text":"

Parses data from Blueliv Crimeserver feed.

Module: intelmq.bots.parsers.blueliv.parser_crimeserver

No additional parameters.

"},{"location":"user/bots/#calidog-certstream_1","title":"Calidog Certstream","text":"

Parses data from Certificate Transparency Log.

For each domain in the leaf_cert.all_domains object one event with the domain in source.fqdn (and source.ip as fallback) is produced. The seen-date is saved in time.source and the classification type is other.

Module: intelmq.bots.parsers.calidog.parser_certstream

No additional parameters.

"},{"location":"user/bots/#cert-eu","title":"CERT-EU","text":"

Parses data from CERT-EU feed (CSV).

Module: intelmq.bots.parsers.certeu.parser_csv

No additional parameters.

"},{"location":"user/bots/#ci-army","title":"CI Army","text":"

Parses data from CI Army feed.

Module: intelmq.bots.parsers.ci_army.parser

No additional parameters.

"},{"location":"user/bots/#cleanmx","title":"CleanMX","text":"

Parses data from CleanMX feed.

Module: intelmq.bots.parsers.cleanmx.parser

No additional parameters.

"},{"location":"user/bots/#team-cymru-cap","title":"Team Cymru CAP","text":"

Parses data from Team Cymru's CSIRT Assistance Program (CAP) feed.

There are two different feeds available:

The new will replace the old at some point in time, currently you need to fetch both. The parser handles both formats.

Old feed

As little information on the format is available, the mappings might not be correct in all cases. Some reports are not implemented at all as there is no data available to check if the parsing is correct at all. If you do get errors like Report ... not implement or similar please open an issue and report the (anonymized) example data. Thanks.

The information about the event could be better in many cases but as Cymru does not want to be associated with the report, we can't add comments to the events in the parser, because then the source would be easily identifiable for the recipient.

Module: intelmq.bots.parsers.cymru.parser_cap_program

No additional parameters.

"},{"location":"user/bots/#team-cymru-full-bogons","title":"Team Cymru Full Bogons","text":"

Parses data from full bogons feed.

http://www.team-cymru.com/bogon-reference.html

Module: intelmq.bots.parsers.cymru.parser_full_bogons

No additional parameters.

"},{"location":"user/bots/#cznic-haas","title":"CZ.NIC HaaS","text":"

Parses data from CZ.NIC Honeypot as a service (HaaS) feed.

Module: intelmq.bots.parsers.cznic.parser_haas

No additional parameters.

"},{"location":"user/bots/#cznic-proki","title":"CZ.NIC PROKI","text":"

Parses data from CZ.NIC PROKI API.

Module: intelmq.bots.parsers.cznic.parser_proki

No additional parameters.

"},{"location":"user/bots/#danger-rulez","title":"Danger Rulez","text":"

Parses data from Danger Rulez SSH blocklist.

Module: intelmq.bots.parsers.danger_rulez.parser

No additional parameters.

"},{"location":"user/bots/#dataplane","title":"Dataplane","text":"

Parses data from Dataplane feed.

Module: intelmq.bots.parsers.dataplane.parser

No additional parameters.

"},{"location":"user/bots/#dshield-asn","title":"DShield ASN","text":"

Parses data from DShield ASN feed.

Module: intelmq.bots.parsers.dshield.parser_asn

No additional parameters.

"},{"location":"user/bots/#dshield-block","title":"DShield Block","text":"

Parses data from DShield Block feed.

Module: intelmq.bots.parsers.dshield_parser_block

No additional parameters.

"},{"location":"user/bots/#eset","title":"ESET","text":"

Parses data from ESET ETI TAXII server.

Supported collections:

Module: intelmq.bots.parsers.eset.parser

No additional parameters.

"},{"location":"user/bots/#dyn-todo","title":"Dyn (TODO)","text":""},{"location":"user/bots/#fireeye_1","title":"FireEye","text":"

Parses data from FireEye MAS appliance.

Module: intelmq.bots.parsers.fireeye.parser

No additional parameters.

"},{"location":"user/bots/#fraunhofer-dga","title":"Fraunhofer DGA","text":"

Parses data from Fraunhofer DGA feed.

Module: intelmq.bots.parsers.fraunhofer.parser_dga

No additional parameters.

"},{"location":"user/bots/#generic-csv","title":"Generic CSV","text":"

Parses CSV data.

Lines starting with # are skipped. Headers won't be interpreted.

Module: intelmq.bots.parsers.generic.parser_csv

Parameters

columns

(required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.

columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n

is equivalent to:

columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n

The fourth column is not used in this example.

It is possible to specify multiple columns using the | character. E.g.

columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n

First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:

columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
columns:\n  - \"source.url|__IGNORE__\"\n

column_regex_search

(optional, object) A dictionary mapping field names (as given per the columns parameter) to regular expression. The field is evaluated using re.search. Eg. to get the ASN out of AS1234 use: {\"source.asn\": \"[0-9]*\"}. Make sure to properly escape any backslashes in your regular expression (see also this issue).

compose_fields

(optional, object) Compose fields from multiple columns, e.g. with data like this:

# Host,Path\nexample.com,/foo/\nexample.net,/bar/\n

Using this parameter:

compose_fields:\n  source.url: \"http://{0}{1}\"\n

You get:

http://example.com/foo/\nhttp://example.net/bar/\n

in the respective source.url fields. The value in the dictionary mapping is formatted whereas the columns are available with their index.

default_url_protocol

(optional, string) For URLs you can give a default protocol which will be prepended to the data. Defaults to null.

delimiter

(optional, string) Character used for columns separation. Defaults to , (comma).

skip_header

(optional, boolean/integer) Whether to skip the first N lines of the input (True -> 1, False -> 0). Lines starting with # will be skipped additionally, make sure you do not skip more lines than needed!

time_format

(optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.

type

(optional, string) Set the classification.type statically. Deprecated in favour of default_fields . Will be removed in IntelMQ 4.0.0.

data_type

(optional, object) Sets the data of specific type, currently only json is a supported value.

Example:

columns:\n  - source.ip\n  - source.url\n  - extra.tags\ndata_type:\n  extra.tags: json\n

It will ensure that extra.tags is treated as JSON.

filter_text

(optional, string) Only process the lines containing or not containing specified text. It is expected to be used in conjunction with filter_type.

filter_type

(optional, string) Allowed values: whitelist or blacklist. When whitelist is used, only lines containing the text specified in filter_text option will be processed. When blacklist is used, only lines NOT containing the text will be processed.

Example (processing ipset format files):

filter_text: 'ipset add '\nfilter_type: whitelist\ncolumns:\n  - __IGNORE__\n  - __IGNORE__\n  - __IGNORE__\n  - source.ip\n

type_translation

(optional, object) If the source does have a field with information for classification.type, but it does not correspond to IntelMQ's types, you can map them to the correct ones. The type_translation field can hold a dictionary, or a string with a JSON dictionary which maps the feed's values to IntelMQ's.

Example:

type_translation:\n  malware_download: \"malware-distribution\"\n

columns_required

(optional, array of booleans) An array of true/false for each column. By default, it is true for every column.

"},{"location":"user/bots/#github-feed","title":"Github Feed","text":"

Parses data publicly available on GitHub (should receive from github_api collector).

Module: intelmq.bots.parsers.github_feed.parser

No additional parameters.

"},{"location":"user/bots/#have-i-been-pwned-callback","title":"Have I Been Pwned Callback","text":"

Parsers data from the callback of Have I Been Pwned Enterprise Subscription.

Parses breaches and pastes and creates one event per e-mail address. The e-mail address is stored in source.account . classification.type is leak and classification.identifier is breach or paste.

Module: intelmq.bots.parsers.hibp.parser_callback

No additional parameters.

"},{"location":"user/bots/#html-table","title":"HTML Table","text":"

Parses tables in HTML documents.

Module: intelmq.bots.parsers.html_table.parser

Parameters:

(required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.

columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n

is equivalent to:

columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n

The fourth column is not used in this example.

It is possible to specify multiple columns using the | character. E.g.

columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n

First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:

columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
columns:\n  - \"source.url|__IGNORE__\"\n

ignore_values

(optional, string/array of strings) A list of strings or a string of comma-separated values which are ignored when encountered.

Example:

ignore_values:\n  - \"\"\n  - \"unknown\"\n  - \"Not listed\"\n

The following configuration will lead to assigning all values to malware.name and extra.SBL except unknown and Not listed respectively.

columns:\n  - source.url\n  - malware.name\n  - extra.SBL\nignore_values:\n  - ''\n  - unknown\n  - Not listed\n

Parameters columns and ignore_values must have same length!

attribute_name

(optional, string) Filtering table with table attributes. To be used in conjunction with attribute_value. E.g. class, id, style.

attribute_value

(optional, string) To filter all tables with attribute class='details' use

attribute_name: \"class\"\nattribute_value: \"details\"\n

table_index

(optional, integer) Index of the table if multiple tables present. If attribute_name and attribute_value given, index according to tables remaining after filtering with table attribute. Defaults to 0.

split_column

(optional, ) Padded column to be split to get values, to be used in conjunction with split_separator and split_index, optional.

split_separator

(optional, string) Delimiter string for padded column.

split_index

(optional, integer) Index of unpadded string in returned list from splitting split_column with split_separator as delimiter string. Defaults to 0.

Example:

split_column: \"source.fqdn\"\nsplit_separator: \" \"\nsplit_index: 1\n

With above configuration, column corresponding to source.fqdn with value D lingvaworld.ru will be assigned as source.fqdn: lingvaworld.ru.

skip_table_head

(optional, boolean) Skip the first row of the table. Defaults to true.

default_url_protocol

(optional, string) For URLs you can give a default protocol which will be pretended to the data. Defaults to http://.

time_format

(optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.

html_parser

(optional, string) The HTML parser to use. Allowed values: html.parser or lxml (see also https://www.crummy.com/software/BeautifulSoup/bs4/doc/). Defaults to html.parser.

"},{"location":"user/bots/#json-todo","title":"JSON (TODO)","text":"

TODO

Module: intelmq.bots.parsers.json.parser

"},{"location":"user/bots/#keyvalue-parser","title":"Key=Value Parser","text":"

Parses text lines in key=value format, for example FortiGate firewall logs.

Parsing limitations

The input must not have (quoted) occurrences of the separator in the values. For example, this is not parsable (with space as separator):

key=\"long value\" key2=\"other value\"\n

In firewall logs like FortiGate, this does not occur. These logs usually look like:

srcip=192.0.2.1 srcmac=\"00:00:5e:00:17:17\"\n

Module: intelmq.bots.parsers.key_value.parser

Parameters:

pair_separator

(optional, string) String separating key=value pairs. Defaults to space.

kv_separator

(optional, string) String separating the key and the value. Defaults to =.

keys

(optional, object) Mapping of original key names to IntelMQ Data Format.

Example:

keys:\n  srcip: source.ip\n  dstip: destination.ip\n

The value mapped to time.source is parsed. If the value is numeric, it is interpreted. Otherwise, or if it fails, it is parsed fuzzy with dateutil. If the value cannot be parsed, a warning is logged per line.

strip_quotes

(optional, boolean) Whether to remove opening and closing quotes from values. Defaults to true.

"},{"location":"user/bots/#malwarepatrol","title":"MalwarePatrol","text":"

Parses data from MalwarePatrol feed.

Module: intelmq.bots.parsers.malwarepatrol.parser_dansguardian

No additional parameters.

"},{"location":"user/bots/#malwareurl","title":"MalwareURL","text":"

Parses data from MalwareURL feed.

Module: intelmq.bots.parsers.malwareurl.parser

No additional parameters.

"},{"location":"user/bots/#mcafee-advanced-threat-defense-file","title":"McAfee Advanced Threat Defense File","text":"

Parse IoCs from McAfee Advanced Threat Defense reports (hash, IP, URL).

Module: intelmq.bots.parsers.mcafee.parser_atd

Parameters:

verdict_severity

(optional, integer) Minimum report severity to parse. Defaults to 4.

"},{"location":"user/bots/#microsoft-ctip","title":"Microsoft CTIP","text":"

Parses data from the Microsoft CTIP feed.

Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line). The provided data differs between the two formats/providers.

The parser is capable of parsing both feeds:

The feeds contain a field called Payload which is nearly always a base64 encoded JSON structure. If decoding works, the contained fields are saved as extra.payload.*, otherwise the field is saved as extra.payload.text.

Module: intelmq.bots.parsers.microsoft.parser_ctip

Parameters:

overwrite

(optional, boolean) Overwrite an existing field feed.name with DataFeed of the source. Defaults to false.

"},{"location":"user/bots/#misp","title":"MISP","text":"

Parses MISP events.

MISP events collected by the MISPCollectorBot are passed to this parser for processing. Supported MISP event categories and attribute types are defined in the SUPPORTED_MISP_CATEGORIES and MISP_TYPE_MAPPING class constants.

Module: intelmq.bots.parsers.misp.parser

No additional parameters.

"},{"location":"user/bots/#n6","title":"N6","text":"

Parses n6 data into IntelMQ format.

Test messages are ignored, this is logged with debug logging level. Also contains a mapping for the classification ( results in taxonomy, type and identifier). The name field is normally used as malware.name, if that fails due to disallowed characters, these characters are removed and the original value is saved as event_description.text. This can happen for names like further iocs: text with invalid ' char.

If a n6 message contains multiple IP addresses, multiple events are generated, resulting in events only differing in the address information.

Module: intelmq.bots.parsers.n6.parser_n6stomp

No additional parameters.

"},{"location":"user/bots/#openphish-free","title":"OpenPhish Free","text":"

Parses data from OpenPhish Free feed.

Module: intelmq.bots.parsers.openphish.parser

No additional parameters.

"},{"location":"user/bots/#openphish-premium","title":"OpenPhish Premium","text":"

Parses data from OpenPhish Premium feed (JSON).

Module: intelmq.bots.parsers.openphish.parser_commercial

No additional parameters.

"},{"location":"user/bots/#phishtank","title":"Phishtank","text":"

Parses data from Phishtank feed.

Module: intelmq.bots.parsers.phishtank.parser

No additional parameters.

"},{"location":"user/bots/#shadowserver","title":"Shadowserver","text":"

The Shadowserver parser operates on CSV formatted data.

How this bot works?

There are two possibilities for the bot to determine which report type the data belongs to in order to determine the correct mapping of the columns:

  1. Automatic report type detection

    Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.

    When processing a report, this bot takes extra.file_name from the report and looks in config.py how the report should be parsed. If this lookup is not possible, and the feedname is not given as parameter, the feed cannot be parsed.

    The field extra.file_name has the following structure: %Y-%m-%d-${report_name}[-suffix].csv where the optional suffix can be something like country-geo. For example, some possible filenames are 2019-01-01-scan_http-country-geo.csv or 2019-01-01-scan_tftp.csv. The important part is the report_name, between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames like scan_tftp.csv are also detected.

  2. Fixed report type

    If the method above is not possible and for upgraded instances, the report type can be set with the feedname parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of the feedname parameter can be found in the table below in the column \"Report Type\".

Module:

intelmq.bots.parsers.shadowserver.parser

Parameters:

feedname

(optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema feed_name field.

For example using curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name.

overwrite

(optional, boolean) If an existing feed.name should be overwritten.

Supported reports:

The report configuration is stored in a shadowserver-schema.json file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.

The parser will attempt to download a schema update on startup when the auto_update option is enabled.

Schema downloads can also be scheduled as a cron job for the intelmq user:

  02  01 *   *   *     intelmq.bots.parsers.shadowserver.parser --update-schema\n

For air-gapped systems automation will be required to download and copy the file to VAR_STATE_PATH/shadowserver-schema.json.

The parser will automatically reload the configuration when the file changes.

Schema contract

Once set in the schema, the classification.identifier, classification.taxonomy, and classification.type fields will remain static for a specific report.

The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.

Sample configuration

  shadowserver-parser:\n    bot_id: shadowserver-parser\n    name: Shadowserver Parser\n    enabled: true\n    group: Parser\n    groupname: parsers\n    module: intelmq.bots.parsers.shadowserver.parser\n    parameters:\n      destination_queues:\n        _default: [file-output-queue]\n      auto_update: true\n    run_mode: continuous\n
"},{"location":"user/bots/#shodan","title":"Shodan","text":"

Parses data from Shodan (search, stream etc).

The parser is by far not complete as there are a lot of fields in a big nested structure. There is a minimal mode available which only parses the important/most useful fields and also saves everything in extra.shodan keeping the original structure. When not using the minimal mode if may be useful to ignore errors as many parsing errors can happen with the incomplete mapping.

Module: intelmq.bots.parsers.shodan.parser

Parameters:

ignore_errors

(optional, boolean) Defaults to true.

minimal_mode

(optional, boolean) Defaults to false.

"},{"location":"user/bots/#spamhaus-drop","title":"Spamhaus DROP","text":"

Parses data from Spamhaus DROP feed.

Module: intelmq.bots.parsers.spamhaus.parser_drop

No additional parameters.

"},{"location":"user/bots/#spamhaus-cert","title":"Spamhaus CERT","text":"

Parses data from Spamhaus CERT feed.

Module: intelmq.bots.parsers.spamhaus.parser_cert

No additional parameters.

"},{"location":"user/bots/#surbl","title":"Surbl","text":"

Parses data from surbl feed.

Module: intelmq.bots.parsers.surbl.parser

No additional parameters.

"},{"location":"user/bots/#threatminer","title":"Threatminer","text":"

Parses data from Threatminer feed.

Module: intelmq.bots.parsers.threatminer.parser

No additional parameters.

"},{"location":"user/bots/#turris","title":"Turris","text":"

Parses data from Turris Greylist feed.

Module: intelmq.bots.parsers.turris.parser

No additional parameters.

"},{"location":"user/bots/#twitter","title":"Twitter","text":"

Extracts URLs from text, fuzzy, aimed at parsing tweets.

Module: intelmq.bots.parsers.twitter.parser

Parameters:

domain_whitelist

(optional, array of strings) domains to be filtered out

substitutions

(optional, string) Semicolon delimited list of even length of pairs of substitutions (for example: .;.;,;. substitutes . for . and , for .).

classification_type

(optional, string) Statically set classification.type.

default_scheme

(optional, string) Default scheme for URLs if not given. See also the next section.

Default scheme

The dependency url-normalize changed it's behavior in version 1.4.0 from using http:// as default scheme to https://. Version 1.4.1 added the possibility to specify it. Thus you can only use the default_scheme parameter with a current version of this library >= 1.4.1, with 1.4.0 you will always get https:// as default scheme and for older versions < 1.4.0 http:// is used.

This does not affect URLs which already include the scheme.

"},{"location":"user/bots/#vxvault","title":"VxVault","text":"

Parses data from VxVault feed.

Module: intelmq.bots.parsers.vxvault.parser

No additional parameters.

"},{"location":"user/bots/#zoneh","title":"ZoneH","text":"

Parses data from ZoneH.

This bot is designed to consume defacement reports from zone-h.org. It expects fields normally present in CSV files distributed by email.

Module: intelmq.bots.parsers.zoneh.parser

No additional parameters.

"},{"location":"user/bots/#expert-bots","title":"Expert Bots","text":"

Expert bots are used for enriching, filtering and/or other data manipulation.

"},{"location":"user/bots/#abusix","title":"Abusix","text":"

This bot adds source.abuse_contact and destination.abuse_contact e-mail addresses. They are obtained via DNS TXT queries to Abusix servers.

Requirements

This bot can optionally use the python module querycontacts by Abusix itself: https://pypi.org/project/querycontacts/

pip3 install querycontacts\n

If the package is not installed, our own routines are used.

Module: intelmq.bots.experts.abusix.expert

Parameters (also expects cache parameters):

No additional parameters.

"},{"location":"user/bots/#aggregate","title":"Aggregate","text":"

Aggregates events based upon given fields & timespan.

Define specific fields to filter incoming events and aggregate them. Also set the timespan you want the events to get aggregated.

The \"cleanup\" procedure, sends out the aggregated events or drops them based upon the given threshold value. It is called on every incoming message and on the bot's initialization. If you're potentially running on low traffic ( no incoming events within the given timestamp ) it is recommended to reload or restart the bot via cronjob each 30 minutes (adapt to your configured timespan). Otherwise you might loose information.

I. e.:

crontab -e\n\n0,30 * * * * intelmqctl reload my-aggregate-bot\n

For reloading/restarting please check the intelmqctl documentation.

Module: intelmq.bots.experts.aggregate.expert

Parameters (also expects cache parameters):

Warning

redis_cache_ttl is not used at it would result in data loss.

fields

(required, string) Given fields which are used to aggregate like classification.type, classification.identifier.

threshold

(required, integer) If the aggregated event is lower than the given threshold after the timespan, the event will get dropped.

timespan

(required, string) Timespan to aggregate events during the given time. I. e. 1 hour

"},{"location":"user/bots/#asn-lookup","title":"ASN Lookup","text":"

This bot uses an offline database to add source.asn and destination.asn based on the respective IP address.

Requirements

Install pyasn module.

pip3 install pyasn\n

Module: intelmq.bots.experts.asn_lookup.expert

Parameters:

database

(required, string) Path to the downloaded database.

Database

Use this command to create/update the database and reload the bot:

intelmq.bots.experts.asn_lookup.expert --update-database\n

The database is fetched from routeviews.org and licensed under the Creative Commons Attribution 4.0 International license (see the routeviews FAQ).

"},{"location":"user/bots/#csv-converter","title":"CSV Converter","text":"

Converts an event to CSV format, saved in the output field.

To use the CSV-converted data in an output bot - for example in a file output, use the configuration parameter single_key of the output bot and set it to output.

Module: intelmq.bots.experts.csv_converter.expert

Parameters:

delimiter

(optional, string) Defaults to ,.

fieldnames

(required, string) Comma-separated list of field names, e.g. \"time.source,classification.type,source.ip\".

"},{"location":"user/bots/#team-cymru-whois","title":"Team Cymru Whois","text":"

This bot adds geolocation, ASN and BGP prefix based on IP address.

Public documentation: https://www.team-cymru.com/IP-ASN-mapping.html#dns

Module: intelmq.bots.experts.cymru_whois.expert

Parameters (also expects cache parameters):

overwrite

(optional, boolean) Whether to overwrite existing fields. Defaults to true.

"},{"location":"user/bots/#remove-affix","title":"Remove Affix","text":"

Remove part of string from string fields, example: www. from source.fqdn.

Module: intelmq.bots.experts.remove_affix.expert

Parameters:

remove_prefix

(optional, boolean) True - cut from start, False - cut from end. Defaults to true.

affix

(required, string) example 'www.'

field

(required, string) Which field to modify. 'source.fqdn'

"},{"location":"user/bots/#domain-suffix","title":"Domain Suffix","text":"

This bots uses an offline database to add the public suffix to the event, derived by a domain. See or information on the public suffix list: https://publicsuffix.org/list/. Only rules for ICANN domains are processed. The list can (and should) contain Unicode data, punycode conversion is done during reading.

Note that the public suffix is not the same as the top level domain (TLD). E.g. co.uk is a public suffix, but the TLD is uk. Privately registered suffixes (such as blogspot.co.at) which are part of the public suffix list too, are ignored.

Rule processing

A short summary how the rules are processed:

The simple ones:

com\nat\ngv.at\n

example.com leads to com, example.gv.at leads to gv.at.

Wildcards:

*.example.com\n

www.example.com leads to www.example.com.

And additionally the exceptions, together with the above wildcard rule:

!www.example.com\n

www.example.com does now not lead to www.example.com, but to example.com.

Module: intelmq.bots.experts.domain_suffix.expert

Parameters:

field

(required, string) Allowed values: fqdn or reverse_dns.

suffix_file

(required, string) path to the suffix file

Database

Use this command to create/update the database and reload the bot:

intelmq.bots.experts.domain_suffix.expert --update-database\n
"},{"location":"user/bots/#domain-valid","title":"Domain Valid","text":"

Checks if a domain is valid by performing multiple validity checks (see below).

If the field given in domain_field does not exist in the event, the event is dropped. If the domain contains underscores (_), the event is dropped. If the domain is not valid according to the validators library, the event is dropped. If the domain's last part (the TLD) is not in the TLD-list configured by parameter tlds_domains_list, the field is dropped. Latest TLD list: https://data.iana.org/TLD/

Module: intelmq.bots.experts.domain_valid.expert

Parameters:

domain_field

(required, string) The name of the field to be validated.

tlds_domains_list

(required, string) Path to a local file with all valid TLDs. Defaults to /opt/intelmq/var/lib/bots/domain_valid/tlds-alpha-by-domain.txt

"},{"location":"user/bots/#deduplicator","title":"Deduplicator","text":"

Bot responsible for dropping duplicate events. Deduplication can be performed based on an arbitrary set of fields.

Module: intelmq.bots.experts.deduplicator.expert

Parameters (also expects cache parameters):

bypass

(optional, boolean) Whether to bypass the deduplicator or not. When set to true, messages will not be deduplicated. Defaults to false.

filter_type

(optional, string) Allowed values: blacklist or whitelist. The filter type will be used to define how Deduplicator bot will interpret the parameter filter_keys in order to decide whether an event has already been seen or not, i.e., duplicated event or a completely new event.

filter_keys

(optional, string) string with multiple keys separated by comma. Please note that time.observation key will not be considered even if defined, because the system always ignore that key.

When using a whitelist field pattern and a small number of fields (keys), it becomes more important, that these fields exist in the events themselves. If a field does not exist, but is part of the hashing/deduplication, this field will be ignored. If such events should not get deduplicated, you need to filter them out before the deduplication process, e.g. using a sieve expert. See also this discussion thread on the mailing-list.

Configuration Example

Example 1

The bot with this configuration will detect duplication only based on source.ip and destination.ip keys.

parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"whitelist\"\n  filter_keys: \"source.ip,destination.ip\"\n

Example 2

The bot with this configuration will detect duplication based on all keys, except source.ip and destination.ip keys.

parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"blacklist\"\n  filter_keys: \"source.ip,destination.ip\"\n

Flushing the cache

To flush the deduplicator's cache, you can use the redis-cli tool. Enter the database used by the bot and submit the flushdb command:

redis-cli -n 6\nflushdb\n
"},{"location":"user/bots/#do-portal","title":"DO Portal","text":"

The DO portal retrieves the contact information from a DO portal instance: http://github.com/certat/do-portal/

Module: intelmq.bots.experts.do_portal.expert

Parameters:

mode

(required, string) Allowed values: replace or append. How to handle new abuse contacts in case there are existing ones.

portal_url

(required, string) The URL to the portal, without the API-path. The used URL is $portal_url + '/api/1.0/ripe/contact?cidr=%s'.

portal_api_key

(required, string) The API key of the user to be used. Must have sufficient privileges.

"},{"location":"user/bots/#field-reducer","title":"Field Reducer","text":"

The field reducer bot is capable of removing fields from events.

Module: intelmq.bots.experts.field_reducer.expert

Parameters:

type

(required, string) Allowed values: whitelist or blacklist. When whitelist is set, tnly the fields in keys will passed along. When blacklist is set then the fields in keys will be removed from events.

keys

(required, array of strings) Can be an array of field names or a string with a comma-separated list of field names.

"},{"location":"user/bots/#filter","title":"Filter","text":"

The filter bot is capable of filtering specific events.

A simple filter for messages (drop or pass) based on a exact string comparison or regular expression.

Module: intelmq.bots.experts.filter.expert

Parameters:

Parameters for filtering with key/value attributes

filter_key

() - key from data format

filter_value

() - value for the key

filter_action

() - action when a message match to the criteria (possible actions: keep/drop)

filter_regex

() - attribute determines if the filter_value shall be treated as regular expression or not.

If this attribute is not empty (can be true, yes or whatever), the bot uses python's `re.search <https://docs.python.org/3/library/re.html#re.search>`_ function to evaluate the filter with regular expressions. If this attribute is empty or evaluates to false, an exact string comparison is performed. A check on string * inequality can be achieved with the usage of Paths* described below.

Parameters for time based filtering

not_before

(optional, string) Events before this time will be dropped. Example: 1 week.

not_after

(optional, string) - Events after this time will be dropped.

Both parameters accept string values describing absolute or relative time:

2015-09-12T06:22:11+00:00\n

time.source

(optional, string) Taken from the event will be compared to this value to decide the filter behavior.