Deployed d5e3e41 to develop with MkDocs 1.6.1 and mike 2.1.3

develop/user/bots/index.html

@@ -108,7 +108,7 @@
 </code></pre></div> <p><strong>Module:</strong> <code>intelmq.bots.parsers.key_value.parser</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>pair_separator</code></strong></p> <p>(optional, string) String separating key=value pairs. Defaults to space.</p> <p><strong><code>kv_separator</code></strong></p> <p>(optional, string) String separating the key and the value. Defaults to <code>=</code>.</p> <p><strong><code>keys</code></strong></p> <p>(optional, object) Mapping of original key names to IntelMQ Data Format.</p> <p>Example:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-30-1 name=__codelineno-30-1 href=#__codelineno-30-1></a><span class=nt>keys</span><span class=p>:</span>
 <a id=__codelineno-30-2 name=__codelineno-30-2 href=#__codelineno-30-2></a><span class=w>  </span><span class=nt>srcip</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">source.ip</span>
 <a id=__codelineno-30-3 name=__codelineno-30-3 href=#__codelineno-30-3></a><span class=w>  </span><span class=nt>dstip</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">destination.ip</span>
-</code></pre></div> <p>The value mapped to <code>time.source</code> is parsed. If the value is numeric, it is interpreted. Otherwise, or if it fails, it is parsed fuzzy with dateutil. If the value cannot be parsed, a warning is logged per line.</p> <p><strong><code>strip_quotes</code></strong></p> <p>(optional, boolean) Whether to remove opening and closing quotes from values. Defaults to true.</p> <hr> <h3 id=malwarepatrol>MalwarePatrol <div id=intelmq.bots.parsers.malwarepatrol.parser_dansguardian /></h3> <p>Parses data from MalwarePatrol feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.malwarepatrol.parser_dansguardian</code></p> <p>No additional parameters.</p> <hr> <h3 id=malwareurl>MalwareURL <div id=intelmq.bots.parsers.malwareurl.parser /></h3> <p>Parses data from MalwareURL feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.malwareurl.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=mcafee-advanced-threat-defense-file>McAfee Advanced Threat Defense File <div id=intelmq.bots.parsers.mcafee.parser_atd /></h3> <p>Parse IoCs from McAfee Advanced Threat Defense reports (hash, IP, URL).</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.mcafee.parser_atd</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>verdict_severity</code></strong></p> <p>(optional, integer) Minimum report severity to parse. Defaults to 4.</p> <hr> <h3 id=microsoft-ctip>Microsoft CTIP <div id=intelmq.bots.parsers.microsoft.parser_ctip /></h3> <p>Parses data from the Microsoft CTIP feed.</p> <p>Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line). The provided data differs between the two formats/providers.</p> <p>The parser is capable of parsing both feeds:</p> <ul> <li><code>ctip-c2</code></li> <li><code>ctip-infected-summary</code> The feeds only differ by a few fields, not in the format.</li> </ul> <p>The feeds contain a field called <code>Payload</code> which is nearly always a base64 encoded JSON structure. If decoding works, the contained fields are saved as <code>extra.payload.*</code>, otherwise the field is saved as <code>extra.payload.text</code>.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.microsoft.parser_ctip</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>overwrite</code></strong></p> <p>(optional, boolean) Overwrite an existing field <code>feed.name</code> with <code>DataFeed</code> of the source. Defaults to false.</p> <hr> <h3 id=misp>MISP <div id=intelmq.bots.parsers.misp.parser /></h3> <p>Parses MISP events.</p> <p>MISP events collected by the MISPCollectorBot are passed to this parser for processing. Supported MISP event categories and attribute types are defined in the <code>SUPPORTED_MISP_CATEGORIES</code> and <code>MISP_TYPE_MAPPING</code> class constants.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.misp.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=n6>N6 <div id=intelmq.bots.parsers.n6.parser_n6stomp /></h3> <p>Parses n6 data into IntelMQ format.</p> <p>Test messages are ignored, this is logged with debug logging level. Also contains a mapping for the classification ( results in taxonomy, type and identifier). The <code>name</code> field is normally used as <code>malware.name</code>, if that fails due to disallowed characters, these characters are removed and the original value is saved as <code>event_description.text</code>. This can happen for names like <code>further iocs: text with invalid ' char</code>.</p> <p>If a n6 message contains multiple IP addresses, multiple events are generated, resulting in events only differing in the address information.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.n6.parser_n6stomp</code></p> <p>No additional parameters.</p> <hr> <h3 id=openphish-free>OpenPhish Free <div id=intelmq.bots.parsers.openphish.parser /></h3> <p>Parses data from OpenPhish Free feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.openphish.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=openphish-premium>OpenPhish Premium <div id=intelmq.bots.parsers.openphish.parser_commercial /></h3> <p>Parses data from OpenPhish Premium feed (JSON).</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.openphish.parser_commercial</code></p> <p>No additional parameters.</p> <hr> <h3 id=phishtank>Phishtank <div id=intelmq.bots.parsers.phishtank.parser /></h3> <p>Parses data from Phishtank feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.phishtank.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=shadowserver>Shadowserver <div id=intelmq.bots.parsers.shadowserver.parser /></h3> <p>The Shadowserver parser operates on CSV formatted data.</p> <p><strong>How this bot works?</strong></p> <p>There are two possibilities for the bot to determine which report type the data belongs to in order to determine the correct mapping of the columns:</p> <ol> <li> <p><strong>Automatic report type detection</strong></p> <p>Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.</p> <p>When processing a report, this bot takes <code>extra.file_name</code> from the report and looks in <code>config.py</code> how the report should be parsed. If this lookup is not possible, and the <code>feedname</code> is not given as parameter, the feed cannot be parsed.</p> <p>The field <code>extra.file_name</code> has the following structure: <code>%Y-%m-%d-${report_name}[-suffix].csv</code> where the optional suffix can be something like <code>country-geo</code>. For example, some possible filenames are <code>2019-01-01-scan_http-country-geo.csv</code> or <code>2019-01-01-scan_tftp.csv</code>. The important part is the <code>report_name</code>, between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames like <code>scan_tftp.csv</code> are also detected.</p> </li> <li> <p><strong>Fixed report type</strong></p> <p>If the method above is not possible and for upgraded instances, the report type can be set with the <code>feedname</code> parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of the <code>feedname</code> parameter can be found in the table below in the column "Report Type".</p> </li> </ol> <p><strong>Module:</strong></p> <p><code>intelmq.bots.parsers.shadowserver.parser</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>feedname</code></strong></p> <p>(optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema <code>feed_name</code> field.</p> <p>For example using <code>curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name</code>.</p> <p><strong><code>overwrite</code></strong></p> <p>(optional, boolean) If an existing <code>feed.name</code> should be overwritten.</p> <p>** <code>auto_update</code>**</p> <p>(optional, boolean) Enable automatic schema download.</p> <p><strong>Supported reports:</strong></p> <p>The report configuration is stored in a <code>shadowserver-schema.json</code> file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.</p> <p>The parser will attempt to download a schema update on startup when the <em>auto_update</em> option is enabled.</p> <p>Schema downloads can also be scheduled as a cron job for the <code>intelmq</code> user:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-31-1 name=__codelineno-31-1 href=#__codelineno-31-1></a><span class=w>  </span><span class=m>02</span><span class=w>  </span><span class=m>01</span><span class=w> </span>*<span class=w>   </span>*<span class=w>   </span>*<span class=w>     </span>intelmq.bots.parsers.shadowserver.parser<span class=w> </span>--update-schema
+</code></pre></div> <p>The value mapped to <code>time.source</code> is parsed. If the value is numeric, it is interpreted. Otherwise, or if it fails, it is parsed fuzzy with dateutil. If the value cannot be parsed, a warning is logged per line.</p> <p><strong><code>strip_quotes</code></strong></p> <p>(optional, boolean) Whether to remove opening and closing quotes from values. Defaults to true.</p> <hr> <h3 id=malwarepatrol>MalwarePatrol <div id=intelmq.bots.parsers.malwarepatrol.parser_dansguardian /></h3> <p>Parses data from MalwarePatrol feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.malwarepatrol.parser_dansguardian</code></p> <p>No additional parameters.</p> <hr> <h3 id=malwareurl>MalwareURL <div id=intelmq.bots.parsers.malwareurl.parser /></h3> <p>Parses data from MalwareURL feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.malwareurl.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=mcafee-advanced-threat-defense-file>McAfee Advanced Threat Defense File <div id=intelmq.bots.parsers.mcafee.parser_atd /></h3> <p>Parse IoCs from McAfee Advanced Threat Defense reports (hash, IP, URL).</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.mcafee.parser_atd</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>verdict_severity</code></strong></p> <p>(optional, integer) Minimum report severity to parse. Defaults to 4.</p> <hr> <h3 id=microsoft-ctip>Microsoft CTIP <div id=intelmq.bots.parsers.microsoft.parser_ctip /></h3> <p>Parses data from the Microsoft CTIP feed.</p> <p>Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line). The provided data differs between the two formats/providers.</p> <p>The parser is capable of parsing both feeds:</p> <ul> <li><code>ctip-c2</code></li> <li><code>ctip-infected-summary</code> The feeds only differ by a few fields, not in the format.</li> </ul> <p>The feeds contain a field called <code>Payload</code> which is nearly always a base64 encoded JSON structure. If decoding works, the contained fields are saved as <code>extra.payload.*</code>, otherwise the field is saved as <code>extra.payload.text</code>.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.microsoft.parser_ctip</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>overwrite</code></strong></p> <p>(optional, boolean) Overwrite an existing field <code>feed.name</code> with <code>DataFeed</code> of the source. Defaults to false.</p> <hr> <h3 id=misp>MISP <div id=intelmq.bots.parsers.misp.parser /></h3> <p>Parses MISP events.</p> <p>MISP events collected by the MISPCollectorBot are passed to this parser for processing. Supported MISP event categories and attribute types are defined in the <code>SUPPORTED_MISP_CATEGORIES</code> and <code>MISP_TYPE_MAPPING</code> class constants.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.misp.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=n6>N6 <div id=intelmq.bots.parsers.n6.parser_n6stomp /></h3> <p>Parses n6 data into IntelMQ format.</p> <p>Test messages are ignored, this is logged with debug logging level. Also contains a mapping for the classification ( results in taxonomy, type and identifier). The <code>name</code> field is normally used as <code>malware.name</code>, if that fails due to disallowed characters, these characters are removed and the original value is saved as <code>event_description.text</code>. This can happen for names like <code>further iocs: text with invalid ' char</code>.</p> <p>If a n6 message contains multiple IP addresses, multiple events are generated, resulting in events only differing in the address information.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.n6.parser_n6stomp</code></p> <p>No additional parameters.</p> <hr> <h3 id=openphish-free>OpenPhish Free <div id=intelmq.bots.parsers.openphish.parser /></h3> <p>Parses data from OpenPhish Free feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.openphish.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=openphish-premium>OpenPhish Premium <div id=intelmq.bots.parsers.openphish.parser_commercial /></h3> <p>Parses data from OpenPhish Premium feed (JSON).</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.openphish.parser_commercial</code></p> <p>No additional parameters.</p> <hr> <h3 id=phishtank>Phishtank <div id=intelmq.bots.parsers.phishtank.parser /></h3> <p>Parses data from Phishtank feed.</p> <p><strong>Module:</strong> <code>intelmq.bots.parsers.phishtank.parser</code></p> <p>No additional parameters.</p> <hr> <h3 id=shadowserver>Shadowserver <div id=intelmq.bots.parsers.shadowserver.parser /></h3> <p>The Shadowserver parser operates on CSV formatted data.</p> <p><strong>How this bot works?</strong></p> <p>There are two possibilities for the bot to determine which report type the data belongs to in order to determine the correct mapping of the columns:</p> <ol> <li> <p><strong>Automatic report type detection</strong></p> <p>Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.</p> <p>When processing a report, this bot takes <code>extra.file_name</code> from the report and looks in <code>config.py</code> how the report should be parsed. If this lookup is not possible, and the <code>feedname</code> is not given as parameter, the feed cannot be parsed.</p> <p>The field <code>extra.file_name</code> has the following structure: <code>%Y-%m-%d-${report_name}[-suffix].csv</code> where the optional suffix can be something like <code>country-geo</code>. For example, some possible filenames are <code>2019-01-01-scan_http-country-geo.csv</code> or <code>2019-01-01-scan_tftp.csv</code>. The important part is the <code>report_name</code>, between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames like <code>scan_tftp.csv</code> are also detected.</p> </li> <li> <p><strong>Fixed report type</strong></p> <p>If the method above is not possible and for upgraded instances, the report type can be set with the <code>feedname</code> parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of the <code>feedname</code> parameter can be found in the table below in the column "Report Type".</p> </li> </ol> <p><strong>Module:</strong></p> <p><code>intelmq.bots.parsers.shadowserver.parser</code></p> <p><strong>Parameters:</strong></p> <p><strong><code>feedname</code></strong></p> <p>(optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema <code>feed_name</code> field.</p> <p>For example using <code>curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name</code>.</p> <p><strong><code>overwrite</code></strong></p> <p>(optional, boolean) If an existing <code>feed.name</code> should be overwritten.</p> <p><strong><code>auto_update</code></strong></p> <p>(optional, boolean) Enable automatic schema download.</p> <p><strong>Supported reports:</strong></p> <p>The report configuration is stored in a <code>shadowserver-schema.json</code> file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.</p> <p>The parser will attempt to download a schema update on startup when the <em>auto_update</em> option is enabled.</p> <p>Schema downloads can also be scheduled as a cron job for the <code>intelmq</code> user:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-31-1 name=__codelineno-31-1 href=#__codelineno-31-1></a><span class=w>  </span><span class=m>02</span><span class=w>  </span><span class=m>01</span><span class=w> </span>*<span class=w>   </span>*<span class=w>   </span>*<span class=w>     </span>intelmq.bots.parsers.shadowserver.parser<span class=w> </span>--update-schema
 </code></pre></div> <p>For air-gapped systems automation will be required to download and copy the file to VAR_STATE_PATH/shadowserver-schema.json.</p> <p>The parser will automatically reload the configuration when the file changes.</p> <p><strong>Schema contract</strong></p> <p>Once set in the schema, the <code>classification.identifier</code>, <code>classification.taxonomy</code>, and <code>classification.type</code> fields will remain static for a specific report.</p> <p>The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.</p> <p><strong>Sample configuration</strong></p> <div class=highlight><pre><span></span><code><a id=__codelineno-32-1 name=__codelineno-32-1 href=#__codelineno-32-1></a><span class=w>  </span><span class=nt>shadowserver-parser</span><span class=p>:</span>
 <a id=__codelineno-32-2 name=__codelineno-32-2 href=#__codelineno-32-2></a><span class=w>    </span><span class=nt>bot_id</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">shadowserver-parser</span>
 <a id=__codelineno-32-3 name=__codelineno-32-3 href=#__codelineno-32-3></a><span class=w>    </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">Shadowserver Parser</span>