Python Network Recon Framework: ivre

Posted on Posted in Tools

IVRE (Instrument de veille sur les réseaux extérieurs) or DRUNK (Dynamic Recon of UNKnown networks) is a network recon framework, including two modules for passive recon (one p0f-based and one Bro-based) and one module for active recon (mostly Nmap-based, with a bit of ZMap).

External programs / dependencies

IVRE relies on:

  • Python 2, version 2.6 minimum
  • Nmap & ZMap
  • Bro & p0f
  • MongoDB, version 2.6 minimum
  • a web server (successfully tested with Apache and Nginx, should work with anything capable of serving static files and run a Python-based CGI), although a test web server is now distributed with IVRE (httpd-ivre)
  • a web browser (successfully tested with recent versions of Firefox and Chromium)
  • Maxmind GeoIP free databases
  • optionally Tesseract, if you plan to add screenshots to your Nmap scan results
  • optionally Docker & Vagrant (version 1.6 minimum)

IVRE comes with:

 

Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework

Installation

Dependencies

If you plan to run scans from a machine, install Nmap and optionally ZMap.

If you plan to analyze PCAP file on a machine, install Bro and p0f.

To install IVRE, you’ll need Python 2, version 2.6 minimum (prefer 2.7), with the following modules:

  • Crypto
  • pymongo version 2.7.2 minimum.

The installation of IVRE itself can be done by:

  • using the setup.py (classical ./setup.py build; sudo ./setup.py install) script.
  • using pip: with on a Debian-based system for example, install the packages python-pip and python-dev (needed to build dependencies) and run pip install ivre (this will download and install for you IVRE and its Python dependencies from PyPI, the Python Package Index).
  • building an RPM package (you can use the provided buildrpm script, or use the setup.py script with your own options) and then installing it.
  • using Docker (in this case you do not need to follow the following instructions, as the Docker containers are already configured).
 Configuration

Default configuration values are hard-coded in ivre/config.py. You should not change this file, unless you are modifying IVRE and you want to change the default configuration. You do not need to do this if you want to install IVRE with a non-default configuration, you just need to distribute a proper configuration file.

You can override default values in three files:

  • two system-wide:
    • /etc/ivre.conf
    • /usr/local/etc/ivre.conf (read after, so higher priority)
  • one user-specific:
    • ~/.ivre.conf (the last to be read, so highest priority)

The file should contain lines of type key = value. Empty lines and comments (starting with the # character) are ignored. The following values can be changed:

  • DB: the URL to use; default is mongodb:///, meaning use default database (ivre) on the default host (localhost). Here is a more complete example: mongodb://user:password@host/db?colname_aaa=bbb&colname_ccc=ddd
  • DB_NMAP, DB_PASSIVE and DB_DATA: specific URLs to use; default is to use the URL from DB setting.
  • GEOIP_PATH: default is [INSTALL PREFIX]/share/ivre/geoip/.

For the full and up-to-date list of settings that can be changed, see the ivre/config.py file.

It might be a good idea to have a read-only account everywhere except for some specific users or hosts that need write access to the database (the users that insert scan results with nmap2db, the users or the hosts that run p0f2db and/orpassiverecon2db). It is best to avoid using a configuration with write access to the database when you only need a read access. This can be achieved with users or hosts dedicated to insertion tasks.

DB creation

Once IVRE has been properly configured, it’s time to initialize its databases.

For that, the command-line tools (namely ipdata, ipinfo, scancli and runscans-agentdb, respectively for information about IP addresses, passive information, active information and running scans through agents) have a --init option.

So you can run, with a user or from a host where the configuration has a write access to the database (add < /dev/null to skip the confirmation):

$ scancli --init
This will remove any scan result in your database. Process ? [y/N] y
$ ipinfo --init
This will remove any passive information in your database. Process ? [y/N] y
$ ipdata --init
This will remove any country/AS information in your database. Process ? [y/N] y
# runscans-agentdb --init
This will remove any agent and/or scan in your database and files. Process ? [y/N] y

Getting IP data

# ipdata --download
$ ipdata --import-all --dont-feed-ipdata-cols

Web Server

Once IVRE has been installed, to also install the web interface, you have to copy or symlink IVRE files to your web server directories, or configure your web server to use IVRE files directly.

The files the web server should serve statically are located in [PREFIX]/share/ivre/web/static, the folder the web server should serve as CGI is located in [PREFIX]/share/ivre/web/cgi-bin, and the (optional) folders to use as Dokuwiki content are located in [PREFIX]/share/ivre/dokuwiki/doc and [PREFIX]/share/ivre/dokuwiki/media. Make sure your Dokuwiki has been configured with server-side URL rewriting; this means using proper rewrite in your Web server configuration (with mod_rewritewhen using Apache; you can use the provided Dockerfiles as examples on how to configure Apache or Nginx) and adding$conf['userewrite'] = 1 in your Dokuwiki config file.

You may want to change some values, by creating the file [PREFIX]/share/ivre/web/static/config.js based on the -samplefile and by creating or modifying /etc/ivre.conf.

On a typical Debian/Ubuntu installation with Apache and Dokuwiki installed with the distribution packages, these files should be copied or (sym)linked at these locations:

  • [PREFIX]/share/ivre/web/static/* -> /var/www or /var/www/html
  • [PREFIX]/share/ivre/web/cgi-bin/scanjson.py -> /usr/lib/cgi-bin/
  • [PREFIX]/share/ivre/dokuwiki/doc -> /var/lib/dokuwiki/data/pages/
  • [PREFIX]/share/ivre/dokuwiki/media/logo.png -> /var/lib/dokuwiki/data/media/
  • [PREFIX]/share/ivre/dokuwiki/media/doc -> /var/lib/dokuwiki/data/media/

The values WEB_SKIP and WEB_LIMIT from IVRE’s configuration must match the values skip and limit in the dflt object inconfig.js.

Agent

If you do not plan to run active scans with remote agents (where IVRE will not be installed), you can skip this section.

The agent does not require IVRE to be installed. It is a script that needs to be adapted to each situation.

The agent is only needed when you cannot install IVRE on the machine used to scan or when you want to use many machines to run one scan.

It requires a POSIX environment, and the commands screen, rsync and nmap (of course). See the AGENT file for more information about that.

Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework

 Passive recon

The following steps will show some examples of passive network recon with IVRE. If you only want active (for example, Nmap-based) recon, you can skip this part.

Using Bro

You need to run bro (2.3 minimum) with the option -b and the location of the passiverecon.bro file. If you want to run it on the eth0 interface, for example, run:

# mkdir logs
# bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -i eth0

If you want to run it on the capture file (capture needs to a PCAP file), run:

$ mkdir logs
$ bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -r capture

This will produce log files in the logs directory. You need to run a passivereconworker to process these files. You can try:

$ passivereconworker --directory=logs

This program will not stop by itself. You can (p)kill it, it will stop gently (as soon as it has finished to process the current file).

Using p0f

To start filling your database with information from the eth0 interface, you just need to run (passiverecon is just a sensor name here):

# p0f2db -s passiverecon iface:eth0

And from the same capture file:

$ p0f2db -s passiverecon capture

Using the results

You have two options for now:

  • the ipinfo command line tool
  • the db.passive object of the ivre.db Python module

For example, to show everything stored about an IP address or a network:

$ ipinfo 1.2.3.4
$ ipinfo 1.2.3.0/24

See the output of ipinfo --help.

To use the Python module, run for example:

$ python
>>> from ivre.db import db
>>> db.passive.get(db.passive.flt_empty)[0]

For more, run help(db.passive) from the Python shell.

 

Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework: IVRE Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework Python Network Recon Framework

Active recon

Scanning

The easiest way is to install IVRE on the “scanning” machine and run:

# runscans --routable --limit 1000 --output=XMLFork

This will run a standard scan against 1000 random hosts on the Internet by running 30 nmap processes in parallel. See the output of runscans --help if you want to do something else.

When it’s over, to import the results in the database, run:

$ nmap2db -c ROUTABLE-CAMPAIGN-001 -s MySource -r scans/ROUTABLE/up

Here, ROUTABLE-CAMPAIGN-001 is a category (just an arbitrary name that you will use later to filter scan results) and MySourceis a friendly name for your scanning machine (same here, an arbitrary name usable to filter scan results; by default, when you insert a scan result, if you already have a scan result for the same host address with the same source, the previous result is moved to an “archive” collection (fewer indexes) and the new result is inserted in the database).

There is an alternative to installing IVRE on the scanning machine that allows to use several agents from one master. See theAGENT file, the program runscans-agent for the master and the agent/ directory in the source tree.

Using the results

You have three options:

  • the scancli command line tool
  • the db.nmap object of the ivre.db Python module
  • the web interface

CLI: scancli

To get all the hosts with the port 22 open:

$ scancli --port 22

Python module

To use the Python module, run for example:

$ python
>>> from ivre.db import db
>>> db.nmap.get(db.nmap.flt_empty)[0]

For more, run help(db.nmap) from the Python shell.

 The Web interface

The top navigation bar

It contains several elements; from left to right:

  • A shortcut to the start page, that cleans every keyword.
  • A button to display this help page.
  • Some menus with shortcuts to add filtering, sort or display commands.
  • Some links to “share” (export) the current page.

The left side bar

The first part allows to navigate within the results. Be careful with the last button that goes to the last result page, as it can be very slow when a lot of results are available.

The progress bar shows where the currently displayed results are within the whole results set.

The second part allows to add, modify or remove filter, sort or display commands.

The third part allows to explore the results by generating graphs displayed in the rightmost part of the screen.

  • The first field displays a graph with the 15 most common values of a variable in the filtered results. This can be slow when the number of results to scan is important. Here is a list of (sometimes) interesting values to try here:
    • category, source
    • country, city, as
    • domains, domains:[level]
    • hop, hop:[number]
    • port, port:[open/closed/filtered], port:[service] portlist:[open/closed/filtered]
    • service, service:[port], probedservice, probedservice:[port], product, product:[port], version, version:[port]
    • cpe, cpe.[type/vendor/product/version], cpe:[cpe spec], cpe.[type/vendor/product/version]:[cpe spec](examples: cpe.product:a:microsoft will show top product names in CPEs from vendor microsoft,cpe.vendor:o:/^m/ will show top vendor names in CPEs that start with an m)
    • devicetype, devicetype:[port]
    • script, portscript (like script), hostscript
    • script:[scriptname], portscript:[scriptname] (like script:), hostscript:[scriptname]
    • smb.os, smb.lanmanager, smb.domain, smb.dnsdomain, smb.forest, smb.workgroup
    • cert.issuer, cert.subject
    • modbus.deviceid, enip.vendor, enip.product, enip.serial, enip.devtype, enip.prodcode, enip.rev, enip.ip
  • The Address space button displays a graphical representation of the filtered addresses. The abscissa axis represents the two high bytes (or the three when the results belong to the same /16 network), and the ordinate axis represents the two low bytes (or the low byte).
  • The Map button displays the locations of the results on a world map.
  • The Timeline and Timeline 24h buttons display time-lines where the abscissa axis represents the time and the ordinate axis represents the IP addresses.

Scan results

Ten results (maximum) are displayed per page by default.

Each result has its own frame. In the default display mode, it displays a summary for the host. Long-clicking a result frame toggles between the summary display and the full display for the result.

The pencil icon in the upper-right corner opens the notepad page for the current host (see below) in the rightmost part of the screen.

Each blue element in the results can be clicked to add a filter.

Available commands

Command specification

The commands might require a parameter, provided after the colon sign :. Some commands can be used negatively, by prefixing them with ! or -.

The commands can be entered in the input boxes in the second part of the left side bar or added by clicking on a shortcut in the top bar menus.

In the following list, a [!] before the command shows it can be used negatively, and a : after the command indicates it requires a parameter.

When a parameter is required the full value must be specified, or when appropriate, a regular expression can be used, with the/[expression]/[flags] syntax (e.g.: hostscript:smb-enum-shares:/WRITE/).

If your command includes spaces, you need to protect it by using single or double quotes.

 Commands list

Filters

  • [!]archives show results from the archives database (and !archives has no effect since it is the default behavior).
  • [!]host:[IP address] filter a specific IP address. Using the IP address directly (without host:) is equivalent.
  • [!]net:[IP address/netmask] filter a specific network (CIDR notation). Using the CIDR notation directly (without net:) is equivalent.
  • [!]range:[IP address]-[IP address] filter a specific IP address range
  • [!]hostname:[FQDN] look for results with a matching hostname.
  • [!]domain:[FQDN] look for results with a hostname within a matching domain name.
  • [!]category: filter a category.
  • [!]country:[two letters code] filter a country.
  • [!]city: filter a city (use with country:).
  • [!]asnum: filter by AS number (lists allowed).
  • [!]asname: filter by AS name (regular expressions allowed).
  • [!]source: filter a source (specify the source name).
  • [!]timerange:[timestamp]-[timestamp] filter results within a specific time range.
  • [!]timeago: filter recent enough results; the value can be specified in seconds or with the appropriate suffix in minutes (m), hours (h), days (d) or years (y).
  • service:[expression], service:[expression]:[port number] look for an expression in the name of a service.
  • probedservice:[expression], probedservice:[expression]:[port number] look for an expression in the name of a service discovered with a service probe.
  • product:[service]:[product], product:[service]:[product]:[port number] look for a product.
  • version:[service]:[product]:[version], product:[service]:[product]:[version]:[port number] look for a specific version of a product.
  • script:[scriptid], script:[scriptid]:[output] look for a specific (port) script (using portscript: is equivalent).
  • hostscript:[scriptid], hostscript:[scriptid]:[output] look for a specific host script.
  • anonftp filter results with anonymous FTP allowed.
  • anonldap look for LDAP servers with anonymous bind working.
  • authbypassvnc look for VNC servers with authentication that can be bypassed.
  • authhttp look for HTTP servers with authentication and a default (e.g., admin/admin) login/password working. The Nmap script seems to get a lot a false positives.
  • banner: look for a specific banner of a service.
  • cookie: look for HTTP servers setting a specific cookie.
  • file: look for a pattern in the shared files (FTP, SMB, …).
  • geovision look for GeoVision web-cams.
  • httptitle: look for a specific HTML title value of the homepage of a web site.
  • nfs look for NFS servers.
  • nis, yp look for NIS servers.
  • mssqlemptypwd look for MS-SQL servers with an empty password for the sa account.
  • mysqlemptypwd look for MySQL servers with an empty password for the root account.
  • owa look for OWA (Outlook Web App) servers.
  • phpmyadmin look for phpMyAdmin servers.
  • smb.dnsdomain:[FQDN] search results with SMB service in a specific DNS domain.
  • smb.domain:[NetBIOS] search results with SMB service in a specific NetBIOS domain.
  • smb.fqdn:[NetBIOS] search results with SMB service in a specific host name (FQDN).
  • smb.forest:[FQDN] search results with SMB service in a specific forest (DNS name).
  • smb.lanmanager:[LAN Manager] search results with SMB service with a specific LAN Manager.
  • smb.os:[OS] search results with SMB service with a specific OS.
  • smb.server:[NetBIOS] search results with SMB service in a specific host name (NetBIOS).
  • smb.workgroup:[NetBIOS] search results with SMB service in a specific workgroup (NetBIOS).
  • sshkey: look for a particular SSH key.
  • torcert look for Tor certificates.
  • webfiles look for “typical” web files in the shared folders.
  • webmin look for Webmin servers.
  • x11open look for open X11 servers.
  • x11srv look for X11 servers.
  • xp445 look for Windows XP machines with TCP/445 port open.
  • os: look for a specific value in the OS discovery results.
  • devtype:, devicetype: look for a type of devices.
  • netdev, networkdevice look for network devices (firewalls, routers, …).
  • phonedev look for telephony devices.
  • cpe(:[type](:[vendor](:[product](:[version])))) look for a given cpe. Each field can be a /regex/.
  • [!]hop: look for a particular IP address in the traceroute results.
  • [!]hopname: look for a matching hostname in the traceroute results.
  • [!]hopdomain: look for a hostname within a matching domain name in the traceroute results.
  • [!]tcp/[port number], [!]udp/[port number], look for an open TCP or UDP port (using [!][port number] directly is equivalent to [!]tcp/[port number]).
  • [!]openport look for hosts with at least one open port.
  • notes search results with an associated note.

Sort

  • skip:[count] skip count first results.
  • limit:[count] only display count results.
  • [!]sortby:[field name] sort according to a field value. Be careful with this setting as consequences on the performances can be terrible.

Display

  • display:host set the default display mode.
  • display:cpe only display CPEs.
  • display:script:[scriptid] only display a particular script output.
  • display:screenshot only display screenshots.

Quelle: CyberPunk

Facebooktwittergoogle_plus