Create a passive network tap for your home network

In my home network, I have a passive tap sitting between my cable modem and my router, instead of spending tons of money, I made my own. They’re surprisingly simple to make, and also extremely simple to use.

Let’s start with the wiring, at a local electronics store, I purchased 4 RJ-45 wiring plugs, I probably shouldn’t have bought solder-less ones, but I didn’t feel like buying a board to solder them to. Anyhow, 2 of the ports will be used for entry and exit, the other two for taps. In this case, we need 2 extra ports so that inbound data is passed through one port, outbound data is passed through the other port.

Set up the wiring as shown in this wiring diagram (credit goes to the Snort team for the diagram):

Personally, I split open a network cable and used the wires inside just so the color coding could be correct, that’s probably the easiest way to wire the ports.

After wiring the ports, you should be able to test that data passed from one host port to the other host port is unchanged, below is a picture of the tap I created. Yes, I know it’s very messy, the box I bought for it didn’t fit the way I wanted.

The next thing to do it connect the two ports (labeled “tap 1″ and “tap 2″ in the picture above) to 2 NICs in the machine of your choice. I’m using FreeBSD to manage the bridge. If you want to monitor outbound and inbound traffic separately, you’re done, just start tcpdump on the interface and you should be able to see all the traffic.

If you want to monitor both outbound and inbound traffic on the same interface, you’ll need to bridge the interfaces. You can accomplish this in FreeBSD with the following:

shell> ifconfig bridge create
shell> ifconfig bridge0 addm ed0 addm ed1 monitor up
shell> tcpdump -i bridge0
(or run snort/bro-ids/argus/etc on interface bridge0)

In this case, my network cards are ed0 and ed1, if you had different network interfaces, substitute them instead. You don’t need to assign an address to the bridge interface, since the only wires that are connected are the receive wires, so it wouldn’t transmit through the taps if it wanted to. For more advanced bridging, check out the FreeBSD manual on bridging.

I should note though, that you’ll need a 3rd network card in the monitoring machine if you want to remotely manage the machine.

Hex 1.0.3 released!

We just released Hex 1.0.3, the Chinese New Year release, although it’s closer to the Valentine’s day release. Congratulations to all the Hex developers for fixing bugs and adding features!

You can grab the iso here. [md5] [sha256]
Or, grab the iso from the mirror. [md5] [sha256]

Since Geek00l already covered a list of the most important changes in his blog post, I’ll just echo the changes in NSM-Console, which is the software that I develop. The version of NSM-Console in Hex 1.0.3 is 0.6-DEVEL, which can be obtained either by checking out the code from subversion (svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console), or waiting until I release 0.6

New Features:
- ‘dump‘ command added, you can now dump packet payloads into a binary file for later analysis
- Significant speedups in the harimau module and ‘checkip‘ command if wget is installed
- tcpxtract configuration file changed to extract more types of files
- Added foremost module
- Added clamscan module (Thanks JohnQPublic)
- Argus and tcptrace have reverse dns turned off by default now, it was causing the module to hang for extremely large pcap files. Can be switched on by changed the module options
- rot13 encoding and decoding added
Bugfixes:
- alias command
- urlescape (en|de)coding
- file existence check
- many other things
All the other enhancements, bugfixes and additions.

Since Hex 1.0.2 had NSM-Console version 0.2 in it, if you haven’t downloaded the console or checked it out from subversion since the last hex release, here’s what else has been added since then:

• Release notes for NSM-Console version 0.5

• Release notes for NSM-Console version 0.4

• Release notes for NSM-Console version 0.3

Or you can view the TODO and CHANGELOG for NSM-Console directly. If you want to check out more information about either of the projects, check out the Trac pages for Hex and NSM-Console:

Now, onward to FreeBSD 7.0!

User-submitted modules: flowtag and clamscan

I’d like to point out a couple of user-submitted modules for NSM-Console that are now included in the distribution.

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command.

NSM-Console version 0.5 release

That’s right, no development release this time around. I’ve been trying to get version 0.5 all finished for the Hex 1.0.3 release, and I’m happy to present the newest NSM-Console release!

Firstly, you can download NSM-Console version 0.5 here:
http://navi.eight7.org/~hinmanm/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

NSM-Console version 0.4 release

Well, it has barely been any length of time and there’s already a new release of NSM-Console, there are so many features that I’ve been coding like crazy to get them all done. First, let’s start with the downloading:

http://navi.eight7.org/~hinmanm/files/nsm-console-0.4.tar.gz
[mirror] https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

NSM-Console version 0.3 release

Yep, I’ve just been cranking out code lately, so I am proud to present the 0.3 release of nsm-console!

You can download NSM-Console here:

http://navi.eight7.org/~hinmanm/files/nsm-console-0.3.tar.gz

This release was focused a bit more on usability, features and bugfixes rather than the addition of new modules, however, there were still a couple that were added. Since this release has some pretty big changes, let’s start by going over some of the notable ones:

Some reference for locality in infosec

A week or so ago I wrote about locality of reference in regards to network security, I found some *actual* research done on the topic and wanted to share it:

http://www.cert.org/netsa/publications/Nspw2003-gates-locality.pdf

I’m still in eager anticipation of the first tool to use locality for malicious activity assesment.

NSM-console version 0.2 release

I found out there is internet here, so I’m finally able to post some code changes I was working on while on the airplane.

Firstly, download the files here.
The static page for nsm-console is here.

I finally got around to releasing the next version of the nsm-console. This version incorporates a large amount of bug fixes and additional features, first, I’ll start with some of the features I’m the most happy about Most of these features are in the new Hex 1.0.2 release which came out yesterday (go download it now!)

•   Categories
  • You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “toggle flow” to toggle the flow category (and all of it’s modules) on and off.
  • You can easily add your own categories to customize your work environment, all you have to do is create a file named the same name as the category name in the modules/categories directory.
• Directory analysis
  • When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “file” command to specify a directory full of files  instead of a single file. When the “run” command is executed, all the toggled module’s operations will be executed on each file in the directory (recursively)
  • To better accommodate this type of operation, I encourage anyone that is writing any modules to write them to output the results into an output file named something like ${PCAP_BASE}.tcpdstat.out (so if you had more than 1 file, the output will go into more than one file)
• The ‘exec’ command
  • I added the exec command because I was tired of spawning an additional shell in order to run a simple ‘tcpdump’. I hope this helps with the automation that I’m going to talk about below.

The directory functionality and the exec command isn’t in the current Hex release, but hopefully it will be in the next release.

Part of the reason I think nsm-console is neat is the ease of automation you can do using simple text files. For instance, if you created a text file called “automate.txt” and put the following lines in it:

file /pcap/data.pcap
output automated-output
toggle aimsnarf
toggle tcpdstat
toggle chaosreader
run
quit

Then, you can run the command:

./nsm < automate.txt > output.txt

Which will run all the commands in the text file automatically, placing all the output in output.txt, simple eh?

If you have any questions, comments or suggestions, feel free to leave a comment or send an email I’d love to hear if/how you’re using nsm-console

Locality of reference in information security

I’ve been kicking this idea around in my head for the last couple of days, trying to decide what to write…

Return with me, for a moment, back to the computational hardware class you took in college (if you did take one, don’t worry if you didn’t). Do you remember discussing program/memory flow? How about locality of reference for RAM access? Well, I wanted to discuss a few ideas about locality of reference in regards to network security.

First off, let’s define the different kinds of locality of reference, first there is temporal locality, then spatial locality and finally sequential locality. Here are the quick-and-dirty definitions:

Temporal locality states that when you access something, it is likely you will access the same thing again

Spatial locality states that when you access a particular piece of data, you are likely to access data near that piece in the future.

Sequential locality states that when you access a piece of data, you are likely to access nearby data in either an ascending or descending fashion.

So, what does this have to do with information security? (if anything) Well, I believe you can apply this kind of data collection methods to network traffic when looking at a whole network. For a simple example, take for instance 2 machines, each one one serving DNS records to clients on a network. In a typical example, a client would query one of the two machines (probably the first one, using the 2nd as backup if the first didn’t respond), retrieve it’s name record and be done with the connection right? How about in the case of something scanning the network for DNS servers as potential targets? You could expect to see both DNS servers queried within a short amount of time, by the same host. In this case, the person doing automated scanning exhibited symptoms of locality (in this case, spatial, although it could have been sequential depending on the IP assignments) when scanning for vulnerabilities.

How does this help us? Well, as we increase in our security monitoring, we may be able to gain an additional edge over scanning tools by classifying network traffic according to the locality of it’s flow. An nmap scan of an entire subnet (even just 1 port), for example, would be displaying sequential locality that would most likely not show up during legitimate use. An nmap scan of 1 host, all ports (let’s pretend nmap doesn’t randomize port scanning order) shows sequential locality as far as ports are concerned (which is also an example of determining wether it was automated or human scanning.

Each of the different kinds of locality you would expect to see in a different environment, in the case of temporal locality, if you had, say, a DHCP server, you would expect to see a small amount of temporal locality between hosts (for legitimate uses) since each host would only send out a DHCP request if it either lost it’s current address (needed to renew), or was just joining the network for the first time. Seeing one host exhibit a great deal of DHCP temporal locality (say…requesting a DHCP license 50 times in 1 minute), should trigger an alarm.

Another thing this can help us determine is whether it is a real-live person generating this traffic, or an automated tool. Example:

A live person is gathering data about auditing a network, they decide to start with DNS servers, they gather a list of DNS servers, then manually gather information about each server by attempting to log in, grabbing banners, etc.

An automated tool scans a subnet, noting any DNS servers it finds, it then (sequentially), attempts to gather information from each server the same exact way. It makes no distinction between different kinds of DNS servers, etc, the network traffic sent is the same for each server.

A sensor, capturing this traffic, looks at the traffic sent by the live person, sees the lack of a sequential scan packets against DNS servers, the direct approach (“I know what information I’m gathering and how to gather it directly without resorting to a scan”) and notes the lack of sequential and temporal locality.

The same sensor looks at the traffic sent by the automated tool, sees the network flow example of sequential locality (scanning a subnet, incrementing IPs by 1) and temporal locality (since many automated tools query the same server multiple times). It also compares the traffic sent to each of the different DNS servers (was the exact same query sent each time?) and from that, determines the amount of locality exceeds the threshold to classify it as an automated scan/attack.

Unfortunately, I don’t have any kind of tool to do this kind of analysis right now, I also don’t know of any tool that specifically handles looking at data with locality of reference in mind. Think about your network, what servers do you have where temporal locality would be expected? Now what kind of locality would you NOT expect to see against the same machine, website crawling? Port traversal? What locality differences would you expect to see in a human vs. automated usage of your network? How about applying it at a filesystem level, expecting to see sequential file access for a group of files?

I certainly don’t claim to be an expert in locality of reference (or NSM, for that matter), but I was curious if anyone has come across anything else like this. If you have, leave me a comment with a link to a paper or article about it, I’m very interested in reading more about it. I would appreciate it

Network traffic IP Location aggregator (iploc)

Have you ever been looking through your pcap files (or live captures) and wondered where all the traffic was coming from (or going to)? I have! Well, I’ve written a small (< 150 lines) script to aggregate all of the packet source addresses into a neatly separated CSV (comma-separated values) file. It includes

<ip address>,<country>,<city and state>,<latitude>,<longitude>,<packet count>

First off, get the script here.

It requires ruby-pcap and wget (like most of the Ruby scripts I write ). Each unique ip address will have the script query the hostip.info database, storing the info in a temp file. If the ip already exists in the script’s list, it won’t query for the information (to save time and bandwidth).

iploc can either run in live capture mode, or read from a pcap file. Here’s how you would run it live:

./iploc -i eth0 >> output.csv

iploc will run until it receives a CTRL-C, at which point it will aggregate the data and dump it to STDOUT. Alternatively, you can read from a pcap file like this:

./iploc -r ~/data.pcap >> output.csv

Here’s an example of what the CSV file looks like after being opened in Numbers (or Excel):

Since it’s a CSV file, it should be easy to parse for any other program. I’m planning on hopefully writing another program to take out the latitude/longitude and plot the points on a map (for a more visual representation).

IPloc also supports BPF filters, just use it like you would tcpdump, for instance, if you only cared about http data:

./iploc -i eth0 port 80 > output.csv

Questions? Leave me a comment below! Now go forth, and visualize your honeypot data!

P.S. Thanks to the hostip guys for having a really nice web api to get this data quickly.