Hex 1.0.3 released!

We just released Hex 1.0.3, the Chinese New Year release, although it’s closer to the Valentine’s day release. Congratulations to all the Hex developers for fixing bugs and adding features!

You can grab the iso here. [md5] [sha256]
Or, grab the iso from the mirror. [md5] [sha256]

Since Geek00l already covered a list of the most important changes in his blog post, I’ll just echo the changes in NSM-Console, which is the software that I develop. The version of NSM-Console in Hex 1.0.3 is 0.6-DEVEL, which can be obtained either by checking out the code from subversion (svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console), or waiting until I release 0.6

New Features:
- ‘dump‘ command added, you can now dump packet payloads into a binary file for later analysis
- Significant speedups in the harimau module and ‘checkip‘ command if wget is installed
- tcpxtract configuration file changed to extract more types of files
- Added foremost module
- Added clamscan module (Thanks JohnQPublic)
- Argus and tcptrace have reverse dns turned off by default now, it was causing the module to hang for extremely large pcap files. Can be switched on by changed the module options
- rot13 encoding and decoding added
Bugfixes:
- alias command
- urlescape (en|de)coding
- file existence check
- many other things
All the other enhancements, bugfixes and additions.

Since Hex 1.0.2 had NSM-Console version 0.2 in it, if you haven’t downloaded the console or checked it out from subversion since the last hex release, here’s what else has been added since then:

• Release notes for NSM-Console version 0.5

• Release notes for NSM-Console version 0.4

• Release notes for NSM-Console version 0.3

Or you can view the TODO and CHANGELOG for NSM-Console directly. If you want to check out more information about either of the projects, check out the Trac pages for Hex and NSM-Console:

Now, onward to FreeBSD 7.0!

User-submitted modules: flowtag and clamscan

I’d like to point out a couple of user-submitted modules for NSM-Console that are now included in the distribution.

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command.

NSM-Console version 0.5 release

That’s right, no development release this time around. I’ve been trying to get version 0.5 all finished for the Hex 1.0.3 release, and I’m happy to present the newest NSM-Console release!

Firstly, you can download NSM-Console version 0.5 here:
http://navi.eight7.org/~hinmanm/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

NSM-Console version 0.4 release

Well, it has barely been any length of time and there’s already a new release of NSM-Console, there are so many features that I’ve been coding like crazy to get them all done. First, let’s start with the downloading:

http://navi.eight7.org/~hinmanm/files/nsm-console-0.4.tar.gz
[mirror] https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

NSM-Console version 0.3 release

Yep, I’ve just been cranking out code lately, so I am proud to present the 0.3 release of nsm-console!

You can download NSM-Console here:

http://navi.eight7.org/~hinmanm/files/nsm-console-0.3.tar.gz

This release was focused a bit more on usability, features and bugfixes rather than the addition of new modules, however, there were still a couple that were added. Since this release has some pretty big changes, let’s start by going over some of the notable ones:

Security blog roundup

Talking with enhanced in IRC, I realized that I should really do a post listing some of the security blogs that I read, so without further ado, I present you with the list (in no particular order)

My awesome wife’s food blog, Eatables

Hex people

• Geek00l – http://geek00l.blogspot.com/
• Enhanced – http://global-security.blogspot.com/
• Giovani – http://www.cblume.com/
• Chflags – http://bsd.b3ta.org/
• Ayoi – http://blog.hazrulnz.net/
• Takizo – http://www.takizo.com/blog/
• Spoonfork – http://mel.icious.net/blog/
• Bejtlich – http://taosecurity.blogspot.com/
• Security.org.my team – http://security.org.my/
• Fenris – http://blog.myfenris.net/

Other security blogs I read

• http://eatingsecurity.blogspot.com/
• http://erratasec.blogspot.com/
• http://blog.vorant.com/
• http://www.inliniac.net/blog/
• http://www.matasano.com/log/
• http://mcwresearch.com/
• http://blog.metasploit.com/
• http://johncrackernet.blogspot.com/
• http://muraliraju.info/
• http://www.joelesler.net/
• http://rationalsecurity.typepad.com/blog/
• http://rdist.root.org/
• http://www.rootsecure.net/
• http://www.sans.org/reading_room/
• http://www.securitydistro.com/
• http://securosis.com/
• http://spiresecurity.typepad.com/
• http://synfulpacket.blogspot.com/
• http://taossa.com/
• http://blog.uncommonsensesecurity.com/
• http://blog.vodun.org/
• http://www.veracode.com/blog/
• http://www.darkreading.com/

This doesn’t include the regular sysadmin blogs that I read also (as well as all the other stuff). Thanks to all the above for having great posts to read, they give me something to look forward to

Have another blog I should be checking out? Leave a comment letting me know about it, I’ll definitely appreciate it!

Some reference for locality in infosec

A week or so ago I wrote about locality of reference in regards to network security, I found some *actual* research done on the topic and wanted to share it:

http://www.cert.org/netsa/publications/Nspw2003-gates-locality.pdf

I’m still in eager anticipation of the first tool to use locality for malicious activity assesment.

NSM-console version 0.2 release

I found out there is internet here, so I’m finally able to post some code changes I was working on while on the airplane.

Firstly, download the files here.
The static page for nsm-console is here.

I finally got around to releasing the next version of the nsm-console. This version incorporates a large amount of bug fixes and additional features, first, I’ll start with some of the features I’m the most happy about Most of these features are in the new Hex 1.0.2 release which came out yesterday (go download it now!)

•   Categories
  • You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “toggle flow” to toggle the flow category (and all of it’s modules) on and off.
  • You can easily add your own categories to customize your work environment, all you have to do is create a file named the same name as the category name in the modules/categories directory.
• Directory analysis
  • When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “file” command to specify a directory full of files  instead of a single file. When the “run” command is executed, all the toggled module’s operations will be executed on each file in the directory (recursively)
  • To better accommodate this type of operation, I encourage anyone that is writing any modules to write them to output the results into an output file named something like ${PCAP_BASE}.tcpdstat.out (so if you had more than 1 file, the output will go into more than one file)
• The ‘exec’ command
  • I added the exec command because I was tired of spawning an additional shell in order to run a simple ‘tcpdump’. I hope this helps with the automation that I’m going to talk about below.

The directory functionality and the exec command isn’t in the current Hex release, but hopefully it will be in the next release.

Part of the reason I think nsm-console is neat is the ease of automation you can do using simple text files. For instance, if you created a text file called “automate.txt” and put the following lines in it:

file /pcap/data.pcap
output automated-output
toggle aimsnarf
toggle tcpdstat
toggle chaosreader
run
quit

Then, you can run the command:

./nsm < automate.txt > output.txt

Which will run all the commands in the text file automatically, placing all the output in output.txt, simple eh?

If you have any questions, comments or suggestions, feel free to leave a comment or send an email I’d love to hear if/how you’re using nsm-console

NSM Console – A framework for running things

Well, I’ve been hard at work for the last couple of days working on a (hopefully) useful tool for aiding in NSM file analysis (for pcap files, live analysis doesn’t work).

Behold! I present NSM-Console! (read more about it here, watch a screencast here)

Download the framework here.
Keep in mind this framework only includes 3 modules (mostly used just for testing)

NSM-Console in a small (< 500 1000 1500 lines) framework for running nsm modules. Essentially, it’s a framework for running things (but we don’t call it that because it sounds like it wasn’t any work ). Here’s the breakdown:

Joining the Hex Network Security Monitoring LiveCD project

Yep, that’s right, I’m going to be helping out with the Hex LiveCD project. I will hopefully be doing some development for some of the NSM tools and fixing bugs in the CD. In case you don’t know what Hex is, let me give you a little synopsis (from the Trac):

“HeX LiveCD is a Network Security Monitoring (NSM) centric Live CD, built based on the principles of NSM, for analsyst, by analyst. Besides containing most of the popular Open Source NSM tools, the HeX Live CD also contains tools to perform network forensics. This Live CD was built based on the 6.2 Release of FreeBSD, and provides Fluxbox as the default desktop environment. It also includes an installer for installing HeX to hard disk.”

I’m already working on extending the NBF (Network-Based Forensics) offline script to include support for my aimsnarf script. I’m excited about being able to help, hopefully I’ll be able to be useful to the project as well as learning some of the tools I haven’t used before myself

Geek00l will be leading the project with enhanced, chfl4gs and others all providing dev support. I was even going to wait until he announced it before writing this post, but he just keeps saying he’ll do it tomorrow

If you’re interested in the project or any of it’s details, feel free to join us on #rawpacket on Freenode for chatting! (I go by the nick dakrone on IRC)