Category Archives: geekery

Decoding the SANS Christmas packet challenge using only NSM-Console

In my never-ending quest to find justification for writing NSM-Console, I hereby present the following tutorial on how to decode the SANS Christmas packet challenge using nothing but NSM-Console:

I’m going to be using NSM-Console version 0.4-DEVEL, which adds the features that allow this analysis to be performed without external tools. You can get the development version here. Alright, let’s get this party started:

First things first, the fellows at SANS point you to the first packet in the xmas_Starter.pcap file, so let’s load up NSM-Console with the packet capture

./nsm ~/xmas_Starter.pcap

Next, let’s do a printout of all the packets in this dump (since it’s a small file, there shouldn’t be too many)

Continue reading

3 Comments

Filed under analysis, base64, challenge, christmas, console, decode, encode, fun, geek00l, geekery, hex, nsm, nsm console, packet, ruby, sans, terminal, urlescape

First suggestion for Project Indiana

Okay, so almost everyone has heard about Project Indiana right? The one where SUN tries to make Solaris like Linux so they can compete in more areas and get all the wonderful features of Solaris on more platforms. Well, I have a suggestion for you:

Don’t use Java for your installer.

Yea, sure, it’s fine if you use Java for the GUI installer *if* the machine can support it, but what about when I want to install on a machine with a minimal amount of RAM? I mean, even your text-based installer uses Java, and for what? Replace your Java text-based installer with something like Curses (or something equivalent). Make it easier to install Solaris for people who are in college. The install for Solaris almost assumes you’ve been through the install before and know what you’re doing. If you really want more adoption in a learner’s market, you need to make it simpler to install.

In other news: Happy Birthday OpenSolaris (you’re 2! whee!). Now if I could only install you on all of my really really old hardware so I could make headless servers. Alas needing much RAM to work.

Anyone know a good PCI SATA card that will work in my Blade 150? I’m tempted to get this, but I’m not sure if it’ll support JBODs without flashing the BIOS on the card, which would be a pain to do on a SPARC system.

1 Comment

Filed under geekery, solaris, sun

Frustrating: Kernel panics

Alright, so for the last 3 days or so, my main Solaris machine has been going crazy and kernel panicing about once every day or so, which is extremely annoying because every time it panics the machine reboots (and this machine has 3 zones that are in current use, so I get 3 calls about “why did my machine reboot”). Luckily, none of our servers here are production, so I get calls from development and not angry customers. So, I’m setting out to try and figure out why the machine is panicing. Here’s what I’m getting from the logs:

From the vmcore file:
ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0

From the /var/adm/messages file:
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::GPN_ID for D_ID=650700 failed
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::N_x Port with D_ID=650700, PWWN=5006016841e019a7 disappeared from fabric
Jun 5 12:01:30 lava2051 scsi: [ID 243001 kern.info] /pci@1c,600000/fibre-channel@1/fp@0,0 (fcp0):
Jun 5 12:01:30 lava2051 offlining lun=0 (trace=0), target=650700 (trace=2800004)
Jun 5 12:06:28 lava2051 unix: [ID 836849 kern.notice]
Jun 5 12:06:28 lava2051 ^Mpanic[cpu2]/thread=2a101061cc0:
Jun 5 12:06:28 lava2051 unix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0a04af0e949e:1a:f3): error 5)
... some stuff ...
Jun 5 12:09:55 lava2051 savecore: [ID 570001 auth.error] reboot after panic: ZFS: I/O failure (write on <unknown> of
f 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0
Jun 5 12:09:55 lava2051 savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/lava2051/*.1

Repeat x 3 so far. Like I said, extremely annoying.

Here’s what I think the problem is so far: I have a 500g ZFS pool built on a single Clariion LUN that is exported to this machine. From the looks of it the machine is having trouble seeing the LUN all the time, when it disappears ZFS freaks out and panics because of a I/O failure. Now that I know what the problem is, I have no idea how to make the LUN stop disappearing. Guess I’m off to check some Clariion logs and see where that gets me. Anyone out there have any other suggestions on how I could go about fixing this problem? I have little experience in working with core dumps. I would be extremely grateful :)

P.S. Yes, I know I should have mirrored the ZFS pool on 2 or more devices in case of a problem like this. This is more my “proof-of-concept” machine where I try out new things and see how developers/QA react to them.

UPDATE:
It looks like the problem was a problem on the Clariion side, for the meantime, we exported a LUN from a different clariion, did a zfs attach, waited for the data to be mirrored and then detached the old one. Fixed! <3 ZFS

UPDATE 2:
Now the data is mirrored to a different Clariion. fun fun. Interestingly enough, EMC doesn’t officially support ZFS on Clariion, only on Symmetrix.

Leave a Comment

Filed under clariion, crash, geekery, panic, software, solaris, zfs

Submission: local/remote zfs snapshot script

Here’s a nifty little submission from Ralf Ramge. It will do a ZFS snapshot backup to a local directory, a remote machine and also clone and promote the filesystem on the remote machine. It keeps the last 7 backups around. Take a look:

#!/bin/bash
# backup_zfssnap.sh, (c) 2007 ralf [dot] ramge [at] webde [dot] de

BACKUPDIR="/export/backup/snapshots"
DSTAMP=`date '+%y%m%d-%H%M%S'`
FILESYS=$1
DEST=$2
REPLICA=$3
BACKUPNAME=`echo $FILESYS | sed 's/\//_/g'`
BACKUPFILE=$BACKUPNAME"-"$DSTAMP".zfs"
SNAPSHOT=$FILESYS"@backup-"$DSTAMP

if [ ! -d $BACKUPDIR ]; then
echo "Backup Directory doesn't exist"
exit 1
fi

cd $BACKUPDIR

# Check here if we have 7 backup files, create them if we don't
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -le 1 ]; then
for COUNT in 1 2 3 4 5 6 7
do
if [ ! -f $BACKUPNAME"-000000-00000"$COUNT".zfs" ]; then
touch $BACKUPNAME"-000000-00000"$COUNT".zfs"
sleep 1
fi
done
fi

# Check here that we have less than 8 backup files
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -gt 7 ]; then
# echo "More than 7 backup files exist"
# exit 1
while [ $COUNT_FILES -gt 7 ]
do
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`
rm $OLDEST_BACKUP_FILE
let COUNT_FILES=COUNT_FILES-1
done
fi

# Find the oldest backup file to delete
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`

# Create the snapshot
zfs snapshot $SNAPSHOT

# Create a filesystem image in the local backup directory
zfs send $SNAPSHOT > $BACKUPDIR"/"$BACKUPFILE

# Check for $2 and, if exists, create a second copy on a remote host for tape archival
if [ ! -z $2 ]; then
`zfs send $SNAPSHOT | ssh root@$2 "cat >$BACKUPDIR/$BACKUPFILE"`
fi

# Check for $3 and, if exists, mirror the filesystem on the remote host
if [ ! -z $3 ]; then
`ssh root@$2 "zfs receive $3 < $BACKUPDIR/$BACKUPFILE"`
fi

# Check for $4 and, if exists, clone and promote the filesystem on the remote host
if [ ! -z $4 ]; then
`ssh root@$2 "zfs clone $SNAPSHOT $4; sleep 30; zfs promote $4"`
fi

# Get the trash out of the house
rm $OLDEST_BACKUP_FILE
if [ ! -z $2 ]; then
ssh root@$2 "rm $BACKUPDIR/$OLDEST_BACKUP_FILE"
fi

SNAPLIST=`zfs list -H | grep $FILESYS | grep @backup | cut -f1`
for i in $SNAPLIST; do
zfs destroy $i
done

# Exit cleanly
exit 0

Thanks for the submission Ralf! (I changed your email address in the script comments so you wouldn’t get spam)

Leave a Comment

Filed under backup, bash, geekery, script, solaris, sun, zfs

Ian Murdock at OpenSolaris users group

So I read quite a few Solaris blogs and when this popped up this morning I decide to take a look (warning, the movie that the post links to is >500 megs)

I had expected to hear a pretty good discussion around the “linuxification” of Solaris and how Ian Murdock plans to approach it, turns out about halfway through I was a little disappointed by the zealotry of some of the audience members. To *me* at least, it seems like they were arguing trivial points that led to the discussion going way off track. About 2/3 of the way through I turned if off so I could concentrate of a perl script I was writing (see below :P ). Here’s what my opinion is about the subject:

  • Who cares if you call the Linux userland “Linux” instead of “GNU”?? Most managers and people engaged in casual conversation reference the entire userland as Linux anyway, it makes it easier to talk about. Yes, everyone that is involved in OSS knows that Linux is just the kernel, but that seems like a pretty trivial point to make when you’re not even discussing that in the first place.
  •  Solaris needs a better userland. This, I agree with, I used to hate Solaris because I didn’t know how to do things with it, I think Ian makes a good point in that in college, the majority of students that ran an “alternative” operating system were running Linux, they knew it, they loved it, they wanted to use it when they got out of college (at least, I did). I certainly wish I had been experimenting with Solaris in college (I think I only did once). Now that I’ve been administering Solaris for the last year, it is by far my favorite administration platform, it might not be great for everything, but I certainly love it for my sysadmin work. Now if only the rest of the world would come to see the way I feel…
  • I commented about GNU having a better userland on a post on OSnews.com some time ago and someone alerted me to the fact that Solaris utilities have a better POSIX standardization than GNU utilities. After doing some poking around I definitely agree with that, I mean, in Linux, do you use -option? –option -option=? Is the manpage helpful ? (Hint: no). What I really miss are the features of the GNU tools, windowing in grep, -iname option for find. Things like that.
  • I read an article a couple days ago about how Solaris has a more powerful administration interface, however, Linux has an easier administration interface. I would say that’s about true. When it comes down to it, a lot of people are going to choose what they think will the best and easiest to administer. More education is needed. That and Linux is beating Solaris in online documentation by about 1000 webpages for every 1. Finding what I need for Solaris has always been a more extensive challenge to my GoogleFu than with Linux.
  • Almost every person that commented in the forum with Ian reminded me of that annoying guy from CS classes in college who thought he knew everything and was very elitist. Ugh, I just want to hit someone.

There you go, personal opinions that have almost no logical reason other than personal preference, way to go internet.

Ugh, re-reading this it is clear I am not an english major. Sorry for the disjointedness.

1 Comment

Filed under blogging, geekery, linux, solaris, sun

Not-as-simple perl script for ZFS snapshot auditing

Hi everyone, I’m back again with another perl script to hopefully be useful to a few of you.

Firstly, the script: http://lee.hinmanphoto.com/files/zdiff.txt (formatting long scripts in wordpress’ crazy editor is a very long and arduous process, thus I’m just linking to the script in this case, if anyone knows of a better place to stick it let me know). chmod +x it and away you go!

Edit: Sun was nice enough to host the file for me, here’s a link to their version in case the other one goes down: http://www.sun.com/bigadmin/scripts/submittedScripts/zdiff.txt

In a nutshell, here’s what it does:

  • Allows you to diff a file inside a ZFS snapshot with the current file in the filesystem and (optionally) print out the line differences
  • Recursively diff an entire snapshot using md5 sums and (optionally) printing out the line differences
  • Display the md5 sums for each file in a ZFS snapshot and filesystem (this can get old to look at very quickly)

Basically, that doesn’t mean a whole lot, here’s the output from the -h option:

ZFS Snapshot diff
./zdiff.pl [-dhirv] <zfs shapshot name> [filename]

-d Display the lines that are different (diff output)
-h Display this usage
-i Ignore files that don't exist in the snapshot (only necessary for recursing)
-r Recursively diff every file in the snapshot (filename not required)
-v Verbose mode

[filename] is the filename RELATIVE to the ZFS snapshot root. For example, if
I had a filesystem snapshot called pool/data/zone@initial. The filename '/etc/passwd'
would refer to the filename /pool/data/zone/etc/passwd in the filesystem and filename
/pool/data/zone/.zfs/snapshot/initial/etc/passwd in the snapshot.

A couple of examples:
./zdiff.pl -v -r -i pool/zones/lava2019@Fri
Checks the current pool/zones/lava2019 filesystem against the snapshot
returning the md5sum difference of any files (ignore files that don't
exist in the snapshot). With verbose mode

./zdiff.pl -d pool/zones/lava2019@Mon /root/etc/passwd
Check the md5sum for /pool/zones/lava2019/root/etc/passwd and compare
it to /pool/zones/lava2019/.zfs/snapshot/Mon/root/etc/passwd. Display
the lines that are different also.

Here’s what the output is going to look like:

-bash-3.00# ./zdiff.pl -d -v -r -i pool/zones/lava2019@Fri
Recursive diff on pool/zones/lava2019@Fri
Filesystem: /pool/zones/lava2019, Snapshot: Fri
Comparing: /pool/zones/lava2019/
to: /pool/zones/lava2019/.zfs/snapshot/Fri/
** /pool/zones/lava2019/root/etc/shadow is different
** MD5(/pool/zones/lava2019/root/etc/shadow)= 04fa68e7f9dbc0afbf8950bbb84650a6
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/shadow)= 4fc845ff7729e804806d8129852fa494
17d16
< tom:*LK*:::::::
** /pool/zones/lava2019/root/etc/dfs/dfstab is different
** MD5(/pool/zones/lava2019/root/etc/dfs/dfstab)= 8426d34aa7aae5a512a0c576ca2977b7
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/dfs/dfstab)= c3803f151cb3018f77f42226f699ee1b
13d12
< share -F nfs -o rw -d "Data" /data

etc, etc, etc.

I am planning on using it so I can audit certain files on different zones (like /etc/passwd) against an initial ZFS snapshot to see what’s changed. Nice little way to keep track of stuff. Email me with any bugs. Matthew dot hinman at gmail dot com.

7 Comments

Filed under geekery, perl, security, software, solaris, sun, work, zfs

Super-simple perl script for zfs snapshots

Here’s a simple script for creating daily zfs snapshots that get rotated every week (so you always have one for Mon, Tues, Wed, etc)

#!/usr/bin/perl

use warnings;
use strict;

my $zfsname = shift || die "Need a filesystem name\n";

my $day = `date`;

$day =~ s/(Sun|Mon|Tue|Wed|Thu|Fri|Sat)[\S\s]+/$1/gi;

my $snapname = "$zfsname\@$day";
my $exist = system("zfs list $snapname");

# if it already exists, delete it
if($exist == 0) {
print "Destroying previous weeks snapshot...";
system("zfs destroy $snapname");
print "done.\n";
}

print "Creating daily snapshot...";
system("zfs snapshot $snapname\n");
print "done.\n";

And here’s the crontab entry:
0 2 * * * /usr/sbin/zsnap.pl pool/zones/lava2019
(replace “pool/zones/lava2019″ with whichever zfs you want a snapshot of)

Easy as cake, you’ll always have a zfs snapshot called <zfsname>@Sun through <zfsname>@Sat to rollback to!

7 Comments

Filed under geekery, perl, solaris, sun, zfs

Linux firewall configuration

Basic iptables firewall conf only letting ssh and DNS through:

# Generated by iptables-save v1.2.11 on Thu May 17 14:52:04 2007
*filter
:INPUT DROP [13164:946396]
:FORWARD ACCEPT [0:0]
:OUTPUT DROP [0:0]
-A INPUT -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 128.222.228.235 -p tcp -j ACCEPT
-A INPUT -s 128.222.228.235 -p udp -j ACCEPT
-A INPUT -s 128.222.228.236 -p tcp -j ACCEPT
-A INPUT -s 128.222.228.236 -p udp -j ACCEPT
-A INPUT -s 128.222.12.10 -p tcp -j ACCEPT
-A INPUT -s 128.222.12.10 -p udp -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A OUTPUT -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -j ACCEPT
-A OUTPUT -p udp -j ACCEPT
-A OUTPUT -p icmp -j ACCEPT
COMMIT
# Completed on Thu May 17 14:52:04 2007

(128.222.228.235/236 and 128.221.12.10 are our DNS servers, I also accept pings too because I’m nice like that and people around here tend to freak out if they can’t ping their machine. I also let anything out, easy to comment out to deny outbound traffic.)

Leave a Comment

Filed under firewall, geekery, linux, security, software

Use SVM to make RAID0 and RAID1 meta-partitions

Firstly, the easy one:

RAID0:
Given 4 slices, each ~5g:

First, need a metadb, I created a 100MB slice on c1t1d0s0 (which I am NOT using for the RAID, entirely separate drive) and ran this command to initiate the database. It is a good idea to mirror the database in a minimum of 3 positions, but that is beyond the scope of this tutorial
metadb -a -f c1t1d0s0

Then, it’s as easy as 1 command to bring multiple drives into one slice/partition with the following command:
metainit d100 1 4 c2t2d0s0 c2t3d0s0 c2t4d0s0 c2t5d0s0
NOTE: I already created slice 0 on each of the drives.

To see the status of your meta-slice:
metastat d100
d100: Concat/Stripe
Size: 40878080 blocks (19 GB)
Stripe 0: (interlace: 32 blocks)
Device Start Block Dbase Reloc
c2t2d0s0 0 No Yes
c2t3d0s0 4096 No Yes
c2t4d0s0 4096 No Yes
c2t5d0s0 4096 No Yes

Device Relocation Information:
Device Reloc Device ID
c2t2d0 Yes id1,sd@n6006048cb0ca0ceeef67fa7a33ce4c94
c2t3d0 Yes id1,sd@n6006048cb275dda20f654d7248d17197
c2t4d0 Yes id1,sd@n6006048c5aa658e3c69370f2bad75bc0
c2t5d0 Yes id1,sd@n6006048cc092136a695a21eeaa948f88

See? Now we’ve got a 19GB slice. Feel free to newfs /dev/md/dsk/d100 and mount it somewhere fun.

Next up: RAID1
This is actually not as hard as it looks. First, make sure you init your database like the first step from above. Then initialize your first meta slice:
metainit d101 1 1 c2t2d0s0

Then, create the mirror for that slice which will become your final RAID1 slice by issuing the following command:
metainit d100 -m d101

Then initialize the other slices in your mirror, in this care there are 3 additional slices:
metainit d102 1 1 c2t3d0s0
metainit d103 1 1 c2t4d0s0
metainit d104 1 1 c2t5d0s0

From there, it’s quite easy to finish it up by attaching the mirrors:
metattach d100 d102
metattach d100 d103
metattach d100 d104

Then, monitor metastat for the sync progress percentage until all the mirrors are sync’d. Finished!
metastat d100

1 Comment

Filed under geekery, hardware, raid, software, solaris, sun, svm, tutorials, work

Getting EMC Celerras to work for iscsi on Solaris 10

For fun and profit!

Basically, for my own categorization:

1. Celerra-side:
Create filesystems (I am using 4 because I want to stripe across all 4:
nas_fs -n iscsiRAID1_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID2_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID3_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID4_5g -c size=5G pool=clar_r5_performance

Mount filesystems:
server_mount server_2 iscsiRAID1_5g /iscsiRAID1_5g
(repeat for all 4 filesystems)

Create iscsi target:
server_iscsi server_2 -target -alias target_3 -create 1000:np=10.5.140.151
(10.5.140.151 is the datamover IP for this Celerra, “target_3″ is the target name)

Create iscsi LUNs:
server_iscsi server_2 -lun -number 1 -create target_3 -size 5000 -fs iscsiRAID1_5g
server_iscsi server_2 -lun -number 2 -create target_3 -size 5000 -fs iscsiRAID2_5g
server_iscsi server_2 -lun -number 3-create target_3 -size 5000 -fs iscsiRAID3_5g
server_iscsi server_2 -lun -number 4 -create target_3 -size 5000 -fs iscsiRAID4_5g

I am creating 4 luns, 1 for each of the 4 filesystems

2. On the Sun side:
iscsiadm modify discovery --sendtargets enable
iscsiadm add discovery-address 10.5.140.151:3260

(10.5.140.151 is the datamover for our Celerra, it will be our iscsi target)

Run this command so you can get the initiator node name:
iscsiadm list initiator-node
It’ll spit out something that looks like this:
Initiator node name: iqn.1986-03.com.sun:01:ba88a3f5ffff.4648d8d8
Initiator node alias: -
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Configured Sessions: 1

We’re interested in the bold part up there, the part that starts with iqn.blahblahblah

Back on the Celerra:
server_iscsi server_2 -mask -set target_3 -initiator iqn.1986-03.com.sun:01:ba88a3f5ffff.4648d8d8 -grant 1-4
(use the initiator you got from the previous command, we are granting access to LUNs 1 through 4 (our raid LUNs))
And start the iscsi service if it hasn’t been started already:
server_iscsi server_2 -service -start
You are now completely done on the Celerra side, you can log off.

Back on the Sun:
Run this command to make sure you can see your targets alright
iscsiadm list target
Target: iqn.1992-05.com.emc:apm000650039080000-3
Alias: target_3
TPGT: 1000
ISID: 4000002a0000
Connections: 1

You should see something similar to the above. If you do, you now have a successful connection to the Celerra for iscsi. Don’t forget to create device nodes for your drives by running this:
devfsadm -i iscsi
Now run “format” and you should be able to see your drives show up. Don’t forget to open port 3260 in your firewall so that iscsi traffic can get through.

You should now be in business with your 4 drives. I’m still working on the RAID/mirror/striping part. I will add another post once I figure this out.

If you run into an error where the iscsi driver will not online, take a look at this link.

3 Comments

Filed under geekery, hardware, iscsi, software, solaris, sun, tutorials, work