" Make hlsearch work like less(1)
set hlsearch
nnoremap <ESC>u :nohlsearch<CR>
hi Search ctermfg=black ctermbg=white

For instance, I have one command that runs mplayer in a loop, in case the connection drops:

while true; do mplayer -cache 48 -prefer-ipv4 http://foo.com/streamthing; done

MPLAYER= while true; do mplayer -cache 48 -prefer-ipv4 http://foo.com/streamthing; done

Now, any time I want to rerun this specific command, I can just do !MPLAYER and we're all set. Doing this is *extremely* useful and allows you to define alias-like procedures in real-time, assuming you have a persistent shell history. If you don't have a persistent shell history, set it up, as it's useful for more things than the above hack.

This was my first Shmoocon. It's like Defcon but with less debauchery, more clueful presentations, more interesting people, less disinteresting people etc. Defcon is more about getting plastered, gambling, general debauchery, and using presentations as an excuse not to drink. In short, Shmoocon is better technical conference.

I flew out on Tuesday because I wanted to take a full week off of work. wxs and I needed to put more time into the puzzles, so I didn't have to kill time by sleeping or visiting landmarks. Time was spent sleeping, working on puzzles, eating, and playing Mario Kart on wxs's Wii.

The conference itself was more or less what I had expected. When I attend conferences, I usually end up spending more time out of the sessions than in them, due to my opinion that lots of talks are super boring. The topic usually sounds neat, but the presentation style sucks or the content is worthless. My favorite part of conferences is the side channel stuff.

This year's Shmoocon broke tradition and made the NOC open and availabile to anyone this year. I'm sad I didn't get a chance to go in and find out how they setup the network. They had "Shmoocon Labs" prior to the event which invited staff and attendees to come and set up the network the day prior to the conference. Sweet idea, but crazy (Let's setup a network for 1000 people in less than a day).

On to presentations! The Jikto talk was cool in that it was code manifestation of already-known vulnerabilities exposed by AJAX, XSS, and web proxies. The speaker accidentally showed the url where the source code lives when he did 'view source' for a few seconds during a demo. Of course, a fair portion of the room scribbled down the url and downloaded it; oops.

My favorite talk was the "No tech hacking" talk. The material was, like Jikto, simply an application of known techniques. In this case, it was social engineering and observation. The style was very engaging. The whole point seemed to be that hacking people is stupid easy because most people have credential and other items visibly on the outside.

I went to a talk about using entropy for statistical analysis, but the first 5 minutes of it were *really* slow and I pretty much got the idea of what the presenter was talking about in that time, so I left to find other things to do.

Hack or Halo. This year the hack was different. It was security/hack-type puzzles instead of the previous year of "exploit these machines as fast as possible". The puzzles ranged from sudoku to lanman hash cracking to port knocking. We had a total of 22 (ish?) puzzles, and only three went unsolved across all of the players

Prior to hack or halo, wxs and I were doing some final checks on the puzzles. We booted the machine and found immediately that none of the vmware instances would start. The folder 'C:\Virtual Machines' was permanently stuck in 'read only' mode. Unchecking 'read only' in the permissions box didn't fix it (it kept resetting to 'read only' again).

What now? Zoom back another day, when wxs and I were finishing the puzzles. My spider sense told me to back up the vmware images before shutting down, so I had wxs back them up to his laptop. He copied them over from his laptop after we realized the vmware images on the hack server weren't good anymore. They worked fine. Thank god for backups.

Other than that hitch, the hack portion of the competition went off without any problems at all. Whew. I have lots of pictures posted on flickr from the competition, greater Shmoocon, and shenanigans at the parties.

As far as conference work goes, working on HoH was pretty great. The other options for working Shmoocon seemed to be NOC or physical security. NOC stuff would've been fun, since it would let you play with the new fancy security network gear being tested or generally using gear I don't have access to on a normal basis. HoH didn't take too much of my time during the con, so it was totally worth it.

HoH was awesome, and I'm considering doing it next year. If you weren't there, you missed a great conference.

See you in DC.

ipv4_addrs_bge0="192.168.0.1-254/24 192.168.5.50-100/24"

and we get:

% ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::20a:e4ff:fe3f:92ee%bge0 prefixlen 64 scopeid 0x1 
        inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2
        inet 192.168.0.3 netmask 0xffffffff broadcast 192.168.0.3
...
        inet 192.168.0.253 netmask 0xffffffff broadcast 192.168.0.253
        inet 192.168.0.254 netmask 0xffffffff broadcast 192.168.0.254
        inet 192.168.5.50 netmask 0xffffff00 broadcast 192.168.5.255
        inet 192.168.5.51 netmask 0xffffffff broadcast 192.168.5.51
...
        inet 192.168.5.99 netmask 0xffffffff broadcast 192.168.5.99
        inet 192.168.5.100 netmask 0xffffffff broadcast 192.168.5.100
        ether 00:0a:e4:3f:92:ee
        media: Ethernet autoselect (none)
        status: no carrier

You have 1500 machines you want in cacti. How do you do it?

My take is that you shouldn't ever need to preregister data types or data sources. Have a system that you simply throw data at, and it stores it so you can get a graph of it later. All I need to do, to graph new data, is simply write a script that produces that data and sends it to the collector.

The collector is a python cgi script that frontends to rrdtool. It takes all cgi paramters and stores the values with a few exceptions:

• machine=XX - Spoof machine to store data for. If not given, defaults to REMOTE_ADDR. Useful if you need to proxy data through another machine, or are reporting data about another machine you are probing.
• timestamp=XX - Override default timestamp ("now").

Example:

kenya(/mnt/rrds/129.21.60.26) % ls
C_bytes_per_page.rrd                            C_pages_inactive.rrd
C_cpu_context_switches.rrd                      C_rfork_calls.rrd
... etc ...

Why are the files prefixed with "C_" ? The data I am feeding in comes from counters, and therefore should be stored as counter datatypes in rrdtool. The 'C_' prefix is a hint that if the variable needs an rrd created for it, that the DS type should be COUNTER. The default without this prefix is GAUGE.

Sample update http request:
http://somehost/updater.py?C_fork_calls=32522875&C_system_calls.rrd=235293874987

Feel free to view the vmstat -s poll script to get a better idea of what this does. I also have another script that will do some scraping on 'netstat -s' in freebsd (probably works in linux too).

vmstat -s looks like this:

456846233 cpu context switches
3220655757 device interrupts
 17964606 software interrupts
  ... etc ...

Anyway, back at the problem. Now how do I graph it? The interface isn't the best, but we use a cgi script again:

Show me all the machines with 'C_system_calls' graphed over the past 15 minutes:
graph.py?machines=129.21.60.1,<...>,129.21.60.26&keys=C_system_calls&start=-15min

This kind of system has the feature that you never need to explicitly define data input variables or data input sources - All you need is to hack together a script that can pump out key-value pairs. No documentation to read. No time consumed registering 500 new servers in your graph system.

repo=file:///path/to/repo
svn ls $repo/trunk | xargs -I@ -n1 svn mv $repo/trunk/@ $repo/@

I used svn poorly at first - one repository per project. To fix that, I needed to dump all of them (with svnadmin) and load them into a central repository:

# svnadmin dump all of my svn repositories
repodir="/home/foo
for i in $repodir/SVN/*; do 
  echo $i;
  svnadmin dump $i > $(basename $i).dump
done

# load all of my dumpped repositories into the new one
repo="/home/foo/NEWSVN"
svnadmin create $repo
for i in *.dump; do 
  proj="$(echo $i | cut -d. -f1)";
  svn mkdir -m "mkdir $proj for import" file://$repo/$proj
  svnadmin load --parent-dir $proj $repo < $i
done

Both FreeBSD's and Linux's (GNU, really) split(1) tool is fairly slow when you want to split by line count. Splitting by byte count is (should be) faster, but that will break logs because the byte boundary probably won't ever align on a log boundary..

I'd prefer both: the speed of byte size splitting and the sanity of line based splitting. To that end, I wrote a small tool to do just that. Yes, there's probably a tool already available that does exactly what I want, but this was only 100 lines of C, so it was quick to write. GNU split's --line-bytes option is mostly what I want, but it's still crap slow.

download fastsplit

Here's a comparison between my tool and gnu's split, run on the fastest workstation I have access to. My tool runs 4 times faster than gnu split for this task.

# Source file is 382 megs, which is tiny.
% du -hs access
382M    access

# (Fast) Split into approximately 20meg chunks while preserving lines.
% time ./fastsplit -b 20971520 -p ./split.fast. access 

real    0m1.260s
user    0m0.018s
sys     0m1.242s

# (GNU) Split into 87000-line chunks, no guarantee on size.
% time split -l 87000 access split.normal..

real    0m4.943s
user    0m0.395s
sys     0m2.440s

# (GNU) Split into 20mb (max) chunks, preserve lines.
% time split --line-bytes 20971520 access split.normal_bytes.

real    0m4.391s
user    0m0.001s
sys     0m1.779s

What's the output look like?

% wc -l split.fast.*
 86140 split.fast.00000
 81143 split.fast.00001
 92725 split.fast.00002
...
 91067 split.fast.00016
 86308 split.fast.00017
 84533 split.fast.00018
  1654604 total

% wc -l split.normal.*
 87000 split.normal.aa
 87000 split.normal.ab
 87000 split.normal.ac
...
 87000 split.normal.ar
 87000 split.normal.as
  1604 split.normal.at
  1654604 total

% wc -l split.normal_bytes.*
 85973 split.normal_bytes.aa
 80791 split.normal_bytes.ab
 92363 split.normal_bytes.ac
...
 86141 split.normal_bytes.ar
 85665 split.normal_bytes.as
  3999 split.normal_bytes.at
  1654604 total

% du -hs split.fast.*
21M     split.fast.00000
21M     split.fast.00001
21M     split.fast.00002
...
21M     split.fast.00016
21M     split.fast.00017
20M     split.fast.00018

% du -hs split.normal.*
21M     split.normal.aa
22M     split.normal.ab
19M     split.normal.ac
...
21M     split.normal.ar
21M     split.normal.as
352K    split.normal.at

% du -hs split.normal_bytes.*
21M     split.normal_bytes.aa
21M     split.normal_bytes.ab
21M     split.normal_bytes.ac
...
21M     split.normal_bytes.ar
21M     split.normal_bytes.as
896K    split.normal_bytes.at