Day 5 is online. It's about how to rock out with your friend, xargs(1).

day 5; xargs

Day 4 is finally ready for consumption, a bit late ;)

This article touches: cat, nc, ssh, openssl, GET, wget, w3m, and others. It's designed to show you a pile of tools you can use to pull data from various places.

day 4; data sources

Day 3 is ready for viewing. It's about awk.

This article has lots of usage examples for the many ways you can use awk to do hard work for you. Check out the article here:

day 3: awk

Day 2 is ready for viewing. It's about cut and paste.

Candice suggested these two tools are underutilized and are very useful when you need the features they provide.

Check out the article here:

day 2: cut and paste

Day 1 is ready for viewing. It's about sed, something I feel many sysadmins (and others) neglect in favor of perl, awk, or other tools. It's a super useful tool. Check out the article here:

Day 1: sed

This past week of work had me stretching my piping-ninja skills to the limits. In the past 4 days, I have created a oneliner that invoked xargs 3 times in a series of piped expressions, a oneliner that had xargs calling xargs, a oneliner that was well over 1000 characters and invoked sed, xargs, grep, perl, awk, xpathtool and others, sometimes twice, and several other ninja-like uses of filter-fu involving unix pipes. It's been a fun week for oneliners.

So, I thought I might spend the next 7 days covering some of the tools I find myself working with almost every day. I hope to cover all the ways I use a given tool, when it should be used over any alternatives, things to look up in the manpage, etc.

I'll try to cover more than one tool per day, because some tools can be batched together because there isn't much to day other than "this is useful for foo, bar, anz baz".

Tentative, incomplete list:

data sources

cat, GET, echo, nc (netcat), shell command substitution

filters

cut, grep, paste, sed, sort, tr, uniq, wc

magical

xargs, sh, awk, perl (for oneliners)

Here's a sneak peek at what you might learn:

'sort' can sort lines, right? But how do you sort words on the same line?

% echo "one two three four five" \
  | xargs -n1 | sort | xargs
five four one three two

Some of you might notice that the xargs(1) invocations can be replaced with tr(1). Yes. I use xargs(1) because it's less typing and handles many case tr(1) won't. This is not to say tr(1) doesn't have it's wonderful uses. More on that later this week ;)

If you have suggestions, let me know.

Using at(8), I can schedule jobs to occur, say, when I need to wake up.

nightfall(~) % atq
Date                            Owner           Queue   Job#
Thu Jul 13 07:30:00 PDT 2006    jls             c       14
Thu Jul 13 08:00:00 PDT 2006    jls             c       15
Thu Jul 13 08:20:00 PDT 2006    jls             c       16

All of those jobs run my 'wakeup.sh' script which is somewhat primitive, but it does the job.

Using this script: scripts/wakeup.sh

I have 89 log files. The average file size is 100ish megs. I want to parse all of the logs into something else useful. Processing 9.1 gigs of logs is not my idea of a good time, nor is it a good application for a single CPU to handle. Let's parallelize it.

I abuse /bin/sh's ability to background processes and wait for children to finish. I have a script that can take a pool of available computers and send tasks to it. These tasks are just "process this apache log" - but the speed increase of parallelization over single process is incredible and very simple to do in the shell.

The script to perform this parallization is here: parallelize.sh

I define a list of hosts to use in the script and pass a list of logs to process on the command line. The host list is multiplied until it is longer than the number of logs. I then pick a log and send it off to a server to process using ssh, which calls a script that outputs to stdout. Output is captured to a file delimited by the hostname and the pid.

I didn't run it single-process in full to compare running times, however, parallel execution gets *much* farther in 10 minutes than single proc does. Sweet :)

Some of the log files are *enormous* - taking up 1 gig alone. I'm experimenting with split(1) to split these files into 100,000 lines each. The problem becomes that all of the tasks are done except for the 4 processes handling the 1 gig log files (there are 4 of them). Splitting will make the individual jobs smaller, allowing us to process them faster becuase we have a more even work load across proceses.

So, a simple application benefiting from parallelization is solved by using simple, standard tools. Sexy.

nfsstat | sed -ne '/Version 3/,/^$/p'

sed++

When I was bored (at 4 am, no-less), I kept trying to parse this information out using some crazy tricks with 'x' (swap pattern/hold) and other stuff, but I forgot the fact that regexps are valid addresses. So, we can print anything between 'Version 3' and blank lines, anywhere in our output.

The next thing I want to try with this is to automagically parse nfsstat output into a format that is more machine readable, this will probably be using awk or perl, seeing as how doing it with sed may hurt my brain a bit. Furthermore, trying to read the sed that did said operations would be somewhat intense ;)

The output looks something like this, on Solaris 9:

Version 3: (535958 calls)
null        getattr     setattr     lookup      access      readlink    
0 0%        242223 45%  20606 3%    52504 9%    20025 3%    41 0%       
read        write       create      mkdir       symlink     mknod       
14138 2%    146618 27%  5525 1%     145 0%      337 0%      0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
6279 1%     7 0%        1539 0%     1518 0%     1606 0%     6587 1%     

Parsing this would mean generating a tree-like dictionary. In perl, it may look like:

%foo = (
	'Version 3' => {
		null => 0,
		getattr => 242223,
		setattr => 20606,
		lookup => 52504,
		# .... etc ...
		}
	)

Should be simple enough, we'll see what happens next time I get bored.

I gave a lecture on Unix basics this past friday in building 70. Attendance was about 20, which was far more than I expected. It was open to anyone who wanted to attend.

The slides can be found in the link below. They are html-based, and may require you to use a non-sucky browser to view them. I have not tested if browsers other than Firefox work, but I have heard that Opera and Safari choke on it. I'll fix that later.

Unix Basics slides

If you've never used my slide tool before, here's how:

• arrow keys (left/right) navigate back/forward
• spacebar goes forward
• 't' will toggle showing of the slide list.