I got A.D.D. tonight while working on Pimp and decided to Google myself and see what came up. I do this once very few months. Normally the results aren't terribly interesting. However, since I'd recently attended a few barcamps, I noticed that some folks had written about some of the talks I'd given.

I found a video of my 'firewall bypass/ssh tunnel/encrypt-your-traffic' session I held at BarCamp Stanford. Thanks to ValleyGeek for posting this. I'm assuming the author recorded it.

Towards the end of the talk I start rambling about nat traversal. Hopefully I got the point across that nat<->nat traversal isn't easy, and that tunneling is a way to do it.

http://video.google.com/videoplay?docid=-3694684117834367516

Along with that video, I found my urban-dictionary entry for 'Fo Shizzle Friday'. I can't remember why I created that.

Once upon a time, there was a Solaris 10 box where I wasn't able to use ssh keys to login.

Thankfully, that time has now passed. The problem was because PAM was denying access with public keys.

Running sshd in debug mode (-ddd) I would see this:

Found matching DSA key: 80:aa:32:03:ef:51:9c:7b:0f:1d:ac:37:17:d5:fd:2b
debug1: restore_uid: 0/0
debug1: ssh_dss_verify: signature correct
debug2: Starting PAM service sshd-pubkey for method publickey
debug3: Trying to reverse map address 69.181.132.53.
debug2: userauth_pubkey: authenticated 0 pkalg ssh-dss
Failed publickey for psionic from 69.181.132.53 port 55957 ssh2

Clearly indicated here, is the fact that it accepted my ssh-dss key, but I failed for some other reason. Listed here, is: Starting PAM service sshd-pubkey for method publickey. Solaris 10's manpage for sshd shows that it uses different PAM service names for each type of authentication.

The solution involved adding a simple service entry in /etc/pam.conf:

sshd-pubkey    auth required           pam_unix_cred.so.1

It works now. This takes effect immediately as the pam config is invoked any time sshd uses pam, so you don't have to restart sshd.

I keep an eye on my apache access logs to see what kind of traffic my site gets and why it gets here. It seems that a non-trivial number of searches are for 'vpn over ssh' and similar variants. These land at the ppp over ssh article.

New versions of openssh have built-in support for tunneling, and do not require ppp at all. Seeing as how I've never really used this new feature, and there's a nontrivial number of searches ending up on the ppp-over-ssh article, I think it's time to write a little article on how to use the new openssh built-in tunneling.

Stay tuned...

For pimp, I want to be able to search a specific column, say, artist, without needing multiple fields for searching. The ability to specify more advanced searches than simple keywords is quite useful. How do we leverage this on the client and turn a search query into a set of key-value pairs?

I must confess I was hesitant to put this kind of logic into Javascript instead of Python. Furthermore, it makes me feel a little uneasy using /foo/ in anything other than Perl. Nonetheless, doing this in Javascript was simple and it's still fast (as it should be).

The particular type of query I want to parse come in the following (hopefully intuitive) formats:

• foo
• artist:Eminem
• album:"Across a Wire"
• artist:"Counting Crows" album:august

The following code does this for me. The parse_query function will return a dictionary of query terms. Values are lists.

Here's an example:

Query

rain baltimore artist:"Counting Crows" album:august

Results of parse_query

    { "artist": ["Counting Crows"],
      "album": ["august"],
      "any": ["rain", "baltimore"],
    }
  

I take the dictionary returned and pass it to jQuery's $.post function to execute an AJAX (I hate that term, it's such a misnomer these days) request. Here's the code:

query_re = /(?:"([^"]+)")|(?:\b((?:[a-z:A-Z_-]|(?:"([^"]+)"))+))/gi,

function parse_query(string) {
  dict = {}
  while (m = query_re.exec(string)) {
    val = (m[1] || m[2]).split(":",2)
    if (val[1]) { key = val[0]; val = val[1]; }
    else { key = "any"; val = val[0]; }

    val = val.replace(/"/g,"");

    dict[key] = dict[key] || [];
    // the following should be .append(val) but 
    // I don't think javascript lists have them
    dict[key][dict[key].length] = val;
  }

  return dict;
}

I finally got my Verizon EVDO card working in FreeBSD. The following steps are necessary:

1. Add a new vendor line to sys/dev/usb/usbdevs
2. Add a new product line to sys/dev/usb/usbdevs
3. Download http://www.cs.cmu.edu/~dga/dot/fbsd_pc5220/ugencom.c
4. Update sys/conf/files with a new entry

usbdevs changes

In /usr/src/sys/dev/usb/usbdevs, add the following:

vendor QUALCOMM3        0x0c88  Qualcomm
product QUALCOMM3 CDMA_MSM      0x17da  CDMA Technologies MSM phone

Download ugencom.c

http://www.cs.cmu.edu/~dga/dot/fbsd_pc5220/ugencom.c"

Download this to /usr/src/sys/dev/usb/ugencom.c. Now, you need to make a slight change to this file. Change the following the following line:

        { USB_VENDOR_AIRPRIME, USB_PRODUCT_AIRPRIME_PC5220, -1 },

To this:

        { USB_VENDOR_QUALCOMM3, USB_PRODUCT_QUALCOMM3_CDMA_MSM, -1 },

Update sys/conf/files

In /usr/src/sys/conf/files, add the following just after the line containing 'uplcom.c'

dev/usb/ugencom.c               optional ugencom ucom

Build your kernel

Add the following to your kernel:

device ucom
device ugencom

Rebuild your kernel. This device should show up as a usb serial device: /dev/cuaU0

ppp.conf

This is mostly stolen from another fellow, but modified the authname and added the comment regarding it.

# Start by running ppp -ddial evdo  
evdo:
 set device /dev/cuaU0
 set speed 230400
 set dial "ABORT BUSY ABORT NO\\sCARRIER TIMEOUT 5 \
           \"\" AT OK-AT-OK ATE1Q0s7=60 OK \\dATDT\\T TIMEOUT 40 CONNECT"
 set phone "#777"
 # The authname and authkey are meaningless, but you need to have them set.
 # Verizon's servers don't seem to care what you auth as.
 set authname "doesn't matter"
 set authkey "doesn't matter either"
 set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.0 0.0.0.0
 add default HISADDR                    # Add a (sticky) default route
 enable dns

I normally use HEAD(1) to view HTTP response headers. However, when developing web applications outside of Apache or any other standard web server, it's often likely that you won't immediately support HEAD requests. This is a problem if all you want to do is see the http headers.

Perl's libwww comes with a tool called lwp-request, which is installed with symlinks GET, HEAD, etc. You can use GET and have it show only the HTTP headers. Or can you?

The response message I am looking for is this:

HTTP/1.0 200 OK
Server: PasteWSGIServer/0.5 Python/2.4.3
Date: Tue, 19 Dec 2006 22:42:20 GMT
content-type: text/html; charset=UTF-8
Connection: close

The response I get using GET -sed is:

200 OK
Connection: close
Date: Tue, 19 Dec 2006 22:43:16 GMT
Server: PasteWSGIServer/0.5 Python/2.4.3
Content-Type: text/html; charset=UTF-8
Client-Date: Tue, 19 Dec 2006 22:43:16 GMT
Client-Peer: 127.0.0.1:5000
Client-Response-Num: 1
Link: </css/style.css>; media="all"; rel="stylesheet"; type="text/css"
Link: </css/tabs.css>; media="all"; rel="stylesheet"; type="text/css"
Link: </css/tabs-ie.css>; media="all"; rel="stylesheet"; type="text/css"
Title: pimp: music for you.

What? LWP appears to be doing some html parsing. :(

So, a solution usiung netcat, which is trash, because lwp should just not be stupid by default. All I want are http headers. Here's my netcat solution:

% echo "GET / HTTP/1.0\r\n\r\n" | nc localhost 5000  | col | sed -e '/^$/q'
HTTP/1.0 200 OK
Server: PasteWSGIServer/0.5 Python/2.4.3
Date: Tue, 19 Dec 2006 23:00:12 GMT
content-type: text/html; charset=UTF-8
Connection: close

Oh look! The headers I wanted. I hate you, LWP.

Sigh.

Update: curl seems to be the best solution, here.

% curl -sS -D - -o /dev/null http://www.google.com
HTTP/1.1 200 OK
Date: Fri, 16 Jul 2010 21:34:01 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=d8703167b9ea01a8:TM=1279316041:LM=1279316041:S=j5mVVrOv81lHeqpa; expires=Sun, 15-Jul-2012 21:34:01 GMT; path=/; domain=.google.com
Set-Cookie: NID=36=plIkUp9cvYQ5Fc6VqIQxBDjjqS_VP1F1EohF0FBv2wW-phBuNTxqm0vduncInbp-MgLd2aikmgRu8JZlZmNnv6LcymNEfsxh33xyUaG8avqwaI8MkAYT6XiiWbxLXnAW; expires=Sat, 15-Jan-2011 21:34:01 GMT; path=/; domain=.google.com; HttpOnly
Server: gws
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked

With pimp, I need to serve two kinds of requests. The first, is standard web requests. The second is data streaming requests.

Pylons serves standard requests like a champ. However, I was finding it hard to convince controller methods to send data gradually in a stream. Some brief searching told me that Pylons 0.9.3 supported iterators for data. That is, you could have a controller yield data (using generators) and each yielded piece would be logically sent as it is generated.

However, I was never able to get it to work. I almost gave up when I did yet another search for answers. This search lead me to this thread which explains the solution. In the project config file (development.ini, or whatever), you need to disable debug mode before generators will work properly.

If debug mode is enabled, pylons will wait for data to finish sending (through generators or otherwise) before the data is actually given to the requesting client.