Search this site


Metadata

Articles

Projects

Presentations

DNS Redirection and how it will break things

There's some hot debate around the implications and rightness of service providers to do things like traffic filtering, session hijacking, etc.

I'm not here to talk about that. The data below aims to focus on the technical failures induced by dns hijacking, or dns redirection. I won't bore you with the moral, political, or philosophical discussion around this topic.

Here's a summary if you don't want to read the details:

  • DNSBLs probably don't work anymore for Comcast users
  • Owned domains (semicomplete.com, google.com, etc) are also subject to hijacking. Domain owners cannot opt-out.
  • Down dns servers may result in full-domain hijacking by Comcast (due to dns search suffix and retry behavior)
  • Privacy/security/cookie leak due to domain hijacking
  • There is an IETF draft to formalize/standarize this destructive behavior that ignores consequences and impact, and neglects important points.
  • Sometimes Domain Helper malfunctions and steals valid hostnames, like www.google.com: screenshot here ; or a video
  • Web browsers aren't the only things using DNS.

All of the samples below are real data I observed while digging into this issue. They are not faked.

Comcast recently finished rolling out their new Domain Helper software. This tool intercepts responses from other DNS servers and replaces them with forged responses that point to comcast's search portal. It currently modifies responses that indicate 'no such hostname'; in DNS this response is called NXDOMAIN.

For example, if you try to visit 'http://www.someinvalidhost12345.com', it doesn't exist, and the DNS 'NXDOMAIN' response saying so is dropped and replaced with an A record pointing to Comcast's search site:

% host http://www.someinvalidhost12345.com
http://www.someinvalidhost12345.com has address 208.68.139.38
The domain helper documentation explains how it works, but incorrectly describes behavior and doesn't describe the consequences of the new behavior. From the docs, it sounds something "must" (per above) include "www". This is false, and is likely just out-of-date documentation, as the documentation goes on to explain that they "will eventually phase in the following pattern matches to enhance this service in the future." This probably explains why valid invalid.valid hostnames are bounced to comcast, and why invalid TLDs are bounced to comcast:
% host -t A abcdefghijklmnopww12345678.semicomplete.com
abcdefghijklmnopww12345678.semicomplete.com has address 208.68.139.38

% host -t A www.dns.is.broken
www.dns.is.broken has address 208.68.139.38
Simply having 'ww' is sufficient, even for valid domains like mine or totally invalid TLDs like 'broken'. Every NXDOMAIN is subject to hijacking. The IP address above is still comcast's search redirector.

So, what is the hard technical impact of this change?

DNSBLs may be broken

DNSBL is 'dns blacklist', check wikipedia for detail. Wikipedia says this about how DNSBLs are used:
Look up this name in the DNS as a domain name ("A" record). This will return either an address, indicating that the client is listed; or an "NXDOMAIN" ("No such domain") code, indicating that the client is not.
% host -t A 111.0.168.192.bl.spamcannibal.org
111.0.168.192.bl.spamcannibal.org.home has address 208.68.139.38
% host -t A 111.111.11.11.zen.spamhaus.org
111.111.11.11.zen.spamhaus.org.home has address 208.68.139.38

According to DNSBL de facto behavior, I should probably get NXDOMAIN given it's unlikely these IPs are in DNSBLs. Alternately, many DNSBLs reply with a special '127.0.0.xxx' address to indicate what the status of the address is if it is in the list. The responses above are neither, but are Comcast's search domain.

A Comcast employee on twitter pointed out that DNSBLs are only used by mail servers. Fine, we don't run mail servers on Comcast residental, but there are mail clients that support DNSBL for local spam filtering (thunderbird may, evolution appears to). The point is, DNSBL is not purely for 'mail servers', meaning this feature breaks DNSBL for clients.

I don't have data about how many people use DNSBL on mail clients. Point is, it's an option not limited to mail servers.

Down DNS servers are fully hijacked

When a DNS server is down, your DNS request generally times out and you get a SERVFAIL response. What happens when your local DNS client gets a SERVFAIL, though?

DNS resolvers support 'search suffixes'. In linux, you see this in /etc/resolv.conf as 'search some list of domains'. At my house, I have a 'home' dns zone managed automatically by my ddwrt router. This means I have 'search home' in my resolver.

If I try doing a dns query for 'some-invalid-host.something.com' and get NXDOMAIN or SERVFAIL, my resolver will try adding a suffix, if it's configured to do so. Here's an example of a dns server not responding and yielding SERVFAIL and the resolver's attempt to try with a suffix 'home' resulting in comcast NXDOMAIN hijacking me:

% host -t A wwwdead.evilcouncil.org 68.87.76.182
Using domain server:
Name: 68.87.76.182
Address: 68.87.76.182#53
Aliases:

wwwdead.evilcouncil.org.home has address 208.68.139.38

Sniffing my local network traffic with tshark during this lookup:
6.755838 192.168.0.97 -> 68.87.76.182 DNS Standard query A wwwdead.evilcouncil.org
6.770844 68.87.76.182 -> 192.168.0.97 DNS Standard query response, Server failure
6.771264 192.168.0.97 -> 68.87.76.182 DNS Standard query A wwwdead.evilcouncil.org.home
6.799348 68.87.76.182 -> 192.168.0.97 DNS Standard query response A 208.68.139.38
Interesting. Here's another data point:
% ping hijackmyww.evilcouncil.org
PING hijackmyww.evilcouncil.org.home (208.68.139.38) 56(84) bytes of data.
^C
Notice comcast's search portal IP repeated again and again...

If we use comcast's default search suffix for my area, hsd1.ca.comcast.net, what happens?; 'www.example.com' may SERVFAIL, then we retry as www.example.com.hsd1.ca.comcast.net, which comes back as comcast's search2.comcast.net, again, due to their NXDOMAIN intercept.

Now, any SERVFAIL may automatically direct you to through their 'dns helper' search redirect.

Security issues

The last issue is something I noticed last night while disconnecting my VPN to work. I had our monitoring site up in my browser - this page autorefreshes itself every few minutes. After disconnecting, it refreshed, and dumped me at search2.comcast.net.

Since the redirect happens in DNS, the browser will believe that you are still talking to the same server (by name) and will share cookies and http authentication freely. What happens if you visit 'www.mail.google.com' in Firefox? Let's look at the network:

snack(~) % sudo ngrep . 'port 80 and host 208.68.139.38'
[sudo] password for jls:
interface: eth0 (192.168.0.0/255.255.255.0)
filter: (ip or ip6) and ( port 80 and host 208.68.139.38 )
match: .
####
T 192.168.0.97:44422 -> 208.68.139.38:80 [AP]
  GET / HTTP/1.1..Host: www.mail.google.com..User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.3) Gecko/20090824 Firefo
  x/3.5.3..Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept-Language: en-us,en;q=0.5..Accept-Encoding: gzip,
  deflate..Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7..Keep-Alive: 300..Connection: keep-alive..Cookie: PREF=ID=0cc23ef3118faec6:U=fffb...
The above shows Firefox sending Comcast my google mail login cookie. Had I been visiting a page that uses basic HTTP authentication, my username and password would've been sent, too.

Sure, cookies have a 'secure' attribute which tells the browser to only send them over SSL connections, but does everyone use them? Nope. Bank of America, for example, doesn't set this flag, which is a security issue itself, but is leaked to Comcast on typoed URLs or during 'domain helper' malfunctions.

Firefox has no idea that 'www.mail.google.com' isn't Google, or www1234.bankofamerica.com isn't Bank of America, and a normal user won't have any idea they just leaked their credentials, private session info, and other cookie data.

Domain Helper Malfunction

Ofcourse, the worst part of this whole system is when the hijacking itself misbehaves, and you get 'www.google.com' hijacked, when it's a real hostname.

Defenses and Actions

How can we prevent some or all of this? Contact your ISP, help them understand the impact of this behavior. Pushing at least to allow domain owners to opt-out of the domain redirection might be a start or to require that providers not hijack owned domains. Not hijacking invalid TLDs would be good too, given the common usage of 'corp' and others as an intranet TLD.

If you are a comcast user and want to opt-out of Comcast's Domain Helper, you can visit dns-opt-out.comcast.net (requires login, only works when viewed from comcast). Alternately, you can point your local dns caches at comcast's non-domain-helper servers

Comcast DNS breaks the internet

I only noticed this now. Perhaps it is old.
% dig +noquestion +nostats +nocmd +nocomments www."I wonder if this query will return".com 
www.I\032wonder\032if\032this\032query\032will\032return.com. 0 IN A 208.68.139.38

% dig +short www."$(dd if=/dev/urandom bs=1 count=30 2> /dev/null)".com 
208.68.139.38
Comcast only appears to be stealing dns queries for www.*.<tld>, which is slightly less annoying than what OpenDNS does, but OpenDNS is opt-in (you have to point your dns at opendns, where Comcast tells your cable modem what it should advertise for dns over dhcp).

Sigh.

DNS providers suck.

Happy Halloween, folks. It's been 20 days since my last post. I've been incredibly busy with work and haven't had a chance to write. As a gift, I give you a rant.

I've been through no less than 3 DNS service providers in the past week, and all of them suck. They suck hard.

The first one I looked at was no-ip. No-IP claims they support 'dynamic dns' - they don't. The first thing you must realize about almost all dns providers is that while they claim they support "dynamic dns" and/or "round robin," what they really mean is their support of 'dynamic dns' is based solely around one single use case. One.

What is that use case? The following picture comes from dynu.com:

What is this? This use case of one computer updating it's own hostname with whatever IP it happens to have at that moment. Businesses can't possibly find this useful. It doesn't scale. If you have more than one server you want to put on a single hostname, this use case fails you miserably.

I've looked at no-ip, dyndns, dnspark, and several others. Trash.

Keep in mind, this rant is becuase both free AND pay-for dns providers suck. Both kinds. Free services actually have an excuse - you get what you pay for.

As a precursor, let me explain what I need from a dns provider:

  1. The ability to add and remove dns entries of any record type, at any time.
  2. The ability to add multiple entries for the same record
Many claim these features. Those I tried fail miserably.

If you are in the market for a real dns provider, as I am, you'll find many dns providers claiming what I listed above. "Sure! We support round robin!" they advertise, "We support dynamic dns!"

What they don't tell you in the same paragraph is that you have to use their own HTTP-based means of pushing dns changes. They absolutely don't tell you that their pathetic attempt at providing this "dynamic" service via a cgi-like interface is absolutely crippled.

Several providers allowed you to mutate records dynamically. However, none of them I tried let me add multiple entries for a single record using the dynamic interface.

An important realization is that my definition of dynamic is not the same as these dns providers' notion of dynamic. This so-called dynamic dns ability hinges on customers who want to be able to host crap out of their dynamic-ip-giving ISP. As such, most of the interface is just "Hey DNS provider! Please update www.foo.com with whatever IP this packet is coming from! Thanks!" This is intolerable!

What is my definition of "dynamic dns," exactly? Let's call it RFC 2136. Heck, I don't care if it's not RFC 2136, just that I'm able to do most things that update specification provides.

To quote ZoneEdit customer support regarding my issues with their service and in particular how to properly use their crippled dynamic update interface:

"You can atleast update hourly .

Updating too often with the same IP address gets your account locked up."

WHAT?! Once hourly? Shit. DNS is hard. Let's go shopping instead.

Doing this right is not hard. For example, I recently posted an article on how to setup dynamic dns and make your dhcp server talk sweetly to dns. I use this same configuration in my apartment. MY APARTMENT. My apartment is considerably smaller than, say, a multidatacenter dns provider. Why doesn't anyone at any of these dns providers have a freaking clue about running a dns server? Let me put it plainly:

I will give you money and you will give me a dnssec key and a server on which to use it. That shall be the extent of our relationship
That's all I want. The worst part is that it doesn't matter who you go with. There are plenty of free dns providers who provide you the same crappy service as give-us-your-money providers.

Really. Come on kids.

Look at it this way - To enable dynamic dns updates, you don't need to write any code. A few tiny named.conf changes. To provide a pathetic http interface you label as "dynamic dns" requires lots of lines of code, lots of testing, and $$$ invested in this kind of product.

To further show how stupid this is. Microsoft supports this properly. Microsoft. You know, that company everyone hates-on for proprietary protocols and ignorance of standards? Microsoft DNS will send updates using BIND's update protocol. How do I know this? I've had a primary dns server running BIND and Microsoft DNS running as a secondary. I told Active Directory that it's primary dns was the BIND server. Guess what happened? Active Directory happily submitted updates to my BIND server. Correctly.

You might be thinking to yourself, "Why don't you just host dns yourself?" Because I dont' have any servers on a static IP address. And no, this isn't running out of my apartment.

Am I the only one who can't find a dns provider that doesn't suck?

Dynamic DNS + DHCP Article

I wrote a new article (due to overwhelming demand of 1 person asking) about how to get dynamic dns and dhcp working.

articles/dynamic-dns-with-dhcp