photo
Jordan Sissel
geek

Wed, 31 Jan 2007

Session affinity and load distribution with Tomcat and Apache

You can scale tomcat webapps somewhat well using session affinity and load distribution. But how? Apache to the rescue.

For each tomcat server, modify the server.xml and change the value for 'jvmRoute' to the ip address of the tomcat server. Example:

  <Engine name="Standalone" defaultHost="localhost" jvmRoute="192.168.0.10">
This affects the last token in your jsessionid cookie. Visiting my tomcat, my cookie gets set to the following:
C40ABF646B07162A621856F459977E9B.192.168.0.10

Use apache's mod_rewrite to use apache as a frontend to your tomcat servers. That is, use apache as a reverse proxy. In your httpd.conf:

RewriteMap SERVERS rnd:/etc/httpd/conf/frontends.conf
RewriteCond "%{HTTP_COOKIE}"          "(^|;\s*)JSESSIONID=\w*\.([0-9.]+)($|;)"
RewriteRule "^(.*)"                   "http://%2:8080%{REQUEST_URI}"  [P,L]
RewriteRule "^.*;jsessionid=\w*\.([0-9.]+)($|;)"  "http://$1:8080%{REQUEST_URI}"  [P,L]
RewriteRule "^(.*)"                    "http://${SERVERS:ALL}:8080%{REQUEST_URI}" [P,L]
This technique is quite similar to the one on tomcat.apache.org in the docs, but I think it's better. Why? Less files to modify when you add or remove tomcat servers means less complexity, less errors and less effort.

  1. RewriteCond "%{HTTP_COOKIE}" "(^|;\s*)JSESSIONID=\w*\.([0-9.]+)($|;)"
    If a jsessionid cookie is found, go to #2 and store match groups (backreferences) as %1, %2, etc.
  2. RewriteRule "^(.*)" "http://%2:8080%{REQUEST_URI}" [P,L]
    Session Affinity: Redirect everything using an internal proxy request to the 2nd group matched in the previous RewriteCond. Since we use the IP as the jvmRoute, that's what is matched, and your request is proxied to the server that gave you your cookie.
  3. RewriteRule "^.*;jsessionid=\w*\.([0-9.]+)($|;)" "http://$1:8080%{REQUEST_URI}" [P,L]
    Session affinity: Tomcat likes to add (who knows why?) ";jsessionid=blah" to the end of the url when it first sets you up the cookie. In case no cookie is found, this will proxy your request to the proper server just like the previous rule.
  4. RewriteRule "^(.*)" "http://${SERVERS:ALL}:8080%{REQUEST_URI}" [P,L]
    Load distribution: Catch-all for anything that didn't have a cookie or jsessionid thing in the url. "ALL" is just a key from the RewriteMap listed below. A random one is chosen and inserted.

Since the server ip is stored in the cookie, apache (using regular expressions) can pull it out and will internally proxy your request through to the proper tomcat server.

That works great for sessions that already exist, but what about for sessions that don't exist? That's what ${SERVERS:ALL} is for. You need something like this in your frontends.conf file:

ALL 192.168.0.10|192.168.0.11
This would be even better if you only used DNS for this. Then, you wouldn't need to update any config files when you added or removed tomcat servers.

If you had the fallback redirect of:

RewriteRule "^(.*)"       "http://${SERVERS:ALL}:8080%{REQUEST_URI}" [P,L]
RewriteRule "^(.*)"       "http://mytomcats.foo.com:8080%{REQUEST_URI}" [P,L]
Apache should redirect internally to "mytomcats.foo.com" which should result in a dns lookup of that hostname. If you have multiple records in that hostname, you get round-robin balancing across all tomcats for new sessions. When you add or remove tomcat servers, you don't have to update any config files.

No config files to change when you add new servers? That makes for healthy, dynamic scaling.

The best way to solve this would be to have tomcat share it's session data, but it uses multicast, and the network where tomcat lives doesn't have multicast routing enabled, so that doesn't seem like an option.

Comments: 4 (view comments)
Tags: , ,
Permalink: /geekery/session-balancing-across-tomcats-with-apache
posted at: 05:08


4 responses to 'Session affinity and load distribution with Tomcat and Apache'

Ryan posted at Wed Jan 31 14:28:43 2007...
But what happens if you take down a server or it goes offline? Does your request get routed to a new server and it gets a new session?

My experience with scaling is to separate your session info into different store (DB or some other scalable, mirrored setup) and use that.

This route seems to have a weakness that if a server goes down, all the sessions are lost. In a scalable app, you want a request to be server agnostic.

-Ryan

Jordan Sissel posted at Wed Jan 31 16:13:06 2007...
Agreed. The last paragraph mentions tomcat's ability to share sessions across workers. However, that requires multicast, and the network (amazon ec2) I'm using tomcat on doesn't support that according to the forumns.

Sessions should be separate from the frontends, but since I'm not coding up the tomcat webapp I don't get to make that decision. The apache solution is good when you haven't got control of the code.

Ryan posted at Wed Jan 31 22:42:20 2007...
Ah, multicast, that's cool. I didn't fully take in that last paragraph before I replied.

Sounds like I need to learn about Tomcat.

-Ryan

Binh Nguyen Thanh posted at Tue Jul 24 23:20:29 2007...
Thankyou very much!


Leave a reply

You need javascript enabled to use this form. Anti-spam efforts ongoing. Also, if the comment doesn't show up, it's because the form expired. Go back and copy your comment, reload the form, and resubmit. Apologies if this is a hassle, I'm just playing with antispam methods right now. If this insists on not working, please email me about it.

Name (required)
E-mail (optional, if you want me to be able to email you back)
URL (also optional)
Comment:


Search this site

Navigation

Metadata

Home About Resume My Code (SVN)

Articles

ARP Security Dynamic DNS with DHCP OpenLDAP+Kerberos+SASL PPP over SSH SSH Security: /bin/false Week of Unix Tools Work Efficiency

Projects

fex firefox tabsearch firefox urledit grok keynav liboverride newpsm (FreeBSD) nis2ldap pam_captcha poor man's backup Solaris audio utility xboxproxy xdotool xmlpresenter xpathtool misc scripts

Presentations

Yahoo! Hack Day '06 Unix Essentials Vi/Vim Essentials

Tag Cloud

Calendar

< January 2007 >
SuMoTuWeThFrSa
  1 2 3 4 5 6
7 8 910111213
14151617181920
21222324252627
28293031   

Friends

BarCamp Kent Brewster Tantek Çelik John Resig Wesley Shields Tyler Shields

Technorati