@jordansissel

DreamHost

who am I?

  • Czar of Logging @ DreamHost
  • full-time work on logstash
  • sysadmin for 8 years

things I like

terminology

(for the purpose of this discussion)

what is a log?

(photo by Susie Blackmon)

Sep 16 05:03:21 carrera kernel: md: md2: data-check done.
(  timestamp  ) (      something that happened          )
          

timestamp + data = log

what is a metric?

what is a log?

proposal:

metrics are logs

108.166.15.188 - - [13/Sep/2012:02:34:22 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.8 (ruby-1.9.3-p0; ohai-0.6.10; x86_64-linux; +http://opscode.com)"
50.56.197.244 - - [13/Sep/2012:02:34:37 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10 (ruby-1.9.3-p194; ohai-0.6.4; amd64-freebsd8; +http://opscode.com)"
37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1.8.7-p334; ohai-0.6.10; i686-linux; +http://opscode.com)"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/feed.png HTTP/1.1" 200 689 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
108.166.15.188 - - [13/Sep/2012:02:34:22 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.8 (ruby-1.9.3-p0; ohai-0.6.10; x86_64-linux; +http://opscode.com)"
50.56.197.244 - - [13/Sep/2012:02:34:37 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10 (ruby-1.9.3-p194; ohai-0.6.4; amd64-freebsd8; +http://opscode.com)"
37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"
184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1.8.7-p334; ohai-0.6.10; i686-linux; +http://opscode.com)"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/feed.png HTTP/1.1" 200 689 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"

  timestamp                 |response code
----------------------------+-------------
[13/Sep/2012:02:34:22 -0400] 200
[13/Sep/2012:02:34:37 -0400] 200
[13/Sep/2012:02:32:49 -0400] 404 
[13/Sep/2012:02:37:24 -0400] 200 
[13/Sep/2012:02:38:12 -0400] 302 
[13/Sep/2012:02:38:23 -0400] 200
[13/Sep/2012:02:38:25 -0400] 200 
[13/Sep/2012:02:38:26 -0400] 200 
[13/Sep/2012:02:38:26 -0400] 200 
[13/Sep/2012:02:38:26 -0400] 200 
[13/Sep/2012:02:38:31 -0400] 200 
[13/Sep/2012:02:38:28 -0400] 200 
[13/Sep/2012:02:38:46 -0400] 200
[13/Sep/2012:02:38:46 -0400] 200
[13/Sep/2012:02:38:46 -0400] 200
[13/Sep/2012:02:38:47 -0400] 200

  timestamp                 |errors
----------------------------+-------------
[13/Sep/2012:02:34:22 -0400] ██████
[13/Sep/2012:02:34:37 -0400] █████
[13/Sep/2012:02:32:49 -0400] ███████
[13/Sep/2012:02:37:24 -0400] ████████
[13/Sep/2012:02:38:12 -0400] ███████████
[13/Sep/2012:02:38:23 -0400] ████████
[13/Sep/2012:02:38:25 -0400] ███████
[13/Sep/2012:02:38:26 -0400] ███████
[13/Sep/2012:02:38:26 -0400] ███████████
[13/Sep/2012:02:38:26 -0400] ████████
[13/Sep/2012:02:38:31 -0400] ██████████
[13/Sep/2012:02:38:28 -0400] █████████
[13/Sep/2012:02:38:46 -0400] ███████
[13/Sep/2012:02:38:46 -0400] ████████
[13/Sep/2012:02:38:46 -0400] ████████
[13/Sep/2012:02:38:47 -0400] ██████

how do we do it?

wrong way

make it easy.

179.44.34.142 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10"

response code: 200

bytes sent: 40923996

input: file

filter: grok

output: statsd

input {
  file {
    type => "web"
    path => "/var/log/httpd/access.log"
  }
}

filter {
  grok {
    type => "web"
    pattern => "%{COMBINEDAPACHELOG}"  
  }
}

output { 
 statsd { 
    type => "web"
    increment => "apache.response.%{response}"
    count => [ "apache.bytes", "%{bytes}" ]
  }
}
output {
   {
    metric => "apache.bytes"
    value => "%{bytes}"
  }
}
output {
   {
    metrics => [ 
      "apache.bytes", "%{bytes}" 
    ]
  }
}
output {
   {
    counter => [ 
      "apache.bytes", "%{bytes}" 
    ]
  }
}
output {
  opentsdb {
    metrics => [
      "apache.bytes", "%{bytes}"
    ]
  }
}
input {
  # 27 input plugins
  # events from any source
}

filter {
  # 24 filter plugins
  # parse, modify, annotate events
}

output {
  # 45 output plugins
  # ship to any destination
}

what time is it?


1304060505 29/Apr/2011:07:05:26 +0000
Fri, 21 Nov 1997 09:55:06 -0600 Oct 11 20:21:47
020805 13:51:24 110429.071055,118
@4000000037c219bf2ef02e94

logstash date filter fixes this bullshit

date fixing

  • standards: ISO8601 / RFC3339 / xs:dateTime
  • Example: 2012-09-19T22:14:30.425-0800
  • unified timestamp format for searching/sorting.
filter {
  date {
    # Turn 020805 13:51:24
    # Into 2002-08-05T13:51:24.000Z
    mysqltimestamp => "YYMMdd HH:mm:ss"  }
}

this is one event

Exception in thread "main" Fooz$FancyPantsException
        at Fooz.bar(Fooz.java:14)
        at Fooz.foo(Fooz.java:10)
        at Fooz.main(Fooz.java:6)

Solution:

filter {
  multiline {
    # If the message starts with whitespace,
    pattern => "^\s"
    # it belongs to the previous line.
    what => previous
  }
}

negative request duration?

less than zero seconds?!

request duration < 0 !?!

  • apache uses gettimeofday()
  • ntp behaves badly with bad hardware clocks
  • time jumps backwards == negative request time

logstash

solving serious problems

bieber vs gangnam

input {
  twitter { # live twitter stream
    type => "twitter"
    user => "secret"
    password => "secret"
    keywords => [ "bieber", "gangnam" ]
  }
}
output {
  elasticsearch { }
}

query elasticsearch

render in graphite

logstashHits("some query")

movingAverage()

divideSeries()

logstash is a unix pipe on steroids
- John Vincent (@lusis)

27 inputs | 23 filters | 45 outputs

inputs

where events come from

filters

process and modify events

outputs

send events somewhere else

scaling out

scaling logstash (transport)

  • inputs with matching outputs
  • redis, zeromq, stomp, amqp, irc, xmpp

scaling logstash (storage)

  • elasticsearch scales horizontally

live demo?

project focuses

function, design, community

feature:

transport and process events
to and from anywhere.

(in any format)

feature:

provide search and analytics

design:

logstash should fit your infrastructure

design:

logstash is extendable

community:

if a newbie has a bad time, it's a bug

community:

contributions: more than code

get your 'stash on


Questions? Now or later (twitter: @jordansissel)