さくらvpsでfluentd + mongodbを試す

apacheのアクセスログをfluentdを経由してmongodbへ
よくあるパターンを1台のサーバで試してみた

mongodbのインストール

$ vi /etc/yum.repos.d/mongodb.repo

[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

$ yum install -y mongodb-org
$ cp /etc/mongod.conf /etc/mongod.conf.org
$ vi /etc/mongod.conf
$ diff /etc/mongod.conf /etc/mongod.conf.org

29c29
< auth=true
---
> #auth=true
77d76

$ chkconfig mongod on
$ service mongod start

mongodb ユーザ作成

$ mongo

MongoDB shell version: 2.6.4
connecting to: test
> db.addUser('letitride', 'password');
WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead
Successfully added user: { "user" : "letitride", "roles" : [ "dbOwner" ] }
> exit;

fluentd インストール

$ curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh
$ chkconfig td-agent on
$ vi /etc/td-agent/td-agent.conf
$ diff /etc/td-agent/td-agent.conf /etc/td-agent/td-agent.conf.org

103,132d102
< 
<
< <source>
<   type tail
<   format apache
<   path /var/log/httpd/access_log
<   pos_file /var/log/td-agent/apache.front-access.pos
<   tag mongo.apache.access
< </source>
< 
< <match mongo.**>
<   type mongo
<   host localhost
<   port 27017
<   database test 
<  
<   # サイズ上限
<   capped
<   capped_size 1024m
<  
<   # DB接続情報
<   user letitride
<   password password
<  
<   tag_mapped
<   remove_tag_prefix mongo.
<  
<   collection misc
< </match>

$ /etc/init.d/td-agent restart

ログ格納確認

$ mongo test -u letitride -p

MongoDB shell version: 2.6.4
Enter password: 
connecting to: test
> show collections
apache.access
system.indexes
> db.apache.access.count();
7
> db.apache.access.find();
{ "_id" : ObjectId("54227bc9a692952ea1000001"), "host" : "113.43.175.2", "user" : "-", "method" : "POST", "path" : "/login", "code" : "200", "size" : "3290", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000002"), "host" : "113.43.175.2", "user" : "-", "method" : "GET", "path" : "/stylesheets/jquery/jquery-ui-1.9.2.css?1393759707", "code" : "304", "size" : "-", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000003"), "host" : "113.43.175.2", "user" : "-", "method" : "GET", "path" : "/javascripts/jquery-1.8.3-ui-1.9.2-ujs-2.0.3.js?1393759706", "code" : "304", "size" : "-", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000004"), "host" : "113.43.175.2", "user" : "-", "method" : "GET", "path" : "/stylesheets/application.css?1393759709", "code" : "304", "size" : "-", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000005"), "host" : "113.43.175.2", "user" : "-", "method" : "GET", "path" : "/javascripts/application.js?1393759706", "code" : "304", "size" : "-", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000006"), "host" : "113.43.175.2", "user" : "-", "method" : "GET", "path" : "/images/exclamation.png", "code" : "304", "size" : "-", "time" : ISODate("2014-09-24T08:06:45Z") }
{ "_id" : ObjectId("54227bc9a692952ea1000007"), "host" : "113.43.175.2", "user" : "-", "method" : "POST", "path" : "/login", "code" : "200", "size" : "3290", "time" : ISODate("2014-09-24T08:07:00Z") }
> exit