Wednesday, April 22, 2015

Installing the collectd for a client node

hi all,

I would like to list down the steps on how I do the collectd installation on the client node.

1. Download the relevant RPM. I am choosing the latest collectd version, which is 5.4.1. The default package that came from EPEL repo is on version 4, which is a bit outdated. The reason behind it collectd on version 4 is not comes with the graphite plugin. So, to take the shortcut, without compiling the binary from scratch of tarball, you can ought to be like me, and download/install the ready made RPM available from the internet.

wget http://anfadmin.ucsd.edu/linux/CentOS/6/x86_64/collectd-5.4.1-1.el6.x86_64.rpm
wget http://anfadmin.ucsd.edu/linux/CentOS/6/x86_64/libcollectdclient-5.4.1-1.el6.x86_64.rpm

rpm -ivh libcollectdclient-5.4.1-1.el6.x86_64.rpm collectd-5.4.1-1.el6.x86_64.rpm

2. Now, you need to edit a couples of lines at the /etc/collectd.conf. It is simple.


vi /etc/collectd.conf

Hostname     "centos66"

    Host "graphite_hostname"
    Port "2003"
    Protocol "tcp"
    LogSendErrors true
    Prefix "collectd."
    #Postfix "collectd"
    StoreRates true
    AlwaysAppendDS false
    EscapeCharacter "."
 


3. Make sure you change your setting accordingly to suit your need. After all, just start the service like normal.


/etc/init.d/collectd start

Voila, you are done!

Please get back to your graphite server, and take a look at the graphite dropdown tree, it should populate your new collectd host. If you want to repeat the same things, on let says, 200 servers? you can always use my pyssh.py script to login to the servers concurrently and make changes and apply the change as you like.

Next I guess, you would like to explore on the presentation layer, like Graphana. With graphana installed, you can show/manipulate the metrics in a narrow scope to fit the need of monitoring needs to you and your team. Or an more intuitive tool like logstash to give you more analytical view on your system log. Or you need a alerting system like graphite-beacon to generate alerts in a different formats.


Congrats!


Tuesday, April 21, 2015

Multiple instance for carbon to leverage the traffic

hi,

if you are keen to leverage your carbon traffics, or may be you wish to categorise the metrics from the different sources. Then, you can follow these steps.

1. The configuration file is /opt/graphite/conf/carbon.conf

2. Please look closely at line until you see this. please uncomment them when you need to spawn another carbon instance.

# To configure special settings for the carbon-cache instance 'b', uncomment this:
[cache:b]
LINE_RECEIVER_PORT = 2103
PICKLE_RECEIVER_PORT = 2104
CACHE_QUERY_PORT = 7102

3. Make sure disable the UDP_LISTENER.
# Set this to True to enable the UDP listener. By default this is off
# because it is very common to run multiple carbon daemons and managing
# another (rarely used) port for every carbon instance is not fun.
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003

4. Once you are done. you need to top up the daemon file like this.

start() {
        echo -n $"Starting $prog: "

        PYTHONPATH=/usr/local/lib/python2.6/dist-packages/ /opt/graphite/bin/carbon-cache.py --instance=a start
        PYTHONPATH=/usr/local/lib/python2.6/dist-packages/ /opt/graphite/bin/carbon-cache.py --instance=b start
        RETVAL=$?
.
.
.
stop() {
        echo -n $"Stopping $prog: "

        PYTHONPATH=/usr/local/lib/python2.6/dist-packages/ /opt/graphite/bin/carbon-cache.py --instance=a stop > /dev/null 2>&1
        PYTHONPATH=/usr/local/lib/python2.6/dist-packages/ /opt/graphite/bin/carbon-cache.py --instance=b stop > /dev/null 2>&1
.
.
.
status()

Setting up a graphite service in CENTOS 6.6

hi all,

I just want list down the steps of graphite installation on CENTOS6.6.

First you have to install the pre-requisite of the packages.

1. yum -y install epel-release python-pip python-devel gcc libev libev-devel pycairo rrdtool-python mod_wsgi git httpd

2. pip install django==1.6.8 carbon whisper graphite-web django-tagging pytz
3. pip install twisted --upgrade
4. chmod a+w /opt/graphite/storage

Start copying the conf files and make changes.

5. cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf

6. cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi

I will commented the lines on the search index portion. for example like this.
# READ THIS
# Initializing the search index can be very expensive, please include
# the WSGIImportScript directive pointing to this script in your vhost
# config to ensure the index is preloaded before any requests are handed
# to the process.
#from graphite.logger import log
#log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
#import graphite.metrics.search

7. cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf

8. cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py 

Once you are done with the copying local_settings.py, please update that TIME_ZONE and SECRET_KEY. please use the tzselect if you don't know how to set your time zone. Secret key can be any strings that you can think of.

Now you are start working on your apache setting.

9. cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite-vhost.conf

Comment out the extra mod_wsgi mod from another conf file.

10. cat /etc/httpd/conf.d/wsgi.conf
#LoadModule wsgi_module modules/mod_wsgi.so

On the default /etc/httpd/conf/httpd.conf, you have to update the ServerName. After all you can start your web server service.

11. ServerName  graph:80

12. chkconfig httpd on && /etc/init.d/httpd start

Now you can start initialise your django framework. Please complete all the questions asked during the initial setup. Put in your admin credential and email.

13. python manage.py syncdb


Now you can start working on the carbon daemon and start it. Get the free daemon file from the github. 

14. wget -q https://github.com/dcarley/graphite-rpms/blob/master/carbon-0.9.8/carbon.init -O /etc/init.d/carbon && /etc/init.d/carbon start

Next you will need to verify your carbon port is listening and your web port is running.

15. netstat -atun | grep 2003
tcp        0      0 0.0.0.0:2003                0.0.0.0:*                   LISTEN

16. netstat -atun | grep 80
tcp        0      0 0.0.0.0:80                0.0.0.0:*                   LISTEN

Wait for a while, after you have started the services. You need to look into your graphite storage for any new whisper file been created. Keep prompt the directory for any new created whisper files.

17. [root@graph storage]# find /opt/graphite/storage/ -name *.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/updateOperations.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/errors.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/cpuUsage.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/avgUpdateTime.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/cache/bulk_queries.wsp
/opt/graphite/storage/whisper/carbon/agents/graph-a/cache/queries.wsp

Once all done, your graphite should be up and running. you can open your browser and go to your graphite web service to verify it is running. At this moment, you should not seeing any data injection from any servers yet. you need to continue working on collectd installation on the client nodes.