Puppet codes - array and looping

Puppet codes - array and looping

It does bothering me at first when I start to code in puppet. Where is my array definition and how am I going to loop. Here is the caveat right after searching for help in the puppet user group.

This is the example.

I have declared some variables at /etc/puppet/manifests/site.pp

$all_nodes = ["node1.hiu.com", "node2.hiu.com", "node3.hiu.com", "node4.hiu.com"]

Here is the code for the looping 

define install_package_zookeeper {

        if $fqdn == "$name" {

                package { mapr-zookeeper:

                        ensure => present,

                }

        }

}

class mapr::install_zookeeper {

        install_package_zookeeper { $all_nodes:;}

}

Basiscally, here is the explanation, the define function can take the argument of array and traverse the array items as a single instance. e.g.

install_package_zookeeper { ["node1.hiu.com", "node2.hiu.com", "node3.hiu.com"] }

as like this

install_package_zookeeper {

                  "node1.hiu.com":

                  .

                  .

                  "node2.hiu.com":

                  .

                  .

                  "node3.hiu.com":

}

So, taking the puppet default behaviour of grant is actually mean to do a looping for an array. lol

Writing your own facter code - puppet

Writing your own facter code - puppet

Puppet agent provides a set of ruby codes and that stay at /usr/lib/ruby/site_ruby/1.8/facter/*.rb. These ruby codes are the one that provides values for facter's output.

[root@test ~]# facter -p

architecture => x86_64

augeasversion => 0.9.0

boardmanufacturer => Intel Corporation

boardproductname => 440BX Desktop Reference Platform

boardserialnumber => None

domain => houston.hp.com

facterversion => 1.6.6

fqdn =>test.hiu.com

hardwareisa => x86_64

hardwaremodel => x86_64

hostname => d9t0578g

id => root

If you stand on the $rubysitedir, /usr/lib/ruby/site_ruby/1.8, you will notice that you can addon more ruby codes to define your own facter parameter and values.

[root@test facter]# pwd

/usr/lib/ruby/site_ruby/1.8/facter

[root@d9t0578g facter]# ls

application.rb       kernel.rb                  puppetversion.rb

architecture.rb      kernelrelease.rb           qpk.rb

arp.rb               kernelversion.rb           rawpartition.rb

augeasversion.rb     lsbmajdistrelease.rb       rubysitedir.rb

Cfkey.rb             lsb.rb                     rubyversion.rb

domain.rb            macaddress.rb              selinux.rb

ec2.rb               macosx.rb                  ssh.rb

facterversion.rb     manufacturer.rb            timezone.rb

fqdn.rb              memory.rb                  uniqueid.rb

hardwareisa.rb       netmask.rb                 uptime_days.rb

hardwaremodel.rb     network.rb                 uptime_hours.rb

hostname.rb          operatingsystem.rb         uptime.rb

id.rb                operatingsystemrelease.rb  uptime_seconds.rb

interfaces.rb        osfamily.rb                util

ipaddress6.rb        path.rb                    virtual.rb

ipaddress.rb         physicalprocessorcount.rb  vlans.rb

iphostnumber.rb      processor.rb               xendomains.rb

kernelmajversion.rb  ps.rb

This is how i wrote my simple facter codes, called raw_partition.

[root@test facter]# cat rawpartition.rb

# Fact: checking the QPK version

Facter.add(:raw_partition) do

        setcode do

                confine :kernel => :linux

                Facter::Util::Resolution.exec("cat /proc/partitions | awk '/sd/ {print $4}' | sed -e 's|sd|/dev/sd|g' | grep -v sda")

        end

end

Basically, you can alter the line that i highlighted and put some meaningful shell script/strings to yield a different output.

The output of this will be like below.

[root@test facter]# facter -p

architecture => x86_64

augeasversion => 0.9.0

boardmanufacturer => Intel Corporation

boardproductname => 440BX Desktop Reference Platform

boardserialnumber => None

.

.

.

raw_partition => /dev/sdd

/dev/sdc

/dev/sdb

Harvesting facter values - puppet_watchdog

Harvesting facter values - puppet_watchdog

This script is meant for harvesting facter -p values from the puppet agents. It lives as in a process and pulls data from $PUPPET_YAML_HOME, then, it bump the data onto a sqlite db. This script should be setup at the puppetmaster node only.

It is good especially when you need to do some data analysis on your agents.

At the same times, you can also write your own facter codes and retrieve customized information of your agent nodes.

#!/bin/bash
# Hiu, Yen Onn
# This is the watchdog script that will harvest
# all the data of yaml into a db

PUPPET_YAML_HOME="/var/lib/puppet/yaml/node"
PUPPET_WATCHDOG_LOG="/var/log/puppet_watchdogd.log"
HADOOP_DB="/etc/puppet/web/test.db"
LATEST_PUPPET_NODE=""


[[ ! -e $PUPPET_WATCHDOG_LOG ]] && touch $PUPPET_WATCHDOG_LOG

function control_c()
{
        echo -en "\n**** process `basename $0` is terminated. ****\n"
        exit $?
}

function log()
{

        while read data
        do
                echo "[$(date +"%D %T")] $data" >> $PUPPET_WATCHDOG_LOG
        done
}

function harvest_data()
{
        sleep 2
        FILE="$PUPPET_YAML_HOME/$LATEST_PUPPET_NODE"
        SQLITE_BIN="/usr/bin/sqlite3"

        UPTIME=`grep "uptime_days" $FILE | awk '{print $2}'|sed -e 's/"//g'`
        PROCESSERCOUNT=`grep "physicalprocessorcount" $FILE | awk '{print $2}'|sed -e 's/"//g'`
        CLIENTCERT=`grep clientcert $FILE | awk '{print $2}'`
        TIMESTAMP=`grep expiration $FILE | awk '{print $2 " " $3}'`

        if [[ -e $HADOOP_DB ]]
        then
                SQL_CMD=""

                # Query if it is a new record
                SQL_CMD_NEW="SELECT hostname FROM PUPPET_FACTER WHERE hostname='$CLIENTCERT';"
                newrecord=`$SQLITE_BIN $HADOOP_DB "$SQL_CMD_NEW"`

                if [[ -z "$newrecord" ]]
                then
                        SQL_CMD="INSERT INTO PUPPET_FACTER (hostname,timestamp,uptime,physicalprocessorcount) \
                                VALUES ('$CLIENTCERT','$TIMESTAMP','$UPTIME','$PROCESSERCOUNT');"
               else
                        SQL_CMD="UPDATE PUPPET_FACTER SET uptime='$UPTIME', \
                                                 physicalprocessorcount='$PROCESSERCOUNT', \
                                                 timestamp='$TIMESTAMP' \
                               WHERE hostname='$CLIENTCERT';"
                fi

                if [[ -z "$SQL_CMD" ]]
                then
                        echo "SQL String is null" | log
                        return false
                fi

                if ! `$SQLITE_BIN $HADOOP_DB "$SQL_CMD"`
                then
                        return false
                fi
        else
                exit 1
        fi
}

# Main run
trap control_c SIGINT

while true
do
        NODE=`ls -1t $PUPPET_YAML_HOME | grep -i yaml | head -n1`
        if [[ ! -z "$NODE" ]] && [[ "$NODE" != "$LATEST_PUPPET_NODE" ]]
        then
                LATEST_PUPPET_NODE=$NODE

                if harvest_data
                then
                        echo -en "Data harvesting from $LATEST_PUPPET_NODE ...... [DONE]\n" | log
                else
                        echo -en "Data harvesting from $LATEST_PUPPET_NODE ...... [FAILED]\n" | log
                fi

        fi
done

Quick and easy init script

Quick and easy init script - puppetwatchdog

This is the init script that developed to launch puppetwatchdog in order to gather all the facter values from the puppet agent. Then, it will bump it into a sqlite3 database for further data analysis.

Puppetwatchdog should be setup at puppetmaster.

If you want to write a quick and easy daemon script at /etc/init.d/*. You can pretty much take a look at this code.


#
# chkconfig: 2345 90 60
# description: This is a watchdog where is collected \
#              all the puppet yaml files and inject it to a db
##### END INIT INFO

RETVAL=0
prog="puppet_watchdogd"
exec=/usr/bin/puppet_watchdogd
lockfile=/var/lock/subsys/puppet_watchdogd

# Source function library.
. /etc/rc.d/init.d/functions

start() {
        if [ $UID -ne 0 ]; then
                echo "User has insufficient privilege."
                exit 4
        fi

        [ -x $exec ] || exit 5
        echo -n $"Starting $prog: "
        daemon "$prog 2>&1 &"
        retval=$?
        echo
        [ $retval -eq 0 ] && touch $lockfile
}


stop() {
        if [ $UID -ne 0 ] ; then
                echo "User has insufficient privilege."
                exit 4
        fi

        echo -n $"Stopping $prog: "
        killproc $exec
        retval=$?
        echo
        [ $retval -eq 0 ] && rm -fr $lockfile
}


restart() {
        stop
        start
}

case "$1" in
        start)
                start

        ;;

        stop)

                stop

        ;;

        restart)

                restart

        ;;

        *)

        echo $"Usage $0 {start|stop|restart}"

        exit 2

esac

exit $?

The hint of extracting tarball

I created a directory called, file. Then, i touched 3 dummy files, file1, file2, file3.

[root@rhel54-1 file]# ls
file1 file2 file3

when i zip those file. I choose two different ways, one with gzip compression and one without

[root@rhel54-1 ~]# tar -cvf file.tar.gz file/
file/
file/file2
file/file3
file/file1


[root@rhel54-1 ~]# tar -zcvf file.tgz file
file/
file/file2
file/file3
file/file1

[root@rhel54-1 ~]# tar -jcvf file.tar file
file/
file/file2
file/file3
file/file1


I have three tarball files right now. Then, I do a file command on my tarballs file

[root@rhel54-1 ~]# file file.tar.gz
file.tar.gz: POSIX tar archive

[root@rhel54-1 ~]# file file.tgz
file.tgz: gzip compressed data, from Unix, last modified: Thu Sep 10 10:27:48 2009

[root@rhel54-1 ~]# file file.tar
file.tar: bzip2 compressed data, block size = 900k

Cool. I have three different formats. POSIX tar archive, and gzip and bzip2 file.

So, when i extract the tarballs. i will prompt in the command.
1. tar -zvxf for gzip tarball
2. tar -xvf for POSIX tar archive
3. tar -jvxf for bzip2 tarball

Obviously, the size of those tarballs also varying.

[root@rhel54-1 ~]# ls -sh file.*
4.0K file.tar 12K file.tar.gz 4.0K file.tgz

gzip and bzip2 is much more better than POSIX tarball.

RPM packaging

NUM_RPM_INSTANCE == 1 is a normal installation

NUM_RPM_INSTANCE == 2 is a normal upgrading

NUM_RPM_INSTANCE == 0 is a normal uninstallation

Here are the actual values passed during an install:

Run %pre of new package (1)
Install new files
Run %post of new package (1)

Here are the values passed during an upgrade:

Run %pre of new package (2)
Install new files
Run %post of new package (2)
Run %preun of old package (1)
Delete any old files not overwritten by newer ones
Run %postun of old package (1)

Here are the values passed during a delete:

Run %preun of old package (0)
Delete files
Run %postun of old package (0)

Getting GREP on some colors

export GREP_COLOR='1;32'

Highlighting matching on regexp

hi all,

If you wish to declare a environment variable to have regular expression operates with grep where the standard output showing the colors, in fact, you can!

export GREP_OPTIONS='--color=auto'

Have fun.

yen_onn_hiu@nasilemak:~/regexp> grep .a.. text1
sales1.xls
sales2.xls
sales3.xls
apac1.xls
na1.xls
na2.xls
sa1.xls

VAS troubleshooting skills

Yesterday, I have been spending times to debug the vastool. That's give me error when prompting the id of the account.

End up, I did a "vas merge" to get all the vas cache load into the /etc/passwd and /etc/group. That's pain. But, It works temporarily.

Here are the bunch of vas command for troubleshooting.

1. To check on the user access

vastool user checkaccess [account_name]

2. To get the vastool status result

ftp://ftp.vintela.com/vas/support/vas_status.sh.gz

To get the vastool snapshot

/opt/quest/libexec/vas/scripts/vas_snapshot.sh

3. To flush the cache

vastool flush

4. To check is vas

vastool isvas user [account name]

5. Test AD connection and user password
vastool kinit [account_name]

6. Test VAS service access control (assuming your using sshd)
vastool user checkaccess -s sshd [acount_name]

7. Check authentication logs
1) Add the words "debug" and "trace" to the end of all pam_vas lines in /etc/pam.d/system-auth
2) Add "*.debug /var/log/debug" to /etc/syslog.conf
3) Restart syslog daemon
4) Attempt login
5) Look at output in /var/log/debug for indication/reason of failure

8. You can also turn on vasd debug if desired
1) vastool configure vas vasd debug-level 3

vmware way of discovering lun

In order to list a vmware lun, please do this.

[root@testdev01 volumes]# esxcfg-mpath -l
Disk vmhba1:0:16 /dev/sdr (204800MB) has 4 paths and policy of Most Recently Used FC 10:0.0 10000000c9610905<->5006016941e02f2b vmhba1:0:16 Standby FC 10:0.0 10000000c9610905<->5006016141e02f2b vmhba1:1:16 On active FC 14:0.0 10000000c96106fc<->5006016041e02f2b vmhba2:0:16 On FC 14:0.0 10000000c96106fc<->5006016841e02f2b vmhba2:1:16 Standby preferred

Disk vmhba1:0:17 /dev/sds (204800MB) has 4 paths and policy of Most Recently Used FC 10:0.0 10000000c9610905<->5006016941e02f2b vmhba1:0:17 Standby FC 10:0.0 10000000c9610905<->5006016141e02f2b vmhba1:1:17 On active FC 14:0.0 10000000c96106fc<->5006016041e02f2b vmhba2:0:17 On FC 14:0.0 10000000c96106fc<->5006016841e02f2b vmhba2:1:17 Standby preferred

Disk vmhba1:0:18 /dev/sdt (204800MB) has 4 paths and policy of Most Recently Used FC 10:0.0 10000000c9610905<->5006016941e02f2b vmhba1:0:18 Standby FC 10:0.0 10000000c9610905<->5006016141e02f2b vmhba1:1:18 On active FC 14:0.0 10000000c96106fc<->5006016041e02f2b vmhba2:0:18 On FC 14:0.0 10000000c96106fc<->5006016841e02f2b vmhba2:1:18 Standby preferred