visudo: Are you really a joker?

What you see sometimes doesn't mean what you think!

I just came across a funny configuration bits on the /etc/sudoers file lately on centos64. The default configuration for /etc/sudoers comes with this (the last 2 lines at the config). Okay, gentlemen. Please pay attention with this. If you would be creating a directory like /etc/sudoers.d/, then basically you are asking sudoer to look at any of the small snippets with in it, or what they called it as drop-in files. Make sure you do have the snippet and stay within this directory. Otherwise, your visudo -c -f /etc/sudoers will complain and shout at you when you save it.

What is make me laugh at it was the syntax of the configuration #includedir. It is telling you that it has been turned on. Don't be fooled because it was commented out (as usual, a sys admin will think the same across, further more most of the tags was commented out). It *doesnt* mean what you think. So, watch out! It bites you!

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d

Python in Action: Paramiko handling ssh connection

I came across the ssh interactive session from bash long time ago using the expect. But, this morning I am exploring paramiko, and start to draft a stupid easy codes to test out the library. It is clean and blazing fast. Hope this small chunks of codes can get you started with all the complicated paramiko codes.

#!/usr/bin/python                                                          
import os                                                                  
import paramiko                                                            
import getpass                                                             
                                                                            
user=os.getlogin()                                                         
print "Connecting via %s: what is your password?" % user                   
pw = getpass.getpass()                                                     
ssh = paramiko.SSHClient()                                                 
#ssh.load_host_keys()                                                      
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())                  
ssh.connect('centos64-1', username=user, password=pw)                      
stdin, stdout, stderr = ssh.exec_command('uname -a')                       
print "output", stdout.read()                                              
ssh.close() 

 
After a day so, I feel like I want to do a little of enhancement out of it. So, I recode it something like below. Still feeling it is not as neath as I love to. But, overall it can carry out the work.

#!/usr/bin/python
import os
import paramiko
import getpass
import socket

def get_credential():
    user=os.getlogin()
    print "Connecting via [%s]: what is your password?" % user
    pw = getpass.getpass()
    return (user, pw)

def ssh_connect():
    (user, pw) = get_credential()
    try:
        ssh = paramiko.SSHClient()
        #ssh.load_host_keys()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect('centos64-1', username=user, password=pw)
        stdin, stdout, stderr = ssh.exec_command('uname -a')
        print "output: ", stdout.read()
        ssh.close()
    except paramiko.SSHException, e:
        print "Password is invalid:" , e
    except paramiko.AuthenticationException:
        print "Authentication failed for some reason"
    except socket.error, e:
        print "Socket connection failed:", e

def main():
    ssh_connect() 

if __name__ == '__main__':                                                    
    main()                 


After a day so, I have done some polishing on the same codes. Now, it can accept a list of target hosts with a comma delimiter. Then, the script will be connecting to target hosts one by one. Feel free to copy/use it. Thanks!

#!/usr/bin/python
import os
import paramiko
import getpass
import socket

def get_host():
    target_host = []
    input = raw_input("What is your target host? ")
    for item in input.split(","):
        target_host.append(item.strip())
    return target_host

def get_credential():
    pw = ""
    user=os.getlogin()
    print "Connecting via [%s]: what is your password?" % user
    while not pw:
        pw = getpass.getpass()
    return (user, pw)


def ssh_connect():
    target_host = get_host()
    (user, pw) = get_credential()

    ssh = paramiko.SSHClient()
    #ssh.load_host_keys()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    for h in target_host:
        try:
            print "Connecting to %s ..." % h
            ssh.connect(h, username=user, password=pw)
            stdin, stdout, stderr = ssh.exec_command('uname -a')
            print "%s>" % h,stdout.read()
        except paramiko.SSHException, e:
            print "Password is invalid:" , e
        except paramiko.AuthenticationException:
            print "Authentication failed for some reason"
        except socket.error, e:
            print "Socket connection failed on %s:" % h, e
    ssh.close()
def main():
    ssh_connect()

if __name__ == '__main__':                                                    
    main()


Here is the output that I got.

hiuy@nasilemak ~ $ ./pyssh.py
What is your target host? centos64-1, centos64-2
Connecting via [hiuy]: what is your password?
Password:
Connecting to centos64-1 ...
centos64-1> Linux centos64-1.hiu.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Connecting to centos64-2 ...
centos64-2> Linux localhost.localdomain 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Script to report the HBA cards status

I have a small function to detect the emulex, and qlogic HBA cards for a quick one. If you find it useful in your automation you can embed this function to your script and start calling it. Hope that it helps you in your works. Thanks!

function check_hba ()
{

        # Checking Emulex HBA cards status

        if grep -q -i "online" /sys/class/fc_host/host*/port_state 2>/dev/null
        then
                num_online_hba=`grep -i "online" /sys/class/fc_host/host*/port_state | wc -l`
                total_hbas=`ls -1 /sys/class/fc_host/host*/node_name | wc -l`

                echo "Linux discovered $num_online_hba HBAs is online out of $total_hbas cards."

                for h in `ls -1 /sys/class/fc_host/host*/node_name | awk -F/ '{print $5}'`
                do
                        wwn=`cat /sys/class/fc_host/$h/port_name`
                        state=`cat /sys/class/fc_host/$h/port_state`
                        #modelname=`cat /sys/class/scsi_host/$h/modelname`
                        #echo "HBA $modelname : $wwn is $state"
                        echo "HBA : $wwn is $state"
                done
        fi

        # Checking Qlogic HBA cards status
        if grep -q -i "ready" /proc/scsi/qla2xxx/* 2>/dev/null
        then
                total_hbas=`ls -1 /proc/scsi/qla2xxx/* | wc -l`
                num_online_hba=`grep -i "ready" /proc/scsi/qla2xxx/*| wc -l`

                echo "Linux discovered $num_online_hba is online out of $total_hbas cards."

                for h in `ls -1 /proc/scsi/qla2xxx/*`
                do
                        wwn=`cat $h|grep "scsi-qla.-adapter-port*" | awk -F= '{print $2}' | sed -e 's|;||g'`
                        grep -i -q "Host adapter.*ready" $h && echo "HBA : $wwn is Online" || echo "HBA : $wwn is Offline"
                done
        fi

}

LVM and hosttag

hi all,

LVM is working well with host tag especially when working for a active/passive cluster nodes. There are just some simple lines needed at /etc/lvm/lvm.conf

1. volume_list = [ "vg00", "vg01", "vg02", "vg03", "vg04", "vg05" ]
2. tags  { hosttags = 1 }


At the same time, you have to create another conf file with this command

echo "activation { volume_list=["@`hostname -f`"] }" >  /etc/lvm/lvm_`hostname -f`.conf

So, the result with the something like this.

[root@node01c lvm]# cat /etc/lvm/lvm_node01c.hiu.com.conf
activation { volume_list=["@node01c.hiu.com"] }

Once you are done, pretty much you are done.

When you are importing the vg01, you will need to add a tag onto it: vgchange --addtag vg01 upon the activation. Otherwise, you have to deport it so another node within the cluster can import it. To deport it, you have to do vgchange --deltag vg01. 

if you are working well, you will notice that the tagging is working against the lvdisplay too. For example:

[root@node01c lvm]# lvdisplay @node01c.hiu.com

  --- Logical volume ---

  LV Path                /dev/vg02/lvol01

  LV Name                lvol01

  VG Name                vg02

  LV UUID                CXLHnj-FWUY-Puej-r2SN-S7d5-Bkse-Ufx9yH

  LV Write Access        read/write

  LV Creation host, time node01c.hiu.com, 2013-10-03 15:34:30 +0800

  LV Status              available

  # open                 1

  LV Size                71.98 GiB

  Current LE             18428

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:19

Whereas, in the passive node, you wont find any output is printed.

[root@node02c lvm]# lvdisplay @node02c.hiu.com

[root@node02c lvm]#

Another method to determine your tagging is working fine by using the command lvs.


[root@node01c lvm]# lvs
  LV     VG   Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  crash  vg00 -wi-ao--  7.81g
  home   vg00 -wi-ao--  4.00g
  opt    vg00 -wi-ao--  4.00g
  tmp    vg00 -wi-ao--  4.00g
  usr    vg00 -wi-ao--  4.00g
  var    vg00 -wi-ao--  4.00g
  lvol01 vg01 -wi-ao--  3.98g
  lvol01 vg02 -wi-ao-- 71.98g
  lvol01 vg03 -wi-ao--  1.98g
  lvol01 vg04 -wi-ao-- 61.98g
  lvol01 vg05 -wi-ao--  4.98g

[root@node02c lvm]# lvs
  LV     VG   Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  crash  vg00 -wi-ao--  7.81g
  home   vg00 -wi-ao--  4.00g
  opt    vg00 -wi-ao--  4.00g
  tmp    vg00 -wi-ao--  4.00g
  usr    vg00 -wi-ao--  4.00g
  var    vg00 -wi-ao--  4.00g
  lvol01 vg01 -wi-----  3.98g
  lvol01 vg02 -wi----- 71.98g
  lvol01 vg03 -wi-----  1.98g
  lvol01 vg04 -wi----- 61.98g
  lvol01 vg05 -wi-----  4.98g

with this output, you can clearly see that there is a string like "wi-ao" appended onto the activated volume at the active node. 

Configuring bonding: RTNETLINK answers: File exists

hi all,

Today, I am trying to configure a active/passive bonding for linux, but keep hitting the same error when I restarted the network services.

[root@node01 network-scripts]# /etc/init.d/network restart
Shutting down interface bond0:                             [  OK  ]
Shutting down interface bond1:                             [  OK  ]
Shutting down interface eth12:                             [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface bond0:                               [  OK  ]
Bringing up interface bond1:                               [  OK  ]
Bringing up interface eth2:  RTNETLINK answers: File exists
                                                           [  OK  ]
Bringing up interface eth12:                               [  OK  ]

I have spent a couples of minutes to trouble but no luck. I took a rest then when back and look closely at my ifcfg-eth* configuration files and I discovered a silly mistake like this. I have stated SLAVE=no at ifcfg-eth2. that's why it keeps on complaining the problem.

[root@node01 network-scripts]# grep bond1 ifcfg-*

ifcfg-bond1:DEVICE=bond1

ifcfg-eth2:MASTER=bond1

ifcfg-eth9:MASTER=bond1

[root@node01 network-scripts]# cat ifcfg-eth2

DEVICE=eth2

BOOTPROTO=none

ONBOOT=yes

MASTER=bond1

SLAVE=no

USERCTL=no

HWADDR=3C:D9:2B:F4:A1:9A

[root@node01 network-scripts]# cat ifcfg-eth9

DEVICE=eth9

BOOTPROTO=none

ONBOOT=yes

MASTER=bond1

SLAVE=yes

USERCTL=no

HWADDR=44:1E:A1:17:F8:76

Hope that you wont be as silly as I am.

After all, you can verify your configuration at

[root@node01 network-scripts]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 1000
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:d9:2b:f4:a1:9a
Slave queue ID: 0

Slave Interface: eth9
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 44:1e:a1:17:f8:76
Slave queue ID: 0

Script to kill the command with timeout.

If you don't bother to wait the command to timeout. You can have a bash script as a wrapper to clean the PID within your defined timeout values. I came across the same function like this before in my senior codes, just can't recall how is the codes goes. Now, I am totally rewrite it based on my memory. Hope that it can help you when doing bash coding.

function run_with_command () {

CMD=$1

TIMEOUT=$2

COUNTER=0

${CMD} &

CMD_PID=$!

while ps $CMD_PID > /dev/null && [ $COUNTER -lt $TIMEOUT ]; do

sleep 1

COUNTER=$((COUNTER+1))

done

if [ $COUNTER -eq $TIMEOUT ]

then

kill $CMD_PID 2>/dev/null

fi

wait $CMD_PID 2>/dev/null

}

Script to rescan the offline disk

I have a function to detect the offline disks, delete them then, re-scan it. Hope that it can help you from your daily Linux operation.



function fix_stale_disk ()
{
        echo_header
        if grep -q -i offline /sys/block/*/device/state
        then

                #Checking for offline disks
                echo "Linux discovered offline disks."
                grep -l -i offline /sys/block/*/device/state

                echo "Do you want to fix the offline disks? (y/n)"
                read ans
                if [[ "$ans" == "y" || "$ans" == 'Y' ]]
                then

                        for block in `grep -l -i offline /sys/block/*/device/state | awk -F/ '{print $4}'`
                        do
                                status=""
                                echo -n "Deleting /dev/$block ..."
                                echo 1 > /sys/block/$block/device/delete
                                if [[ "$?" -eq "0" ]]
                                then
                                        status="Done"
                                else
                                        status="Failed"
                                fi
                                sleep 5
                                echo " $status"
                        done

                        echo ""
                        #Rescanning the disks.
                        for f in `ls -1 /sys/class/fc_host/*/issue_lip 2>/dev/null`
                        do
                                echo "Rescanning $f ..."
                                echo 1 > $f
                                sleep 5
                        done
                else
                        echo "You have halted the operation."
                        exit 1
                fi
        else
                echo "All disks are good."

        fi
}

Linux, Who eaten up my memory?

Linux, Who eaten up my memory?

if you want to know which PID that chewing up your memory and not letting go? Here is the small bash script.

#!/bin/bash
echo
for pid in `grep -i VmSize /proc/*/status | sort -k +2n | tail -n10 | awk -F"/" '{print $3}'`
do
        echo "************************"
        echo "PID: $pid"
        echo "Allocated memory: `grep VmSize /proc/$pid/status | awk -F":" '{print $2" "$3}'`"
        echo `ps -aef | grep $pid | grep -v grep`

done
echo

Python in Action: Search for big file in Linux

 Python in Action: Search for big file in Linux

Here is the codes. Hope that you will love it.

import os

def search():
    filesize = 0
    filename = ''
    for root, dirs, files in os.walk('/'):
        for f in files:
            f = os.path.join(root, f)
            if os.path.isfile(f):
                if os.path.getsize(f) > filesize:
                    filesize = os.path.getsize(f)
                    filename = os.path.getrealpath(f)
                    print "file: %s ==> size: %s" % (filename, filesize)
                else:
                    break
 
if __name__ == '__main__':
    search()

whoami?? whoami??

hi all,

I came across a weird case when I am working today. But, I am able to solve the problem in a minute. Just hope that I can share it out. The problem sounds as like below.

login as: hiuy
Last login: Fri Jun  7 09:01:01 2013 from 16.189.90.136
-bash: /etc/profile: Permission denied
-bash-3.2$

-bash-3.2$ id

uid=121738973 gid=6347 groups=10,6347,12132,12178

-bash-3.2$ whoami

whoami: cannot find name for user ID 121738973

[root@test ~]# ls -al /etc/passwd /etc/shadow /etc/group

-rw-r--r-- 1 root root 29575 Jun  7 08:02 /etc/passwd

-rw-r--r-- 1 root root 4368 Jun  7 08:55 /etc/group

-r-------- 1 root root 9012 Jun  7 08:02 /etc/shadow

The permission for the three important files are perfectly looks okay. So, What is the fault??

This is the hint....

-bash-3.2$ ls -al /etc/passwd

ls: /etc/passwd: Permission denied

The Answer is here...

[root@test /]# ls -al | grep etc

drwx------  89 root root 12288 Jun  7 08:55 etc

etc is having 700 only. No permission for mortal user to read and accessing /etc/passwd! So, chmod it to 755 /etc, and that's the end of the story!

This is cunning, but I like it....