ERROR: This RRD was created on another architecture

Recently I have to migrate our cacti installation to new machine during which i encounter few issues.

After copying the RRD files to new machines and accessing cacti web http://myserver/cacti gives empty graphs.  (wired)

After digging the logs (in apache error log) found out that RRD files is not portable between different architechture.

ERROR: This RRD was created on another architecture

In my case the old machine was 32bit arch with 32bit OS, and the new machine is 64bit with 64bit OS.

Quick googling tells that using rrddump and rrdrestore one can transfer RRD between architectures with ease.


EXAMPLES

To transfer an RRD between architectures, follow these steps:

1.  On the same system where the RRD was created, use rrdtool dump to export the data to XML format.

# for i in ./*.rrd;do rrdtool dump $i ../rrd/$i.xml;done
(to converted all the files in rrd directory to XML format)

2.  Transfer the XML dump to the target system.

3.  Run rrdtool restore to create a new RRD from the XML dump.

# for i in ./*.xml; do rrdtool restore "$i" "../rrd/${i%.xml}"; done

After this cacti was up and running with all the old graphs migrated successfully.

Installing Nagios-plugins and NRPE on Centos 6

To monitor remote hosts CPU load, disk partitions, processes etc with Nagios, requires to install NRPE and nagios-plugins on the remote host.

Nagios-plugins and NRPE is not available from Centos official repositories, so first of all we need to configure RPMforge> repo from where to install the required packages.

Read my previous article before configuring and installing anything from third party repo.

Installing RPMforge on Centos 6

Download and install the rpmforge-release package. Choose one of the two links below, selecting to match your host's architecture. If you are unsure of which one to use you can check your architecture with the command uname -i

x386
# rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-1.el6.rf.i686.rpm

x86_64
# rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-1.el6.rf.x86_64.rpm

Do not forget to set the priority,

vi /etc/yum.repos.d/rpmforge.repo

[rpmforge]
name = RHEL $releasever - RPMforge.net - dag
baseurl = http://apt.sw.be/redhat/el6/en/$basearch/rpmforge
mirrorlist = http://apt.sw.be/redhat/el6/en/mirrors-rpmforge
#mirrorlist = file:///etc/yum.repos.d/mirrors-rpmforge
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-dag
gpgcheck = 1
priority=10

Now everything is setup now you can install nagios-plugins and nrpe packages from RPMforge repo

# yum install nagios-plugins nagios-nrpe

Running NRPE under xinetd

Edit /etc/xinetd.d/nrpe for

disable         = no
only_from       = 127.0.0.1 Nagios ServerIP

Restart xinetd service
# /etc/init.d/xinetd restart

My KVM howto

Enabling KVM support on your hardware

The host machines need to be running either Intel VT or AMD-V chipsets that support hardware-assisted
virtualization.

# grep -E 'vmx|svm' /proc/cpuinfo

If this command returns output, then your system supports KVM. The vmx CPU feature flag represents
Intel VT chipset while the svm flag represents AMD-V. Note which KVM flag was returned as it will be
useful for loading the right module later.

Next you need to ensure the KVM-related feature is enabled in the BIOS

1. In the BIOS menu, select Advanced Step → CPU Options.
2. Make sure the Intel Virtualization Technology option is Enabled.

Installing and configuring KVM-related package

Install the KVM software using yum:
# yum install kvm

Install additional virtualization management packages:

# yum install libvirt python-virtinst libvirt-python libvirt-client bridge-utils


Configuring KVM after installing the packages

After you have installed the KVM-related packages, load the right KVM modules by following these
steps:

Procedure
1. Insert KVM module by running the following command:
# modprobe kvm

2. Insert the chip-specific KVM module by running one of these commands:
For the AMD chip (svm flag)
# modprobe kvm-amd

For Intel chip (vmx flag)
# modprobe kvm-intel

You can verify that the modules are inserted and running.
# lsmod|grep kvm
kvm-intel    86248 3
kvm     223264 1 kvm_intel

3. Start the libvirtd daemon service:
# /etc/init.d/libvirtd start

Starting libvirtd daemon:
# /etc/init.d/libvirtd status
libvirtd (pid 6584) is running.

4. Set up libvirtd to start on every reboot:
# chkconfig libvirtd on

5. Better to reboot the host machine.


Creating LVM for Guest OS (Run lvdisplay before creating new LVM)

# lvcreate -L 4G -n kvm5 vg_kvm

-L 10G = replace with filesystem size
-n kvm1 = replace with kvm#

Creating KVM guest using virt-install

virt-install --name kvm1 \
--ram 512 \
--os-type='linux' \
--disk path=/dev/mapper/vg_kvm-kvm1 \
--network network:default \
--accelerate \
--vnc \
--cdrom /iso/CentOS-5.5-x86_64-bin-DVD-1of2.iso

For network installation replace --cdrom with --location with URL to OS installation link.

Eg, Centos 5.6 64bit
--location http://mirror.its.sfu.ca/mirror/CentOS/5.6/os/x86_64/

Creating Scientific Linux 6 64bit vm with bridge networking and two virtual CPUs

Note: br0 bridge interface should exist before using it for KVM

virt-install --name kvm3 \
--ram 1024 \
--vcpus 2 \
--os-type='linux' \
--disk path=/dev/mapper/vg_kvm-kvm3 \
--network bridge:br0 \
--accelerate \
--vnc \
--location http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/os/


*** Installing Slackware off NFS (NFS running on xxx.xxx.xxx.60 machine)

virt-install --name slack64 \
--ram 512 \
--vcpus 1 \
--os-type='linux' \
--disk path=/dev/mapper/vg_kvm-kvm8 \
--network bridge:br0 \
--accelerate --vnc \
--cdrom /iso/slackware_x86_64-13.37-mini-install.iso

Cloning KVM VM
1. LVM based
Cloning KVM (Create LVM partition for cloned first ie 'lvcreate -L 4G -n kvmX vg_kvm' run lvdisplay to check existing LVMs)
virt-clone --original kvm1  \
             --name kvmX \
              --file /dev/mapper/vg_kvm-kvmX --prompt


2. File based
virt-clone --original kvm1  \
             --name kvmX \
              --file /var/lib/libvirt/images/centos6-2.img

Resizing KVM VM disk storage
1. install libguestfs-tools package.
2. Best to have separate parititons for /, /boot, and swap, then following http://askaralikhan.blogspot.com/2011/07/expanding-kvm-guest-disk-image-using.html


Troubleshooting For network interface

1. Open /etc/sysconfig/network-scripts/ifcfg-eth0 and delete the following line and save the file
HWADDR=52:54:00:8f:62:01
2. Remove the following filen and reboot the virtual machine
/etc/udev/rules.d/*persistent-net*'
After reboot eth0 network interface will be available.

Removing LVM partition: Remove LVM partition which is no longer need.


# lvchange -an /dev/mapper/vg_kvm-kvm8 (change LVM status to inactive)
# lvremove /dev/mapper/vg_kvm-kvm8 (remove the LVM paritition)

Expanding a KVM guest disk image using virt-resize

Decide on how you wish to expand the guest. Run virt-df -h and virt-list-partitions -lh on the guest disk, as shown in the following output:

# virt-df -h /dev/vg_kvm/kvm7
Filesystem                                Size       Used  Available  Use%
/dev/vg_kvm/kvm7:/dev/sda1              484.2M      48.8M     410.4M   11%
/dev/vg_kvm/kvm7:/dev/sda3               2.5G       1.3G      25.4G    5%

# virt-list-partitions -lh  /dev/vg_kvm/kvm7
/dev/sda1 ext4 500.0M
/dev/sda2 swap 1.0G
/dev/sda3 ext4 2.5G

This example we will increase the size of /dev/sda3 to 30G

1. Shutdown guest

2. Rename the original disk for the backup purpose, for lvm use lvrename command, for file use 'mv' command

# lvrename /dev/vg_kvm/kvm7 /dev/vg_kvm/kvm7.backup
  Renamed "kvm7" to "kvm7.backup" in volume group "vg_kvm"

3. Create new disk
# lvcreate -L 30G -n kvm7 vg_kvm
Logical volume "kvm7" created

4. Now resize the /dev/sda3, refer to virt-resize man for detail.

# virt-resize \

       /dev/vg_kvm/kvm7.backup /dev/vg_kvm/kvm7 \

       --expand /dev/sda3

Summary of changes:
/dev/sda1: partition will be left alone
/dev/sda2: partition will be left alone
/dev/sda3: partition will be resized from 2.5G to 28.5G
Copying /dev/sda1 ... done
Copying /dev/sda2 ... done
Copying /dev/sda3 ... done

Parition /dev/sda3 contains a filesystem or LVM PV, then this content is not automatically resized.  You can resize it afterwards either using guestfish(1) (offline) or using commands inside the guest (online resizing).

Here we resize the filesystem using guestfish

[root@kvm ~]# guestfish -a /dev/vg_kvm/kvm7

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

> run
> resize2fs /dev/sda3
libguestfs: error: resize2fs: resize2fs 1.41.12 (17-May-2010)
Please run 'e2fsck -f /dev/vda3' first.
> e2fsck-f /dev/sda3
> resize2fs /dev/sda3
> exit

[root@kvm ~]# virt-df -h sl6-5
Filesystem                                Size       Used  Available  Use%
sl6-5:/dev/sda1                         484.2M      48.8M     410.4M   11%
sl6-5:/dev/sda3                          28.1G       1.3G      25.4G    5%

Try to boot the virtual machine. If it works you can delete the backup disk.

# lvchange -an vg_kvm/kvm7.backup

# lvremove vg_kvm/kvm7.backup
  Logical volume "kvm7.backup" successfully removed

Clone KVM with virt-clone

virt-clone is a command line tool for cloning existing virtual machine images using the "libvirt" hypervisor management library. It will copy the disk images of any existing virtual machine, and define a new guest with an identical virtual hardware configuration. Elements which require uniqueness will be updated to avoid a clash between old and new guests

virt-clone of python-virtinst package is a great tool for creating cloned KVM virtual machine quickly. 

Example:  LVM based kvm

# virt-clone --original sl6-3  \
             --name sl6-4 \
              --file /dev/mapper/vg_kvm-kvm6 --prompt

In the above example /dev/mapper/vg_kvm-kvm6 LVM should be created before executing the virt-clone command.

Creating LVM
# lvcreate -L 4G -n kvm6 vg_kvm

Example file based storage
virt-clone \
              --original ubuntu11 \
              --name ubuntu-2 \
              --file /var/lib/libvirt/images/ubuntu-2.img

--original sl6-3 = name of the existing KVM (make sure the original KVM is shutdown before creating clone out of it)

--name se6-4 = Name of the New KVM (cloned)
--file = In LVM case path to the existing storage, in file based storage the file will be automatically created by virt-clone

Troubleshooting Network issue
Upon booting the new cloned KVM VM I encounter with problem of not showing any network interface. After digging more it appears that udev on virtual machine changed the eth0 interface with eth1 due to change in MAC of the interface. Quick search on Google suggested the following solution to get back the 'eth0' interface on cloned vm.

1. Open /etc/sysconfig/network-scripts/ifcfg-eth0 and delete the following line and save the file

HWADDR=52:54:00:8f:62:01

2. Remove the following filen and reboot the virtual machine

/etc/udev/rules.d/*persistent-net*'

After reboot eth0 network interface will be available.

Scientific Linux 6

Recently I start using Scientific Linux on KVM virtual hosts. For anyone new to Scientific its a Enterprise Linux based on RHEL just like Centos maintained by Fermi National Accelerator Laboratory and the European Organization for Nuclear Research 'CERN' - and RHEL, of course :)

Scientific Linux folks is closely following the upstream srpm provider (RHEL) and its already shipped with version 6 just like RHEL 6.

The other things which inspired me to give a try to Scientific Linux is the latest version of Linux Kernel with latest hardware support, the ext4 filesystem is the default, KVM enhancement, latest versions of software eg PHP, Mysql which means I don't have to mess around with test repos or yum priorities

For detail What's new in RHEL/Scientific Linux please refer to RHEL.

I encourage anyone interested in RHEL and/or building your own test server, to experiment with this distro.

Medialess installation of Redhat Enterprise 6/Centos/Scientic Linux 6

This procedure assumes you are already using Red Hat Enterprise Linux or another relatively modern Linux distribution, and the GRUB boot loader. It also assumes you are a somewhat experienced Linux user.

To perform medialess installation of Redhat Enterprise 6/Centos/Scientific Linux 6 without media or a PXE server, the system must have two files stored locally, a kernel and an initial RAM disk

Copy the vmlinuz and initrd.img files from a Red Hat Enterprise Linux DVD (or DVD image) to the /boot/ directory, renaming them to vmlinuz-install and initrd.img-install.

Editing the GRUB Configuration

The GRUB boot loader uses the configuration file /boot/grub/grub.conf. To configure GRUB to boot from the new files, add a boot stanza to /boot/grub/grub.conf that refers to them


title Installation
    root (hd0,0)
    kernel /boot/vmlinuz-install ip=xxx.xxx.xx.x netmask=255.255.255.0 gateway=xxx.xx.xx.x dns=xxx.xx.xx.x vnc vncpassword=qwerty repo=http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/os/ lang=en_US keymap=us
    initrd /boot/initrd.img-install

If using network interface other than eth0 then you must have to specify it with ksdevice=ethX, then the kernel line will read something like this for eth1

kernel /boot/vmlinuz-install ip=xxx.xxx.xx.x netmask=255.255.255.0 gateway=xxx.xx.xx.x dns=xxx.xx.xx.x ksdevice=eth1 vnc vncpassword=qwerty repo=http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/os/ lang=en_US keymap=us

The following options are generally useful for medialess installations:

IP = yourIP
netmas = Netmask
gateway = your Gateway
dns = your dns
vnc (for remote graphical installation)
vncpassword = Yourpassword
repo = Redhat/Centos/SL repo (In the sample I use the Scientific Linux 6 repo)

Also change the entry "default 0" to boot from new kernel entries, when done reboot the system.

Completing the installation:
From your local system use vncviewer to connect to the remote server and complete the installation process

$ vncviewer RemoteHostIP