oh grub2 how I hate thee

let me count the ways…

infinity plus 1) you think you are smart but you are not,
infinity plus 2) you do not give the people the power they need to help out when you are not smart — see number 1.

I simply wanted to upgrade my 1TB of software raid 1 to 1.5 TB of software raid 1, besides the time for copying and syncing it should be a slam dunk, but thanks to grub2 it was a super utter cluster f*ck.

I started by transitioning to a non mirrored 1.5 Tb disk with boot, swap and root, then after I was able to boot that, I built a set of degraded raid1’s on the other disk and then tried to bring the the first 1.5.

however my partition on the first half of the degraded raid was slightly larger than partition on the second half, I tried to resize it but then it wouldn’t boot… so I got to do the whole 1TB copy with billions of hard links over again… These are the steps to resize a raided partition if you are not using it for root http://www.howtoforge.com/how-to-resize-raid-partitions-shrink-and-grow-software-raid

Finally on the system 10 days later, still need to add the other half of the degraded disk, but this time they are the same size.

For grub2 help this page was helpful, (mostly pages 2 and 3): http://www.howtoforge.com/how-to-set-up-software-raid1-on-a-running-system-incl-grub2-configuration-ubuntu-10.04

I also manually updated the device map /boot/grub/device.map which may have helped things, dunno…

There was a post I read along the way but can’t find the URL that talked about using chroot, after mounting some things like dev and proc into the future chroot, that sounded promising, but I didn’t have to do that.

I also tried a shit ton of different ways to tell grub to boot my degraded mirror, ran grub-install, update-grub, and update-initramfs many, many, many times, what ultimately seemed to work was to run update-grub and grub-install again after I hacked my way to getting grub to boot up with root on /dev/md2. After the raid is done re-syncing I’ll try to pull the half that I built first and try to boot from the new half.

grub2 help for kernel upgrade on Ubuntu 12.04 domU on XCP

I wouldn’t be posting if this wasn’t yet another corner case, where things don’t work as they should… I likely built my xcp-xapi box at a bad time for xcp-xapi and grub2, hopefully this helps others…

Situation: installing new virtual kernel on Ubuntu 12.04 domU kernel upgrade crashes royally:

Error: Starting VM – Using to parse /boot/grub/grub.cfg – WARNING:root:Unknown directive load_video – WARNING:root:Unknown directive terminal_output – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive export – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown image directive gfxmode – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown directive submenu – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown image directive gfxmode – WARNING:root:Unknown image directive recordfail – Traceback (most recent call last): – File “/usr/lib/xcp/lib/pygrub.xcp”, line 853, in – raise RuntimeError, “Unable to find partition containing kernel” – RuntimeError: Unable to find partition containing kernel

Background: Ubuntu 12.04 dom0 running xcp

Work around (http://blog.403labs.com/post/1546501840/paravirtulization-with-citrix-xenserver-5-5-and-ubuntu):

 xe vm-param-set uuid=<VM-UUID> PV-bootloader-args="--kernel=/boot/vmlinuz-3.2.0-29-virtual --ramdisk=/boot/initrd.img-3.2.0-29-virtual"
xe vm-param-set uuid=<VM-UUID> PV-args="root=UUID=<disk-UUID> ro -- console=hvc0"

ubuntu 12.04 xcp bare metal recovery via rsnapshot

I’m verifying my backup process via a bare metal recovery to identical hardware so I thought I’d take some notes…

I got started on my ubuntu xen xcp/xapi setup with this excellent guide:
http://francispereira.com/blog/step-by-step-guide-to-setting-up-xen-and-xenapi-xcp-on-ubuntu-12-04-and-managing-it-with-xencenter/

Booted a usb based rescue disk to partition my disks the same as oringinal dom0, then rsync over the stuff from my dom0 rsnapshot, created /dev /proc and /sys directories, mounted them with –bind and then chrooted on to the actual partition to rebuild grub

sudo grub-install /dev/sdb
sudo grub-install --recheck /dev/sdb
sudo update-grub

Then umounted things and rebooted, and volia… but networking wasn’t happy… after some banging around I discovered my old nemesis the udev persistent crap, who ever thought of this has some explaining to do, how could this have ever been worth it? Maybe I’m just not in the use case where it’s good for my network adapter to seemingly randomly stop working. None the less, a quick

rm of /etc/udev/rules.d/70-persistent-net.rules

and a reboot and things were back to the way they should be. (also updated my rsnapshot exclude list so I can forget about udev for a while).

UPDATE(9/3/12): just booted the box and realized that it was unhappy about UUIDs of the new filesystems (/dev/sda1 (root) and /dev/sda2 (swap) in my case so some grub editing was needed to boot, and then also updating of /etc/fstab and running update-grub.

With xen xcp-xapi on ubuntu we’re pretty much off book so for this portion of the restore I’m planning to clean things out, create a new LVM, and then import from basic generic vxa images, and then restore domU data/settings from rsnapshot.

I cleaned out the xe setup with a number of commands including xe sr-forget, xe vm-destroy, xe pbd-unplug, xe pdb-destroy, xe vdi-destroy, and xe vbd-destroy, also the -list version of the above helps to identify what to destroy.

I recreated the LVM (synopsis of steps from above guide).

fdisk, n, p, 3, , t, 8e, W. #(use the rest of the disk)
pvcreate /dev/sda3
vgcreate VolumeGroup /dev/sda3
pvdisplay #(get size from here)
lvcreate --size G -n LocalStorage VolumeGroup
xe sr-create type=ext name-label=Local Storage device-config:device=/dev/mapper/VolumeGroup-LocalStorage

Next I set the recently created sr as the default SR and imported a vxa. then used nmap to find it, as that’s faster way to find it than logging in to my wrt.

xe sr-list
xe pool-list
xe pool-param-set uuid= default-SR=
xe vm-import filename=.vxa
xe vm-start vm=
nmap -PN -sS 10.0.0.0/24 192.168.1.0/24

I then rsync’ed the domU settings/data back and rebooted. I use the -anv in order to see what it’s going to do before it mucks things up…

rsync -anv --delete --exclude-from= . host:/|less
rsync -a -delete --exclude-from= . host:/

update(9/3/12): also snapshots weren’t working as I needed to specify the default SR

xe sr-list
xe pool-list
xe pool-param-set suspend-image-SR=  uuid=

Things I learned:

  • good to have the output of fdisk
  • I still really hate the udev permanent ethernet crap
  • bunch of xe commands

wordpress rss cache

This is a little off topic but wordpress RSS seems to cache things for 12 hours and nobody seems to have a clue.

I set lifetime from 43200 to 3600 in wp-includes/class-feed.php.
Which will hopefully let me see the morning news in the morning… As it seems that I have to wait for the initial cache to time out.

I also left the following in the wp-config.php file just incase it was part of the solution

define('ENABLE_CACHE', FALSE);
define('MAGPIE_CACHE_ON', 0);
define('MAGPIE_CACHE_AGE', 600);

perc 5/i

This seems to be the equivalent of a megaraid 8404E from LSI, it’s a great in that it’s compatible with SAS and SATA disks, and is available for cheap on ebay.

using it on linux you’ll want to get the megaCLI from LSI and for ubuntu, you’ll want to convert the rpm to deb with alien and then install some 32bit libraries, and then do some creative symlinking in in /lib to make it happy

you can also flash it to the latest dell firmware with the dell centos live cd, and then download the latest firmware from dell, put it on a usbdrive and voila.  There’s talk on the interwebs about using the lsi firmware, but as I’ve got mine in a dell, I thought I keep it au natural.

—
this also works with xenserver 6.5 – rpms to get are
Lib_Utils-1.00-09.noarch.rpm and
MegaCli-8.04.07-1.noarch.rpm

qmail on ubuntu

basically the qmail package in ubuntu kinda works, but things are in really unusual places, so for your sanity you should probably hand crank your qmail install.

If you don’t, you’ll want to replace the defaults in /var/lib/qmail/aliases for .qmail-default .qmail-postmaster and .qmail-root as the automagic install tries to do something fancy with an alias user who doesn’t exist.

smtproutes is your friend if you want to use qmail to forward email… ssmtp is my goto but my favorite monitoring package nefu doesn’t work with ssmtp, so I’ve taken to installing qmail everywhere.

/var/lib/qmail/control/smtproutes (it’s /var/qmail/control/smtproutes for sane installs)

either use:

 :hostname

to forward all email to said host, or you can use an ip like this

:[192.168.1.1]

also you can make it only listen to localhost (avoid adding yet another open relay to the internet) by editing

/var/lib/qmail/control/qmail-smtpd/run

and changing the 0 before smtp to 127.0.0.1

qmail is kindof complex to control, so when in doubt kill everything off, and let svnscan, runsv, or what ever is keeping an eye on qmail processes restart them for you… again for your sanity you should roll your own or use qmailtoaster on centos (I mean how hard is it to create an init script that behaves the way that you’d expect?)