I don’t know that I linked to these slides yet: Using Capistrano for System Administration from today’s GSLUG meeting. Good times.

After a script ran “gem update –system” and blew up a debian box for the nine millionth time, I’ve been on a kick to get rid of gem’s altogether. Here are some directions for build a deb for capistrano:

apt-get  install build-essential fakeroot ruby-pkg-tools
svn checkout svn://svn.debian.org/pkg-ruby-extras/packages-wip/capistrano/trunk/
wget http://rubyforge.org/frs/download.php/33072/capistrano-2.2.0.tgz
tar -xvzf capistrano-2.2.0.tgz
mv trunk/* capistrano-2.2.0/
cd capistrano-2.2.0/
dch –v2.2.0
dpkg-buildpackage –rfakeroot

Or you can just grab my capistrano deb (for etch / hardy, any arch): capistrano_2.2.0_all.deb.

OSCON 2008 turned me on to some new virtualization tools. Right now I build VMs by using a Capistrano script that connects to a vmware server, builds a vmdk, creates a custom vmx via an erb, and netboots the box for a debian install.

I’m looking at moving to having ubuntu-vm-builder create ubuntu boxes instead. So I built an ubuntu blade first. I don’t know how long it takes to build a vmware box, but it takes a bit for the preseeded debian install to run, maybe 15 minutes. The first box I built with ubuntu-vm-builder made me happy:

apt-get install ubuntu-vm-builder
time sudo ubuntu-vm-builder kvm hardy –mem 2048 –mirror http://ubuntu.widemile.com/ubuntu
[snip]
real    2m21.705s
user    0m33.254s
sys     0m22.941s

I then installed kvm (apt-get install kvm) but got this error when it tried to start the kvm init script:

FATAL: Error inserting kvm_intel (/lib/modules/2.6.20-16-generic/kernel/drivers/kvm/kvm-intel.ko): Operation not supported

LP #104297 made me check the dmesg, and I saw:

  kernel: [ 899.340000] kvm: disabled by bios

I restarted, hit bios, and enabled VT. And then on startup the kvm module was already loaded.

It took me a bit to figure out qemu though. It’s important to read both the kvm and qemu man pages. When I first started it I saw errors about the framebuffer, and realized that since I was ssh’d into a remote machine, there wasn’t one. I found the “-nographic” option, but I later confirmed that the ubuntu install doesn’t configure a serial console by default (too bad). I spent a lot of time messing around with -vnc (an example in the man page would have been awesome). Starting up qemu and switching to the console I checked out ‘info vnc’ and it kept teling me that it wasn’t loaded. I tried ‘-vnc :0,password’ but when I ran ‘change vnc password’ in the console it would also segmentation fault.

Finally I figured out to run kvm/qemu WITH ‘-vnc’ and WITHOUT ‘-nographic’. Also, the standard network format for is ‘-vnc :n’ where an is an integer. The port you connect to becomes 5900+n. So if you start qemu/vnc with ‘-vnc :1’ you would connect to port 5901.

This isn’t any sort of advance performance benchmark, but running ‘time bzip2 -d linux-source.2.6.24.tar.gz2’ on the ubuntu kvm install took 15.967s (real) and on a debian vmware-server install it took 27.870s.

We’ve been joking for a while about how we need a big red flashing light to be triggered by nagios alarms. I’m not going to do it right now, but I thought about it.

There’s a debian x10 package for controlling x10 modules. Get a big rotating red light that runs on AC. Plug it into an X10 lamp module. I can’t find a “CP-290” module which is what the x10 package calls for. You could use this Powerlinc Serial/TW523 controller. It has source available, so you could figure out the protocol if it wasn’t compatible. Then write a script and keep it running (or cron) to do something like ‘/usr/sbin/nagios2stats -m -d NUMSVCPROB,NUMHSTPROB’ which prints out the number of service and host problems. If either of those are > 0, use the x10 module to turn the light on.

I recently had to explain to a math researcher why I wouldn’t deploy R packages directly to servers, which are basically tarballs. R has a CPAN like infrastructure too called CRAN. Today I’m screwing around with gems.

CPAN/CRAN/gems/etc are not debs.

/usr/bin/gem:23: uninitialized constant Gem::GemRunner (NameError)

This comes from running ‘gem update –system’ on a debian or ubuntu machine. Don’t. I’ll say that again, don’t. Rails people appear fond of gems, we need to stop this nuisance in it’s tracks. First, go read the debian position on gems, and appreciate it. Then, go and clean out ‘/usr/local/lib/site_ruby’. Run ‘gem -v’ and make sure gem is working again.

Fortunately lenny appears to protect you against running this command, but hardy doesn’t.

Maintaining servers is much more than getting software installed. I feel like I’ve ranted about this before, so I won’t again. Please, debianize your ruby gems, don’t use gem to install them. That goes for all your other forms of distributing language specific modules too.

I was troubleshooting some vlan issues today and was annoyed that when dumping eth0 (a trunk) I was not seeing the vlan headers for each packet. This makes it pretty difficult to tell which vlan the traffic is destined for, because I have to dump the vlan interface and the physical interface at the same time and use deduction.

I found this in the man page for vconfig which explains that Broadcom devices will strip vlan information:

VLAN will use Broadcom’s NICE interface when the network device supports it. This is necessary, since usually the hardware of these devices already  removes  the vlan tag from the ethernet packet. The set_flag option on vlan-devices created on such a physical network device will be ignored.  Dumping the network-device will show only untagged(non-vlan) traffic, and dumping the  vlan-devices  will  only  show  traffic intended for that vlan, without the tags.

This is probably in the top ten lamest hacks of my life. But I blame capistrano for not being what I want it to be.

This replaces BINDIPADDRESS in the configuration file with the IP address of eth0 on each host.

def get_ip_address (ethernet_device = "eth0")
  ip_addresses = Hash.new
  run("/sbin/ifconfig #{ethernet_device} | grep inet | grep -v inet6") do |ch, stream, data|
    case data
    when /s*inet addr:((d+.){3}d+)s+.*/
      ip_addresses["#{ch[:host]}"] = $1
    end
  end
  return ip_addresses
end  
task :put_auto_install_xml, :roles => :app do
    ip_addresses = get_ip_address()
    auto_install =
      IO.read(
        File.join(File.dirname(__FILE__), "files", "#{application}", "#{application}_auto_install_#{env}.xml")
      )
    put(
      auto_install,
      "/tmp/#{application}_auto_install-#{env}-#{build}.xml",
      :mode => 0644
    )
    run("/bin/bash") do |ch, stream, data|
      ch.send_data "/bin/sed -i -e 's/BINDIPADDRESS/#{ip_addresses[ch[:host]]}/' /tmp/#{application}_auto_install-#{env}-#{build}.xmln"
      ch.send_data "exitn"
    end
  end

1) When preseeding a debian install, you need the network up to be able to get to the preseed file.
2) The kernel used by debian-installer defaults to only allowing 256 characters on the append/cmdline.

Phil Hands did some stuff for etch / debian-installer to try to help out with this. This is the auto-install udeb. Most importantly is trying to get all the network information on the command line. We can do some tricks to save space.

1) Use preseed aliases where possible (Debian Manual, Appendix B 2.4)
2) Use ‘auto=true’ on the append line. This will use ai-choosers to delay locale/keyboard so you can specify these in your preseed file instead. It also lets you set ‘url=hostname.domain.org’ rather than a full path to your preseed file, but you’ll need symlinks so that ‘http://hostname/d-i/etch/preseed.cfg’ points to your config file. This is a little obtuse in the directions because most documentation talks about ‘auto’, but this is actually a label in the default debian pxelinux/syslinux.cfg files. Look in debian-installer/i386/pxelinux.cfg from the netboot.tar.gz for an example.
3) Setup symlinks for your initrd so you can use something like ‘initrd=d/e/3/i” which is short for ‘initrd=debian/etch/i386/initrd.gz’

My append line ends up looking something like this:

append initrd=d/e/3/i auto=true url=debian.example.org hostname=myhost \

domain=example.org interface=eth0 netcfg/get_ipaddress=10.0.0.100 \

netcfg/get_netmask=255.255.255.0 netcfg/get_gateway=10.0.0.1 \

netcfg/get_nameservers=10.0.0.250 netcfg/disable_dhcp=true

It’s unfortunate that you’re required to add ‘netcfg/disable_dhcp=true’ to get netcfg to acknowledge that you’ve given it a static configuration. that’s a lot of wasted cmdline that may get you in trouble if you’re using high octets in ip addresses or a long hostname. That’s exactly 256 characters right there.

If I keep having problems with this, I’ll probably start using DHCP reservations instead of static addressing. You can even push the url out via dhcp. I’ve also gone back and shortened my url hostname to ‘d.example.org’ to gain a few more characters. I probably could just recompile the kernel to support more than 256 chars on the command line as well.

I deploy vmware-server with puppet and wanted to automate the installation of a serial number. I thought I was going to use ‘vmware-cmd’ for this, but I got the truth out of vmware-cmd.pl, that you use ‘vmware-vmx’. I really need to stop using vmware, most of it’s features seem to be that it was windows admin friendly gui interfaces all over the place, and while it has API’s, they’re poorly documented because they just aren’t that widely used. I story the key manually in iclassify for now, producing the vmwarekey variable because it’s an attribute.

if $vmwarekey {

  exec { "vmware-server-license":

    command => "/usr/lib/vmware-server/bin/vmware-vmx --new-sn ${vmwarekey}",

    onlyif => "/usr/bin/test ! -f /etc/vmware/license.vs.1.0-00",

    require => Package["vmware-server"],

  }

}

I wanted to use ssh authentication agent forwarding on a few hosts at work because I only keep my private keys at work on my workstation. You enable this on a case by case agent with the ‘-A’ option, but I don’t like remember such things, so you can add ‘ForwardAgent yes’ to your ~/.ssh/config file. However, it seemed unneeded to enable this for all hosts, as it’s got a slight security issue or two. You can limit the commands to a certain host by preceding the command with a ‘Host server_hostname’ line, but ‘server_hostname’ must match whatever you write on the command line. You can use wildcards to match a domain like ‘*.example.org’ but I really only use hostnames not fqdns when I use ssh, so I wanted to list multiple hostnames. You can do this like so:

Host *
ForwardX11 yes
ForwardX11Trusted no

Host ops ops.example.org secretbox otherbox
ForwardAgent yes

Outlook likes putting attachments and other crap in a stupid attachment called ‘winmail.dat’. Request-tracker doe not like this and neither do I. On debian, install the ‘ytnef’ package. ‘/usr/bin/ytnef-filter’ appears to be the ‘ytnefprocess.pl’ script metioned in the README file. Make sure you’ve got the perl package MIME requrements. I didn’t have to but you may need to install the ‘libmime-perl’ package. Then modify your RT email configuration to look like this:

helpdesk: "|/usr/bin/ytnef-filter|/usr/bin/rt-mailgate \
--queue helpdesk --action correspond \
 --url http://rt.example.com"
helpdesk-comment: "|/usr/bin/ytnef-filter|/usr/bin/rt-mailgate \
--queue helpdesk --action comment \
 --url http://rt.example.com"

That’s two lines altogether. WordPress sucks with this stuff. It’ll filter out the attachments it can, the useful ones like actual file attachments and leave the rest alone, like the proprietary MS crap.

Talked about infrastructure for a bit at GSLUG today, you can grab the slides if you want.

I need to put these somewhere because I keep losing them. Good reading if you’re into puppet / infrastructure.

Stories about people using puppet: Digg, Google, iLike. SA vs Devs.

I had not heard of R until very recently when I was asked to set up the ‘server’ interface to R on a box. Apparently a bunch of math dudes thought they were too smart for bc and wrote R… yeah, I dunno. Lots of parts of R seem like a reinvention of the wheel. R has it’s own packaging system, like CPAN, called CRAN. It’s a little frustrating to have cran, cpan, gem, and all these different different methods to install libraries. I’m glad debian is still taking to packaging them up in debs.

I had to rebuild the R packages from sid on etch as the binary packages were built against a newer version of libc. I did this by adding a deb-src line for sid to sources.list:

deb-src http://ftp.us.debian.org/debian sid main

And then apt-getting the source for each package and building it (dpkg-buildpackage -rfakeroot). This was a lot of work because of long list of build dependencies which I had to backport, build and install (including tcl/tk). When the dependencies for ‘r-cran-rserve’ were finally met I was good to go.

R has some cheesy imitation of tcp-wrappers built in. It’ll bind to 0.0.0.0 by default, but will drop connections from another host. Without even looking at the code I’m sure it’s crap, but you need to put “remote enable” in /etc/Rserv.conf, to disable this foolishness. This file didn’t exist out of any of the packages, but it’s just a flat config file. I won’t bother talking about the authentication infrastructure, except that it’s useless, and you should probably wrap this around ssl and certificates or whatnot if you care.

You can start Rserve with ‘R CMD Rserve’, but that’s a little lame. Or ‘/usr/lib/R/bin/Rserve’ which will probably need R_HOME defined (see next). There’s also ‘/usr/lib/R/bin/Rcmd Rserve’ which doesn’t need R_HOME defined.

Fatal error: R home directory is not defined
This isn’t a well written error. You need to export R_HOME=’/usr/lib/R’, not an actual home directory (not to be confused with the working directory either). I wanted to use runit but without recompiling Rserve it lacks an option to not fork.

setgid(1): failed.
apparently R has a system function that lets you run shell commands. you can use setuid/setgid in the config file, but setgid won’t work. I don’t know why.

The newer EEsensors em01b has a contact module for monitoring normally closed circuits. My HVAC has a common alarm circuit that can close if any of the alarms (such as the high water switch or low pressure switch) go off, but it’s normally open.

The logic behind normally closed circuits is that you can detect a wiring fault. I’m not sure why eesensors went with NC instead of NO. They’re generally pretty helpful at answering questions via email though.

The em01b normally returns ‘N’ in the contact section when open and a ‘W’ when closed. Normally you’re in ‘W’ and if it trips to ‘N’ you need to send a reset command to get it back to ‘W’ (/index.html?eL) after you fix the problem. So if you’re querying an attached circuit that is normally open, you need to send a reset every time you query, and keeping in mind that the result that’s returned from the reset command is not a valid query.

I modified the C version of their nagios script to support this with a ‘D’ option instead of a ‘C’ option. You can find the source in git at check_em01.c.