btm.geek

kvm-img convert small.img small.raw # this is your old image
kvm-img create large.raw 15G # or whatever size
losetup /dev/loop0 small.raw
losetup /dev/loop1 large.raw
dd if=/dev/loop0 of=/dev/loop1
losetup -d /dev/loop0
losetup -d /dev/loop1
kvm-img convert large.raw large.qcow2


Start KVM up again with the new large image. Go to the ‘Computer Management’ mmc applet under ‘Administrative Tools’. Choose ‘Disk Management’ under ‘Storage’. Right click your existing volume and choose extend. Step through the wizard. I got a message that made it appear it didn’t work, but I guess I didn’t read the fine print right and it did work fine enough. There’s always the old small image file if you run into problems. Speaking of which, be careful to not be dyslexic when inputting dd options.

Needing to set ‘Password Never Expires’ across an entire OU in Active Directory, I managed to write a powershell script to accomplish as much. Sure is nice having scripting languages on Windows machines beyond BASIC.

# Finds all user objects in the searchroot and forces the password never expires value in user account control to set
# 2009-09-04 -- Bryan McLellan <btm@loftninjas.org>

$Never_Expire=0x10000

$objou = New-Object System.DirectoryServices.DirectoryEntry("LDAP://ou=test,dc=example,dc=com")
$objSearcher = New-Object System.DirectoryServices.directorySearcher
$objsearcher.searchroot = $objou
$objsearcher.filter = '(&(objectCategory=User)(Objectclass=user)(!isCriticalSystemObject=*))'
$objsearcher.searchscope = "subtree"

$results = $objsearcher.findall()

foreach ($result in $results) {
  $user = [adsi]$result.path
  $value = $user.useraccountcontrol.item(0)
  $value = $value -bor $Never_Expire
  $user.useraccountcontrol = $value
  $user.name
  $user.setinfo()
}

The openid-ldap project provides a simple OpenID to LDAP gateway that supports Active Directory so you can leverage your existing SSO database hosted in LDAP to provide OpenID logins.

All the documentation lives in docs/README. Configuration is mostly a matter of unpacking the source into an apache hosted directory, editing ldap.php as described to contain the correct ldap URIs, and configuring apache. The LDAP configuration is relatively straight-forward if you’re familiar with setting up LDAP authentication elsewhere. The apache part took some tinkering for my setup.

I ran into three problems, the first was needing to modify the filter to remove ‘(mail=*)’ since these weren’t mail enabled accounts. I used ldapsearch (example in the README) based on my settings in ldap.php to see that no accounts were getting returned and realized these accounts weren’t mail enabled.

The next problem was because my production webservers are behind a load balancer and the configuration wants to use mod_proxy to connect back to itself, which would try to go back out the backside of the load balancer and cause all sorts of confusion. I used an internal hostname to pass the proxied requests directly back to the server. You’ll see this in the attached apache configuration below

The third was because of the load balancer and I discovered this by turning debug to true in index.php and dumping a log file in /tmp. Part of the authentication request was going to different servers. Only having a single server in this particular pool resolved that.

The test page on openid-ldap.org didn’t work for me and failed with “Authentication error; not a valid OpenID”, but logging into livejournal worked okay.

<VirtualHost *:80>
	ServerAdmin webmaster@example.org
	ServerName openid.example.org

  RewriteEngine On

  RewriteRule ^/(.*) https://openid.example.org/$1 [R,L]
</VirtualHost>

<VirtualHost *:80>
  ServerName openid

  RewriteEngine On

  RewriteCond %{HTTPS} !=on

  RewriteRule ^/(.*) https://openid.example.org/$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
	ServerAdmin webmaster@example.org
	ServerName openid.example.org
  ServerAlias openid

	DocumentRoot /var/www/example.org/openid

	<Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>
	<Directory /var/www/example.org/openid>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
	</Directory>

	ErrorLog /var/log/apache2/openid.example.org-error.log
	LogLevel warn

	CustomLog /var/log/apache2/openid.example.org-access.log combined

  <Proxy https://openid-internal.example.org/*>
    Order allow,deny
    Allow from all
  </Proxy>

  ServerSignature On
  RewriteEngine On

  RewriteCond %{REQUEST_URI}      !^/(.+)\.php(.*)$
  RewriteCond %{THE_REQUEST}      ^[A-Z]{3,9}\ /([A-Za-z0-9]+)\?(.*)\ HTTP/
  RewriteRule ^/(.*)$         https://openid-internal.example.org/index.php?user=%1&%2 [P]

  RewriteCond %{REQUEST_URI}         !^/(.+)\.php(.*)$
  RewriteRule ^/([A-Za-z0-9]+)$  https://openid-internal.example.org/index.php?user=$1 [P]

</VirtualHost>

I’m migrating a number of internal web application databases off of a mysql server and I wanted a way to see which databases are being accessed and by which hosts.

# tshark -R "mysql.opcode == 2" -e ip.src -e mysql.schema -T fields port mysql

When run on the mysql server this produces a tab separated list of values compromised of the client ip address and the database name when a mysql client specifies a database. See the man page for tshark for more information.

Update:

This catches the circumstance where database is set on login as well:
# tshark -R "mysql.schema" -e ip.src -e mysql.schema -T fields port mysql

I’ve amassed enough mysql databases that it’s time there should be some high availability. Note that this isn’t a single huge database, it’s a pile of wordpress, request-tracker, mediawiki, etc databases. Performance isn’t the goal here, it is automatic failover in case of impending doom.

I happen to have an iSCSI san, but in the efforts of simplicity I’m looking at Heartbeat+DRBD or Heartbeat+Replication.

Most tutorials, and comments from colleagues lean towards using Heartbeat+DRBD. There is good discussion of the two, and more recent followup regarding when to use DRBD. There’s a nice little table at the bottom of this page. If you dig deeper, there are respectable comments about using what’s appropriate to the situation, the exercise of which is left up to the reader.

The problem is that mysql defaults to using MyISAM as the storage engine, which lacks a transactional journal. When your primary host crashes and your secondary host comes up, unless there’s a journal to replay you’re just assuming everything isn’t corrupt without some kind of through consistency check. Which sounds time consuming. So switch all your tables to a transactional storage engine like InnoDB?

Replication has both a slave IO and a SQL process running, which I believe avoids this, since the replication slave isn’t going to run an incomplete SQL statement if the master dies while sending it to the slave it is dropped. Which leaves you possibly behind the master, but consistent.

So I’m going to try to configure heartbeat with mysql running replication between two guests. The best information I’ve found so far is from Charles Bennington. I’ll post a followup when I’m done with that project.

Playing with drbd8 on Ubuntu, loosely following these instructions, and I ran into a couple problems.

First, you need to use a kernel that has the drbd module as there is no drbd8-module-source, -server definitely has the drbd module, -virtual did not. Instructions about building the drbd module are old.

My secondary was also stuck in a connection state of “WFBitMapT”. I noticed the secondary was Ubuntu jaunty while the primary was Ubuntu intrepid. Upgrading the primary to jaunty resolved this.

I saw the error “local disk flush failed with status -95″ in the logs and wasn’t entirely sure about it but eventually found an explanation that made some sense and made me not worry about it.

drbd (/etc/init.d/drbd) doesn’t start on startup on it’s own. Most of the debugging information you’re looking for is in /proc/drbd or in your syslog output in /var/log. The only trouble is deciphering what is good and what is bad.

Upon returning from Open Source Bridge in Portland last week, I collected my thoughts from the convergence of configuration management developers and wrote The Configuration Management Revolution, centered around the idea the something bigger is happening than we’re acknowledging.

Today Tim O’Reilly posted a blog entry about the origins of Velocity. He says “I had been thinking in the abstract about the fact that as we move to a software as a service world, one of the big changes was that applications had people “inside” of them, managing them, tuning them, and helping them respond to constantly changing conditions.” which builds on his post three years ago about operations becoming the “elephant in the room”.

That article is worth revisiting. It tails off commenting on the lack of open source deployment tools. That has definitely changed, as we have a number of open source options in the operations tool space now. O’Reilly has published a few books on operations as well, although hasn’t taken the step of considering it a category in their book list yet.

The web is full of howtos, blog posts and assorted notes on piecing together open source software to build a server. One doesn’t have to be an expert on all of the ingredients, but rather be able to figure out how to assemble them. As time goes on, the problems of the past become easier to solve; former creative solutions become mainstream and the industry leverages those advantages. This frees up mindshare for something new. I’ll emphasize that this doesn’t mean one no longer needs to have some understanding of why the server works, but the time spent engineering that particular solution is reduced because we already have the wheel, so to speak.

Writing configuration management and thus infrastructure howtos may get one started, but it’s the old way of thinking. If you can write infrastructure as code, you can share infrastructure as code. It is essential that this is achieved in a format that both promotes sharing and is relatively easy. Take the Munin and Ganglia plugin sites for instance. Munin is relatively easy to get started with and has a simple enough site for exchanging plugins. While I consider Ganglia technically superior, it’s community is not. I tried submitting to Ganglia’s plugin site once and failed. This step has to be more than a site where files are dumped, it needs community support.

I asked Luke about this at OSBridge and he said Reductive Labs plans to have a module sharing website online soon for puppet. For now, you can find an number of puppet modules in the wiki. Opscode is on track, with their chef cookbooks available as a git repository on github, combined with a ticketing system allowing users to fork, modify and contribute changes. There’s even a wiki page helping to instruct how to leverage these.

Of course, you’ll always need experienced engineers to design and tune your infrastructure. However, the time and mindshare savings from creating a LAMP stack by setting a tag or role to ‘lamp’ is immense. As Opscode produces more open APIs between parts of their product, my mind imagines the offspring of the Chef UI and virt-manager. How long until the popup touting “New features are available for your web cluster”?

The revolution is coming, and it’s about time I wrote about it.

About a year and a half ago I was settling in to a new system administration job at a startup. I was told a consulting company would be coming in to bootstrap configuration management for us. I had previously glanced at cfengine out of curiosity, but ended up spending only a couple of hours looking at it. In my mind configuration management was analogous to unattended software installation, which I was definitely in support of, but had yet to perceive how it was going to change how I viewed infrastructure.

That consulting company was HJK Solutions. Some of my coworkers had previously established relationships with a couple of the partners of HJK, but I didn’t know anything about them myself. I was along for the ride. They gave us a presentation where they showed iClassify and puppet working together to automate infrastructure for other clients, but it wasn’t until the next meeting where we made technical decisions about the implementation that I really came to appreciate their insight. It is much more interesting why someone makes a choice than the choice itself, and this was my first of many since opportunities to incite the opinions of Adam Jacob.

A year of using puppet later, not only was I hooked but my excitement about the possibilities of configuration management had grown beyond what the software could do at the time. Both my excitement and frustration was apparent and got me a sneak peak at Opscode’s Chef. The design of Chef embodies “the unix way” of chaining many tools together insofar that it allows us to take building blocks that are essentially simple on their own but from behind our backs present a system that is revolutionary enough we almost fail to recognize the familiar pieces of it.

Chef is a systems integration framework, built to bring the benefits of configuration management to your entire infrastructure.

This is not an article about Chef, this is about the big picture. However, if you take enough steps back from that statement it becomes apparent that Opscode is building toward that picture. I want to share with you the excitement that short description garners inside of me.

Configuration management alone is the act of programmatically configuring your systems. Often the benefits are conveyed in support of process, but in more agile communities different advantages are touted; such as allowing one to wrangle larger number of servers by reducing build times in the name of vertical scalability, building more maintainable infrastructures by leveraging the self-documenting side-affect of configuration languages, and reducing administrator burnout by cutting a swath in the number of repetitive tasks one must perform. These are unarguably significant boons. Nevertheless, one does not have to look hard to find a curmudgeon reluctant to change, claiming they don’t want to learn another language, that having systems run themselves will surely cause failure, or perhaps some skynet-esque doomsday scenario. History is not short of examples of luddites holding steadfast against new technology, but it is somewhat paradoxical to see this mentality held in such a technologically oriented field.

The recent Configuration Management Panel at the Open Source Bridge conference in Portland amassed many relevant core developers in one city long enough to provide a good vibe for the direction of the available tools and underscore our common charge. But the focus was more about how we will get more users of configuration management tools than why they are going to have to use them. In retrospect, perhaps I should have asked of the panel their views of how configuration management will reshape systems administration.

Configuration management is about more than automation. Some who have foreseen this have started to convey this by discussing managing infrastructures rather than systems. In analogy, the power loom, Gutenberg press, and intermodal shipping container were not merely time saving tools of automation. These inventions reshaped more than their workforce and industry, but also the global economy.

I’m fully aware of the tone set by such a call of prophecy. How will a tool that helps us configure multiple machines at once make such significant ripples in our day to day lives of the future? It will because we will be enabled to solve new problems that we did not yet realize existed. As other technological advances served as a catalyst for globalization, the industrial and scientific revolutions; changing how we build our information infrastructure leaves us poised for an exciting set of challenges that do not yet exist.

I recently upgraded Dell OMSA to 6.0.1 on a number of Ubuntu Intrepid and Jaunty hosts using sara.dl’s packages and got a warning that the mptsas driver version 3.04.07 was below the minimum supported version. The version from ‘modinfo mptsas’ confirmed I was on the right track looking at this driver. A quick look revealed no update in 2.6.29.4 or 2.6.30-rc8, so I went searching for the drivers source.

LSI’s site is terrible. I have Dell 1955 blades, and the Dell SAS5/iR chipsets are really LSI SAS1068s. I searched the drivers page for SAS1068 eventually and found the right download page. I grabbed the 4.18.00 archive file.

After decompressing it I found a dkms folder and rpm. I eventually gave up and used this to build a dkms deb with the following commands:

sudo apt-get install dkms
sudo rpm -i mptlinux-4.18.00.00-1dkms.noarch.rpm --no-deps
sudo dkms mkdeb -m mptlinux -v 4.18.00.00
scp /var/lib/dkms/mptlinux/4.18.00.00/deb/mptlinux-dkms_4.18.00.00_all.deb OTHERHOST:


Then install that deb on the otherhost (with the LSI based chipset) and it will install the correct modules via dkms. I rebooted and used modinfo to verify that mptsas was now version 4.18.00 and ‘omreport storage controller’ now reports ‘Ok’ instead of ‘Degraded’ again.

I’m no fan of software raid. Pretty much, ever. At my last job, for whom I still consult, my predecessor was really into technology creep. All of the workstations used that awesome fake raid that is actually implemented in the mass storage driver and is therefore pretty useless and can actually reduce your paths to recovery from disk failure. I’ll leave out the list of arguments against software raid. It just simply isn’t worth it.

I showed up to a call with a server with an 0×7b error. Of course, Microsoft has this cool feature by default where servers automatically reboot when they blue screen. So nobody knew this was the error until I showed up and tried the “don’t automatically restart on BSOD” option under the F8 startup menu. I’m used to this error from moving system images between hardware, especially with virtual machines. As it turns out, the other values inside the parenthesis are actually useful. If the second value inside the parenthesis is 0×00000010, then you’re likely dealing with a disk in a software raid mirror set (dynamic disk) that Windows has marked as failed, and thus won’t start from.

The trick, which took me a while to nail down, is getting a boot.ini setup to boot from another disk. Since you can’t actually access this partition even in the Recovery Console, you can’t edit the boot.ini to tell it to start from the other disk. In the end, I formated a floppy using simply ‘format A:’ on an XP desktop (would you believe this entire data center lacks a Windows server with a floppy drive?), then copied ntldr, ntdetect.com and boot.ini from another Server 2003 machine with the same service pack to this floppy. Then I changed the boot.ini to contain:

[boot loader]
timeout=60
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=”DISK 0″ /noexecute=optout /fastdetect /3GB
multi(0)disk(0)rdisk(1)partition(1)\WINDOWS=”DISK 1″ /noexecute=optout /fastdetect /3GB
multi(0)disk(0)rdisk(2)partition(1)\WINDOWS=”DISK 2″ /noexecute=optout /fastdetect /3GB
multi(0)disk(0)rdisk(3)partition(1)\WINDOWS=”DISK 3″ /noexecute=optout /fastdetect /3GB


If you’re not familiar with this file, you may want to read about ARC paths. Remember that ntldr and ntdetect.com are hidden, system and read-only by default, although it’s fine to leave this options unset. ‘attrib -s -h -r C:\ntldr’ will make the file accessible so you can copy it to a floppy. I have to assume when you format a floppy from an NT based operating system it puts a bit of code in the bootsector to look for these files.

I then booted from the floppy and for me I then chose ‘DISK 1′ and the system started up fine. I went pulled the failed disk (carefully guessed which disk it was by the disk order in disk management and the scsi id jumper settings) and replaced it. In disk management, right click the good disk, “remove mirror” and choose the missing disk. Then right click again, “add mirror” and choose the new disk. Drink coffee.

It’s late and I can’t figure out how to run ‘fixboot’ and ‘fixmbr’ with a disk mirror, so I’m still using the floppy disk to boot and choose either disk to start from.

Windows XP / 2003 have always been treacherously unstable when moving them between hardware, so much so that fresh installs are wired into my head as being the only option. I’m tired of having three different virtualization platforms, and I don’t want to rebuild these machines. ‘Virtual Server 2005 R2′ is the first to go.

To deal with getting 0×000000ce / 0xce BSOD’s with processr.sys, set both/either of these key/values to ‘4′ in the registry before migrating the machine, or on the first startup:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Processor\Start
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Intelppm\Start

To avoid getting a 0×0000007b / 0×7b BSOD on startup due to the mass storage controller changing, run the registry modifications in MS KB 314082. I save the text inside the “copy here” block to a .reg file and ran it before converting below, and it was enough.

Then use the free VMWare vCenter Converter to convert the Virtual PC image (it has to be off) to a vmware image. On the third step of the wizard, under options, I set the disk controller from automatic to ide. Copy the resulting vmdk file to your KVM host.

Use qemu-img or kvm-img to convert the disk image from vmdk to qcow2:

kvm-img convert -O qcow2 server-ide.vmdk server-ide.qcow2

You can then use KVM to run this disk image. I use libvirt, I simply copied another libvirt xml file, removed the MAC addresses from it, removed the uuid, updated the guest name, and point it to this disk, specifying ‘ide’ as the ‘bus’ in the disk’s target element.

It takes the keyboard and mouse a few moments to work on the console the first time, I assume the devices are being detected in the background. I had one ‘PCI device’ detected without a driver, I let it stay that way. I set the resolution up by hand. On an early run I had issues with the VGA driver, but I can no longer recreate this.

If you were using a static address, you may need to follow the directions in MS KB 269155 to delete the old network interface that is now hidden or you’ll get an error about the address being in use.

Definitely leave a comment about how this works for you. It’s like playing with fire.

Here’s a libvirt xml for kicks:

<domain type='kvm'>
  <name>server</name>
  <memory>786432</memory>
  <currentMemory>786432</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='i686' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/srv/kvm/server/server-ide.qcow2'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <interface type='bridge'>
      <source bridge='br0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'/>
  </devices>
</domain>

The Cisco 3524XL doesn’t support PaGP or LaCP, you simply configure etherchannel by adding ‘port group N’ to each interface. The port group takes the configuration of the first interface in the port group

! Cisco 3524XL
interface FastEthernet0/1
 description uplink to 6509
 port group 1
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

The 6509 supports more dynamic protocols, and will try to use them unless you specify ’switchport nonegotiate’ on the portchannel interface, which is key. Otherwise everytime you turn on ‘channel-group 4 mode on’ the ports will go down on the 3524XL and the ports on the 6509 will go into the ‘err-disabled’ state until you ’shut’ / ‘no shut’ them.

! Cisco 6509
interface GigabitEthernet7/7
 description sw03 - rack 3
 no ip address
 switchport
 switchport mode trunk
 switchport nonegotiate
 channel-group 4 mode on
end

interface Port-channel4
 description sw03 - rack 3
 no ip address
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport nonegotiate
end

It took me a little while to narrow this down. Building a kvm guest with vmbuilder via libvirt I was getting the error “unknown OS type hvm”. When I compared the output of ‘virsh capabilities’ on a good host and the one that wasn’t working, the later was missing the kvm hvm entries. When I checked out the init script for kvm, I realized the the kernel module wasn’t loaded and a quick check of dmesg confirmed that virtualization was disabled in the bios.

Why can’t System Administrators get network design?

Sometime around 1997 I built my first ISP. I was doing computer repair for a man at the time. Internet access was just getting situated in my small city. This man wanted in, but showed up at my house in frustration one night because he couldn’t figure out how to get the router to work. He came sporting a $100 bill and told me it was mine if I fixed it. I suppose it was going to be much more than he had been paying me hourly, but I was more interested in the problem then the pay, and he was frustrated. He had a Livingston Portmaster 2ER, a pile of external modems, and a 56K frame relay uplink to another local ISP. This ISP was always more network gear than computers, because he was “thrifty” mostly, despite owning a computer store. There was an NT 3.5.1 box, a Linux box, and for a little while before it got reappropriated, a FreeBSD machine as well. As fanciness like 56k modems came out and customers grew, hardware scaled out. It remained mostly network hardware.

Ever since then, every network I’ve inherited has been a mess. There have been design ideals focused around age old buzzwords like “security” that results in a pile of expensive security gear that’s essentially useless because proper implementation and design simply wasn’t understood. All of them have grown their L2 infrastructure out horizontally, usually with terribly cheap switches, but often with terrible not so cheap switches as well. Patch Panels and cabling have always run amok, usually with patch cables two to three times longer than necessary stuffed into the cable ducts.

VLANs are almost always used on a single switch, then individual switches are plugged into access ports to provide a switch for every VLAN. Or worse, the switches are all broken up into multiple vlans, with an uplink cable for each VLAN. It’s obvious that concepts like trunking and vtp are simply not understood. These don’t add complexity cost, they simplify what otherwise tends to be a disaster.

I find myself up early lying in bed thinking about the second round of ripping out erroneous unmanaged switches and migrating a live production network to a proper hierarchal design. Suddenly I realized it shouldn’t have to be this way, and really wish more administrators had at least the knowledge of a CCNA. Small companies don’t usual get the benefit of administrators who take the time to understand technology, and usually suffice on consultants who draw a direct line between something functioning and it being right, unfortunately between something not working and it being wrong as well. The latter is almost always because they failed to understand the problem and instead blamed the vendor or technology, from then on spouting that using a SAN creates a SPOF, domain controllers can’t be virtual machines, portable A/C doesn’t actually do anything.

As I trudge through my memory recalling these kinds of misguided attempts at wisdom, they all have a common denominator: not knowing the cause of the problems they are having. You have to understand the technology you’re leveraging. It’s absolutely essential that you know why your network works, not only that it does at the moment.

K2 defaults to adding:

‘Published by btm on April 16, 2009 in Uncategorized’

to posts, which doesn’t include the time, which is sometimes contextually important. This is controlled in ‘theloop.php’ in K2, which uses the date_format, which you can set under ‘Settings -> General’ in the wordpress configuration. The format is the php date format. Simply using ‘r’ is the best, since it provides a nice RFC 2822 formatted date like:

‘Published by btm on Mon, 20 Apr 2009 09:28:48 -0700 in Uncategorized’.