Why can’t System Administrators get network design?

Sometime around 1997 I built my first ISP. I was doing computer repair for a man at the time. Internet access was just getting situated in my small city. This man wanted in, but showed up at my house in frustration one night because he couldn’t figure out how to get the router to work. He came sporting a $100 bill and told me it was mine if I fixed it. I suppose it was going to be much more than he had been paying me hourly, but I was more interested in the problem then the pay, and he was frustrated. He had a Livingston Portmaster 2ER, a pile of external modems, and a 56K frame relay uplink to another local ISP. This ISP was always more network gear than computers, because he was “thrifty” mostly, despite owning a computer store. There was an NT 3.5.1 box, a Linux box, and for a little while before it got reappropriated, a FreeBSD machine as well. As fanciness like 56k modems came out and customers grew, hardware scaled out. It remained mostly network hardware.

Ever since then, every network I’ve inherited has been a mess. There have been design ideals focused around age old buzzwords like “security” that results in a pile of expensive security gear that’s essentially useless because proper implementation and design simply wasn’t understood. All of them have grown their L2 infrastructure out horizontally, usually with terribly cheap switches, but often with terrible not so cheap switches as well. Patch Panels and cabling have always run amok, usually with patch cables two to three times longer than necessary stuffed into the cable ducts.

VLANs are almost always used on a single switch, then individual switches are plugged into access ports to provide a switch for every VLAN. Or worse, the switches are all broken up into multiple vlans, with an uplink cable for each VLAN. It’s obvious that concepts like trunking and vtp are simply not understood. These don’t add complexity cost, they simplify what otherwise tends to be a disaster.

I find myself up early lying in bed thinking about the second round of ripping out erroneous unmanaged switches and migrating a live production network to a proper hierarchal design. Suddenly I realized it shouldn’t have to be this way, and really wish more administrators had at least the knowledge of a CCNA. Small companies don’t usual get the benefit of administrators who take the time to understand technology, and usually suffice on consultants who draw a direct line between something functioning and it being right, unfortunately between something not working and it being wrong as well. The latter is almost always because they failed to understand the problem and instead blamed the vendor or technology, from then on spouting that using a SAN creates a SPOF, domain controllers can’t be virtual machines, portable A/C doesn’t actually do anything.

As I trudge through my memory recalling these kinds of misguided attempts at wisdom, they all have a common denominator: not knowing the cause of the problems they are having. You have to understand the technology you’re leveraging. It’s absolutely essential that you know why your network works, not only that it does at the moment.

K2 defaults to adding:

‘Published by btm on April 16, 2009 in Uncategorized’

to posts, which doesn’t include the time, which is sometimes contextually important. This is controlled in ‘theloop.php’ in K2, which uses the date_format, which you can set under ‘Settings -> General’ in the wordpress configuration. The format is the php date format. Simply using ‘r’ is the best, since it provides a nice RFC 2822 formatted date like:

‘Published by btm on Mon, 20 Apr 2009 09:28:48 -0700 in Uncategorized’.

It recently clicked in my head that all the blades with small swap partitions were because they had their OS installed when they had very little RAM in them. So I set out to modify the Ubuntu 8.10 preseed install to create a larger swap partition and configure LVM while we were at it.

This proved difficult. Mostly because the better documentation of debian-installer (preseed, partman-auto) has features that aren’t in the version in Ubuntu.

Just got this working:

d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true 
d-i partman-lvm/device_remove_lvm_span boolean true
d-i partman-auto/purge_lvm_from_device  boolean true
d-i partman-auto-lvm/new_vg_name string system
#d-i partman-auto/init_automatically_partition \
#  select Guided - use entire disk and set up LVM
d-i partman-auto/expert_recipe string                         \
      boot-root ::                                            \
              40 300 300 ext3                                 \
                      $primary{ }                             \
                      $bootable{ }                            \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ /boot }                     \
              .                                               \
              2000 10000 1000000000 ext3                      \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ / }                         \
              .                                               \
              8000 8000 200% linux-swap                       \
                      $lvmok{ }                               \
                      method{ swap } format{ }                \
              .

d-i partman-lvm/confirm boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select Finish partitioning and write changes to disk
d-i partman/confirm boolean true

It took me a little while to put the pieces together to figure out how to take a vm-builder created vm and use briding with it instead of kvm/qemu’s user-mode networking. All the pieces are available on the internet, but there was some emphasis lacking to make it all clear to me.

You’ll need to have a bridge set up on your host. Install the ‘bridge-utils’ package first. Then the relevant section of my /etc/network/interfaces file looks like:

# The primary network interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE up

auto br0
iface br0 inet static
address 10.0.0.60
netmask 255.255.255.0
gateway 10.0.0.1
bridge_ports eth0
bridge_stp off
bridge_maxwait 0
bridge_fd 0
bridge_hello 0

You could probably use ‘dhcp’ instead of a ‘static’ address on the bridge. The point is that your ipv4 address should be on the bridge, not on the actual interface.

Then create a ‘br-ifup’ script in your vm directory. This is based on /etc/qemu-ifup. This script is passed the name of the interface (tap0) which brings the interface up, and then adds it to your bridge.
#!/bin/sh
#sudo -p "Password for $0:" /sbin/ifconfig $1 172.20.0.1
sudo /sbin/ifconfig $1 up
sudo /usr/sbin/brctl addif br0 $1


Then run kvm with something like this:

sudo kvm -m 128 -smp 1 -drive file=disk0.qcow2 -net nic -net tap,script=br-ifup

‘/etc/kvm-ifup: could not launch network script’ means that the script passed in ‘script=’ could not be found.

‘Could not initialize device ‘tap” means that kvm is unable to create the TAP/TUN interface. Running kvm as root via sudo is the easy solution.

‘warning: could not open /dev/net/tun: no virtual network emulation’ probably means that the ‘tun’ module isn’t loaded. You can load it with ‘sudo modprobe tun’.

The tap interface is removed from the bridge when the guest is shutdown.