Category Archives: Operations

Using for loops in Chef

Leave a reply

One of the great features of Chef is that you write configurations in Ruby. When wanting to push a number of configuration files out for nagios, I initially turned to the Remote Directory resource. However this could interfere with configuration files created and owned by the debian package, so I needed to be more specific. In the past with Puppet I’ve had a remote file definition (that uses the file type) for each configuration file. This works fine, but gets to be repetitive when it doesn’t need to be. With Chef, you can combine a little ruby with the Remote File resource like so:

for config in [ "contacts.cfg", "contactgroups.cfg" ] do
  remote_file "/etc/nagios3/#{config}" do
    source "#{config}"
    owner "root"
    group "root"
    mode 0644
    notifies :restart, resources(:service => "nagios"), :delayed
  end
end

The benefit of this approach is that it makes your configuration management cleaner and more DRY. This is perhaps at the cost of a little complexity, albeit at a degree that I think is pretty easily understood by reading the code.

Paravirtualized disks with KVM

1 Reply

In the midst of Jaunty testing, I decided to try out paravirtualized disks with KVM. I switched to virtio for networking a while ago with good results. I already built the guest I’m testing with, so I wanted to modify the libvirt xml file, but didn’t see anything of note related to virtio and storage. I did however deduce that the trick is to change the bus attribute in the target element from ‘ide’ to ‘virtio’. Because Ubuntu uses UUID’s instead of paths, the change from ‘sda’ to ‘vda’ didn’t affect startup. I was confused at first though as mount still showed ‘/dev/sda1’ but the dmesg output clearly lacked an ‘sd’ device but had a newly acquired ‘vd’ device.

Bonnie++ was run on a single cpu guest with 768MB ram. The guest is Ubuntu Jaunty 9.04 with all of todays packages, using virtio for network + disk. Otherwise pretty standard with everything that might matter. The Host is also Jaunty, running on a Dell 1955 blade with a couple Xeons and about 7GB of RAM. KVM is ‘1:84+dfsg-0ubuntu7’, and libvirt is ‘0.6.1-0ubuntu1’.

The numbers aren’t all that interesting. Virtio was a little bit faster. I’m not that familiar with using bonnie. Who knows if caching or anything negated these tests, I didn’t try to research turning it off. The performance testing was mostly an afterthought of setting it up, as I see no reason not to use it now.

With ide disk driver:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
it-util01     1496M 37162  94 56108  20 39997  14 42209  89 247590  59  4706  93
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
it-util01,1496M,37162,94,56108,20,39997,14,42209,89,247590,59,4705.6,93,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

With virtio disk driver:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
it-util01     1496M 39831  88 56480  13 41427  14 45489  86 291109  57  7915  90
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
it-util01,1496M,39831,88,56480,13,41427,14,45489,86,291109,57,7914.5,90,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

CouchDB and binary attachments

1 Reply

After a couple of the couchdb developers commented on my earlier post about binary data and couchdb, I took their advice in the next round of testing. I upgraded to CouchDB pre-0.9.0 from svn, then wrote read/write tests for storing data using byte[], base64 String, and byte[] via attachment. The updated code is available in the same github gist. These tests were not scientific. I ran each combination of data type over n threads for 100 iterations and compared the total times. I averaged the results when using more than one thread for the total. Consider the data completely empirical, but the relationship stands. The binary attachments are fast. That’s all we wanted to know.

So fast even I can’t get consistent numbers and the java libraries throw “httpMethodBase.getResponseBody – Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.” I left the ‘attach read’ out of the graph to keep the numbers on a closer scale.

Writing binary data to CouchDB

2 Replies

I’m doing some performance testing with CouchDB and jcouchdb and I wanted to know if I should write binary data using a bytearray or as a base64 encoded string. The latter is definitely the correct answer. I initially tried using couchdb4j, but I found that it’s exception handling is flawed, or well, doesn’t exist. So I dropped that after a day of tinkering with it. I’ve since been writing a performance testing tool in Java to reuse some of the code in a Java product we have when I’m satisfied with the results. You can find the source that produced these numbers on github for now. I’ve got some more tests to add, and will spend some time thinking about where to put the final tool.

I’m using couchdb 0.8.0-1 as installed out of the box on Ubuntu 8.10 from the package. The graph on the left (which I quickly made in OO and is terrible) is the result of four total runs. Each run was ten threads each writing one hundred documents. The first two runs are writing a binary array and then a base64 encoded string of an 88k image, then again with a 9.5k image. The base64 runs include the time it took to encode the file, but the binary array runs took three times longer to complete. Futon also hates displaying the binary array data.

I’ll be adding another method to test reads since that’s what we’ll be doing primarily. I want to test the concurrency on the reads, then compare those numbers to the results of running multiple couchdb nods behind nginx to ensure the overhead is low and performance really increases. I know Tim Dysinger has been doing some testing and that he and other folks from #couchdb on irc.freenode.net are going to test some pretty large clusters, so it will be interesting to see how our numbers compare.

The number of threads changes the results quite a bit. Tuning may make significant difference or none at all. The one hundred iterations of the 9.5k image takes

number of threads:[base64 seconds, bytearray seconds]
1:[2,5]
5:[8, 21]
20:[34, 90]

I’ll let make another post next week after more testing is done.

Chef 0.5.2 and Ohai 0.1.4 released

Leave a reply

We contributed a lot of work to the latest Chef release, which made it out over the weekend. Most notably we got a lot of FreeBSD support in, and it looks like a few people are going to give that shot. The release notes are the best source of information about what was added. As we’re moving puppet recipes over to chef we stumble across pieces of configuration that we’d rather be a resource, and try to add that support. We’re really excited about what we’re getting into Ohai. I tested support that Ben Black is working on for reading libvirt data through their ruby API, and it’s just going to be awesome. With puppet+iclassify I had some convoluted ways of getting guest information, but this implementation is going to be first class enterprise stuff.

Building scalable operations infrastructure with OSS

Leave a reply

I’m the lead systems administrator at Widemile and run operations here. Sometimes I do other things, but operations is most interesting. My linkedin profile may give you an idea of where I’m coming from, but it misses all those late nights working with OSS because I enjoy it. I usually blog on my own site, but it often serves more as a technical journal than what we are up to at Widemile, which will be the differentiator. As a rule, I’m not a developer. Certain facts may convince you otherwise, but I try to stay out of our product. You’ll start to hear from Peter Crossley , our lead software architect, soon enough. Perhaps some other developers after that. I’ll christen this blog with a round-up of how we’re building our infrastructure at Widemile.

Most recently I’ve been in heavy development on Chef and Ohai. We’ve been using puppet for about a year and a half now. Check out my Infrastructure Engineering slide deck for where we were with that a few months ago. I’ve been happy with it except for a few issues which ended up being mostly major architectural changes to fix. Adam at Opscode has a good post entitled 9 things to like about Chef, that outlines some of these difference. There’s a lot of e-drama around Opscode’s decision to write a new product rather than usher changes into puppet. I won’t touch that other than to say that we had problems with puppet that chef fixes.

Almost all of our servers are in configuration management. Which means that no one-off work is done on the production servers so that all changes are self-documenting. Granted, we’re a small shop and sometimes I’ll do minor debugging on a production server, but any changes do end up in CM.

Our servers are almost all kvm guests under libvirt running on Dell blades. There’s some information about how we got here in a slidedeck I made for GSLUG entitled Virtual Infrastructure. Apparently using kvm in production isn’t all that heard of, but since we’re a small shop we’re able to leverage new technology very quickly to make the best of it. With the use of vmbuilder, libvirt, kvm and capistrano, we build out new nodes in a matter of minutes. More importantly, it’s just a couple commands.

Once Chef is intergrated with the libvirt API we expect to be able to further simplify our deployment. The idea behing is that it will be a ghetto version of Solo, which EY built using Chef. Eventually we’ll pull out capistrano. While it’s nice for interacting with multiple machines at once, it really was written for a different purpose than what we use it for. There will be replacement functionality in Chef shortly.