More and more I run into problems with puppet’s DSL. Today a coworker came to me with problems with a munin plugin definition we have. Normally if you want to add a munin plugin that isn’t in the standard base, you use our munin_plugin_file definition, which calls the remotefile definition that simplifies copying files via puppet, and also calls the munin_plugin definition which essentially makes the symlink to enable the plugin.
Today we wanted to do this with wildcard plugins, but more than one call to munin_plugin_file would fail, because the remotefile would get defined multiple times and puppet can’t handle that.
err: Could not retrieve catalog: Puppet::Parser::AST::Resource failed with error ArgumentError: Duplicate definition: Remotefile[munin-plugin-slapd_] is already defined in file /etc/puppet/site-modules/munin/manifests/definitions/munin_plugin_file.pp at line 10; cannot redefine at /etc/puppet/site-modules/munin/manifests/definitions/munin_plugin_file.pp:10 on node
The solution is to use puppet’s immature conditionals to test against if the type was already defined and not redfine it.
define munin_plugin_file($plugin_config = “/etc/munin/plugins”, $plugin_dir = “/usr/share/munin/plugins”, $plugin) {
if defined(Remotefile[“munin-plugin-$plugin”]) {
debug (“$munin-plugin-$plugin already defined”)
} else {
remotefile { “munin-plugin-$plugin”:
path => “$plugin_dir/$plugin”,
module => “munin”,
source => “plugins/$plugin”,
owner => root,
group => root,
mode => 755,
require => Package[“munin-node”]
}
}
munin_plugin { $name:
plugin_config => $plugin_config,
plugin_dir => $plugin_dir,
plugin => $plugin,
require => Remotefile[“munin-plugin-$plugin”]
}
}
Note that the debug line is there because puppet conditionals can’t have empty blocks, see bug #1109 (tracker is down now, I’m guessing at that link).
I’m really wondering because I’ve had these sorts of problems twice now today. Normally it’s every once in a while. In shorter form:
Bryan Mclellan [10:59 AM]:
production-sites includes apache::module::php4, which includes the package, and runs apache_module. i wanted the php4-ldap package, which the php4 class installs. so I added an include for php4 in the production-sites.
but php4 also installs the apache2 php4 module, so there was a naming conflict.
so I removed the package from apache::module::php4 and added an include to php4 there, but it simply wouldn’t do the include. perhaps too many levels deep.
You have to put a lot of thought into your design if it’s going to scale. Especially when you put everything in puppet like we do. Someone told me recently that our puppet code base was much larger than most.
~/puppet$ find site-modules/ -name ‘*.pp’ -exec cat ‘{}’ \; | wc
4166 10820 101647
~/puppet$ find site-modules/ -name ‘*.erb’ -exec cat ‘{}’ \; | wc
3565 12773 112231
$ grep -R class site-modules/ | wc
152 578 12264
modules and site-modules have a lot of overlap. As others are picking up puppet, I wonder how long it takes them until they start running into this. Of course, if you avoid nesting definitions, and keep all of your classes separate, you won’t see this. But you’re doing a lot of work too.
Hi, I found your blog on this new directory of WordPress Blogs at blackhatbootcamp.com/listofwordpressblogs. I dont know how your blog came up, must have been a typo, i duno. Anyways, I just clicked it and here I am. Your blog looks good. Have a nice day. James.
Totally agree about putting a lot of thought into the puppet design.
As you start setting up your classes and manifests, you get more comfortable with puppet. Then, you create the second node that needs your class, and you realize that what worked great for one node, scales horribly to multiple nodes. So, you fix that with templates.
After about three or four apps, you realize all these files and templates are getting blurred together, so you migrate to plugins/modules, and you think you’ve solved all the worlds problems.
Then, you hit a design snag, or you discover some fantastic way to clean up your original classes or definitions. Unfortunately, you realize that your original design won’t fly anymore, and you hit that shocker moment when you realize “crap, this won’t work, and it’s going to be a pain to go back and fix.”
Tricky part, of course, is going back and fixing all the old cruft left behind. I wish there were some way to figure out if a class is even being used anymore.
So, yes, it’s an evolving process for us. We’ve come a lot way with our design, but there’s always that application we haven’t “puppified” yet that will probably create more challenges.
Why not use virtual resources for your custom munin plugin files?
@Judd,
I’m not sure how that would be any easier. You’re saying make the file type virtual, and realize it in the definition? Sure that works (in theory), but the above example isn’t a lot of code, it’s more a silly hack to deal with how things are scaling (or not). Using a virtual resource might save 2 of the four lines of code added by that conditional.
I recently came back to puppet after having ignored it for a few months (although having it happily chugging along in the background keeping my machines in sync).
And right off the bat I hit this one again… duplicate defines. They are irritating, and especially in the case of package defines that are often exactly the same anyway. One solution to that part of the problem would be to just ignore defined resources that are *exactly* the same. Just fold them together, as there actually *is* *no* conflict if they are defined the same. If they aren’t the same, you *should* error out, of course, because you apparently are trying to do different things with the same resource. This, I think, is a bug in puppet at the moment as it always errors out with duplicate defined resources.
The problem with the conditional is that it is haphazard. I’m currently actually looking at modifying all my modules to use virtual resources. One way is to make all ‘general’ resources virtual (and actually using the same kind of conditional you use in your example, because I don’t want to mess about with checking in what module I define what virtual resource either), and then realise them at the appropriate places. I’m not liking this very much, though, because it just adds a lot of cruft, and you can just do the same without using the virtuals.
The other I’m looking at is to not define *any* general resources in the modules themselves, but to put them in a general module called ‘virtual’. This virtual module is then added to all the modules’ search paths, making the virtual resources realizable wherever I want, and all resource definitions are unique and only happen once, and centralised. Not sure if this is a win either, but I’m still looking.
Wonder what others have done to solve this…
@Thijs,
I centralized all my user/group resources into a user module, and that makes sense because when dealing with UID/GID assignment, it is easiest for me to see the resources sorted numerically by UID/GID.
I shudder thinking about doing the same thing with packages though. The module you’re writing is still self documenting, because you see the package realized, but the attributes of the resource become disconnected. I think of enable/ensure, and we do some seeding as well (debian).
The awesome thing about puppet is that you can write a recipe like:
Install this package
Install this config file
Add this user
Run this service
When you have to start dealing with many workarounds because the breadth of your usage makes you rewrite simple things into complex ones to deal with the languages limitations, that sucks.
Modules are good because they allow code reuse and everything, but if I want to write a simple module to install piece of software and configure it, but when a five minute task turns into thirty because another module uses the same package for something completely different; something is a afoul.
Most of the talk of the capacity of puppet is around how many nodes it can scale to. There is much less discussion (that I’ve seen on the internet at least) about when the size of your puppet recipes gets unwieldy. It’s a conversation I find useful and pretty interesting, having worked around it in a couple ways.
http://reductivelabs.com/trac/puppet/wiki/PuppetBestPractice
I agree.
Working on this yesterday I found an ugly hack to make it possible to create virtual resources from within a module that are accessible everywhere. But as you say, the disconnect between the attributes and the resource is unwelcome.
The downside to checking if the resource is defined, and if not, define it, is that you are basically blindly using a resource that is defined elsewhere if that exists, with unknown and possibly completely different attributes than those you were planning to use. The debug message actually helps to alert you to this, but do you always go through your logging to check for things like that? The possibility exists that you miss it and incorporate a resource that differs from what you think you are using. Not a good thing.
At least with virtual resources, you *know* you have to check elsewhere to see what it is you’re getting.
@Thijs,
You’re totally right. I wouldn’t use ‘if defined()’ for package for those reasons. It works in that example because the only place I ever use that remotefile resource is in that definition. I can’t move the resource out of that definition and make it virtual, because I need the values being passed to the definition to construct the resource definition.
The solution in other places, like apache modules, has been to write a class for every module and include it. This could be done with munin plugins, but there’s many of them and they’re relatively small. It’s frustrating and a waste of time to have to write all of that out:
class Munin::Plugin::Foo { remotefile { "foo": # stuff } munin_plugin { "foo:" # stuff } }When you could just write a definition and have it be a one liner every time you need another munin plugin.
The debug is mostly there because of bug #1109 that didn’t allow you to have the first block of an if/else empty. Since the if conditional is immature, you can’t do any sort of negation test, requiring you to use else if you want to test for something to be false.
Yeah, I see what you mean. I’m now actually leaning towards moving potentially conflicting packages (and other resources) into their own separate small classes and then just include the class anywhere I want. Not ideal either.
On the if-problem: I use a case for that case, as in:
case defined( Package[bla] ) {
false: {
# do stuff
}
}
Pingback: configuration management with chef announced at btm.geek