The Chef site starts out with, “Chef is a systems integration framework, built to bring the benefits of configuration management to your entire infrastructure.” There is an important point hidden in that description; Chef is not a CM tool. Of course it can be used as one, and many do, but from its beginning it has been leveraged by others such as Engine Yard inside of their own infrastructure. You can safely bet it will be an integral part of the Opscode platform when released as well.
While I was off dealing with my startup’s acquisition this fall, a chef user wrote knife. It was a command line client for interacting with a chef server. An impressively simple prototype, made possible and powered by the chef libraries and API. This has happened before with chef; for instance a while ago in the 0.6 era, an OS X GUI client called Casserole was written by another chef user with an itch to scratch. However, something has happened with knife that is interesting enough I’d like to specifically point out; it got mainlined and heavily expanded.
This happened for a handful of reasons. For one, it was simply a great idea. The kind of user who would be attracted to chef as a tool is very likely to be a power user who would rather not spend their time clicking around a graphical interface. It’s much easier to script a command line tool where needed, passing data in and out for quick hacks to your infrastructure. The folks at Opscode saw this, agreed, and set out to add full functionality to it for the upcoming 0.8 release.
What I think is most important is the planning a full API from the start. From hearing Adam talk about other tools being “first-class citizens” in chef-land, and knowing his experience writing iClassify as an early open source external node tool for puppet, I know this design was intentional. Using iClassify to tell puppet about your nodes was great, but puppet assumed that you only wanted this tool to classify nodes in the way puppet thought about nodes. Consequentially, when you wanted to to use data in iClassify about your nodes to make decisions about your infrastructure on the fly, you were forced to do it in templates. This created the expected repetition of loading the iClassify library and accessing iClassify in many templates, but also required you at times to do some fancy footwork to get data between templates when you really wanted puppet to know about the data itself.
Reductive Labs recently announced a dashboard for puppet. I was hoping this meant those barriers had been removed. It certainly creates really nice graphs from your puppet report data. However from the README it looks like you’re still pushing limited data into puppet using the external node interface. Reductive is going to have to expand this interface greatly if dashboard is to have any meaningful node integration benefits that we didn’t already have two years ago with iClassify.
Just as you can see some concepts from other configuration management tools in chef, you can see parts of iClassify. It was a great start and it was disappointing that the puppet community didn’t engage it further. Perhaps it was simply before its time, but I believe it was that there were too few doors into puppet-land to let you really take advantage of and grow external tools.
I think this was the lesson that Opscode learned, and consequently chef was born with an API. With it we can accomplish nearly anything we dream up. What is most exciting about this is that we can do whatever everyone else dreams up. I can’t wait to see what that is.
Hi,
Thanks for the interesting pointer on this development in the Chef world.
It’s worth noting, to start, that our dashboard is following a “minimum viable product” model – we’re releasing as soon as we have something that’s functional enough to be useful, so don’t assume that missing functionality will always be missing.
There are multiple reasons why Puppet didn’t adopt the iClassify way of thinking about the world, but none of them relate to missing APIs. Puppet has tons of extension points, and we’re adding more all the time. We don’t do the best job of documenting and advertising them, but it’s not like they’re hidden, either – I meet people every month who are doing interesting things with them.
That being said, we are adding new ways of exchanging rich data. I never liked the way iClassify builds APIs on raw data, expecting the consumers of that data to add all of the necessary bits. I much prefer exchanging richer data – transfer resources instead of simple hashes, for instance.
Anyway, don’t worry; we’ll start copying the Opscode guys, too – information exchange won’t all be from us to them. 🙂
Hmm, I should have looked more closely at Knife before posting – it’s basically a CLI to the GUI stuff, whereas Puppet starts with the CLI and is slowly adding the GUI. You can bet that before too long you’ll be able to do just about anything in our Dashboard that you can do in the GUI, and it’s got little if anything to do with, say, iClassify. Our 0.25 release laid the groundwork for this integration, and the next few releases of the Dashboard will begin to see the benefits.
Knife is a consumer of the Chef REST API, just like the Chef Web UI is – and the API came first. 🙂
As Adam chimed in with, from a chicken or the egg point of view, knife is adding the ability to change information you could already access via the Chef WebUI in earlier versions. In Chef 0.8, they both speak to the chef-server entirely via a REST API, which is of course indifferent to how the data is being displayed.
The argument for passing raw data instead of resources is that you limit what one can do with the data to what you expect and allow them to, in the latter. When I spoke of the API limitations I was specifically speaking of the external node interface, and how it made it difficult to access data about nodes in modules, compile that data and use it to build content in configurations.
I think your reasoning makes for an interface that is easier for people with less development experience to pick up and manipulate on their own, but eventually that design has always eventually formed roadblocks that frustrated me personally.
Ultimately this post was the tools that users have developed, extending by scratching their own itch. James Turnbull commented on twitter as well that this is an early release of the dashboard and there is much more to come, even going so far as to file an issue for the feature I always wanted.
Perhaps this will always be a defining difference between configuration management tools and libraries. Which is great of course, as the diversity will bring more users into the fold realizing the absurdity of configuration by ssh-for-loop. With more users I believe we will benefit with more innovation from the additional mindshare.
I definitely agree that the external node interface is limited, but that was essentially its goal – be a limited means of passing specific data around.
The internal work in 0.25 to add a framework for passing objects around, including serialized objects over the wire, generalizes that concept to the point where it’ll be easy to add other interfaces for other resource types, and, in fact, we’ve already got plenty of them (e.g., certs and catalogs are exposed via a REST api).
The Dashboard uses that same API, as does the client. Most of the executables don’t use a network API but rather the internal Ruby API, but the network is certainly there and able to be manipulated. Even better, because the Ruby and Network API go through the same framework, you’re guaranteed that anything that can be done in Ruby can also be done over the network.
I don’t think the structured, resource-like data is any more limited than raw data – if anything, because it’s got built-in semantics you’ve got *more* power, not less. It’s true that I don’t ever want to given an SQL-like interface that just lets anyone do anything, but I don’t think anyone’s proposing that here.
In 0.25, Puppet shipped with a REST api for essentially everything that matters to it — catalogs, files, certificates, and more — and we use it internally everywhere. We didn’t finish a couple of small things, but this API is certainly at the heart of everything we do here.
And, of course, before we had a REST API we had an XMLRPC API, which is no less an api for all that it used less cool tech. But then, that’s one of the downsides of having a project that’s 4.5 years old – we used tech that was best of breed in 2005, rather than 2008. If you want to write a tool that talks to Puppet, you can. If you wanted to write a tool that talked to Puppet in 2005, you could have.
Chef was developed so recently that it was able to take advantage of a huge renaissance in web services; we can’t all be so lucky.
So there’s no library/tool split here, there’s just a philosophical difference about how libraries and services should work. Knife is a great example someone scratching their own itch with Chef, but then, iClassify and Foreman and all of the report processors out there are similar examples with Puppet.
Either way it’s clear neither of us has had our last word on this – the future is clearly APIs and integration all around, and it’ll be interesting to see how each ecosystem develops over time.
Pingback: The Chef Way « Kallistec