[Puppet-dev] Re: your mail

[Luke Kanies]

On Sat, 1 Apr 2006, leakin wrote:

> Luke,
>  I have been attempting to implement cfengine yet again, and still
> finding deficiencies.  I decided to search again for alterntatives and I
> found puppet.  I am very excited about the possibilities, and curious
> about the direction of development.

Great.  (I'm CC'ing my response to the puppet-dev list because it ended up
being pretty long and I figured people might appreciate reading it.)

> I'd like to propose a couple of ideas for your consideration.  First the
> simple one, multiple servers.  I want to provide a list of servers the
> clients can pull from, and have them use the first that responds.  Some
> machines are behind firewalls, and will only reach a mirror inside their
> network segment.  The list would also help if the primary server is down
> for any extended length of time.  The machines we admin are in a mixed
> environment where others may also use puppet at some point, so DNS lookup
> of the puppet host would cause collisions.  I could extend puppet myself,
> but this is likely to be something others want/need as well.

I've been mostly operating on the assumption that people who have multiple
servers will put them behind a load balancer, but it probably does make
sense to support some level of failover directly within Puppet.

As to name collisions with the name 'puppet', it's easy enough to change the
name of the server you pull from.  I just wanted to provide a simple default
so that for normal environments there would be no configuration on the
client.

> The second is more difficult in that it would likely require major code
> changes, and I suggest it while procrastinating on a major change in a
> piece of my own code (I've actually learned Ruby while delaying a
> re-structure/re-write of my File::Rsync perl module ;).
>
> Have you considered using the Ruby interpreter itself to parse the
> configuration file?  It might solve some of the issues with scope and
> conditionals, and would provide virtually unlimited expansion and hacking
> abilities to the admin.  For the basic/beginner, the Ruby part could be
> mostly hidden by the use of constants, globals, and/or methods.  For the
> more advanced, mentioning the fact in the docs would be enough to trigger
> all sorts of ideas.

I considered this early on, and in fact just about everyone who comes across
Puppet suggests this to me, and I still think it makes sense to have a
simple language front end with its own Ruby-independent grammar and parser.

I don't have a pat list of my justifications, but I'll try to summarize why
I'm doing it this way.  The most important point is that Puppet is supposed
to represent a clean abstraction layer between the system and the
administrator -- if the admin has fully Ruby access to the system, then
complete abstraction is not possible.  If Puppet developers are not forced
to separate the function (in Ruby) from the interface (in Puppet), then they
will tend to get lazy and muck them together.  Once that starts happening,
people write modules that work fine for them but not for anyone else, and
suddenly Puppet isn't an abstraction layer any more.

Also, the parsing of a Puppet configuration takes place on the server, while
the action obviously takes place on the client.  If users can write Ruby in
the configuration, they're going to want to be able to actually modify the
system with that Ruby, which means that I'd have to have a far more
complicated transportable format.

With my own domain-specific language, I control the interfaces, I can
control the transportable format (which is currently insanely simple, and
I'd like to keep it that way), and I can make sure that modules on the
backend are cleanly separated from the front-end.  Here are some other
reasons for writing my own language:

     * First and foremost, I am trying to create a full abstraction layer
       above the operating system; this is quite a bit more than just a
       new configuration file format.

     * I don't want to tie Puppet to Ruby indefinitely -- I expect to port
       some aspects of it to other languages at some point (e.g., rewrite the
       library in C so any language can use it).

     * I want to make it simple to provide other ingresses into the Puppet
       system, and a simple declarative language is very easy to map to a GUI
       or a web front end.

     * While many sysadmins know some scripting, very few of them seem
       willing to really invest much in their scripting ability, so it makes
       sense to provide a much smaller and simpler language.

     * It really actually does make sense to create a domain-specific
       language for this problem space, and it's actually not that hard.
       Certainly I've had far more trouble defining the grammar (which I'd
       have to do even if Ruby parsed it directly) and writing the library.

     * It is quite difficult to use imperative languages like Ruby to provide
       declarative behaviour.  Although Puppet has not succeeded in being
       fully declarative, it would basically be impossible to be completely
       declarative if users had full access to an imperative parser.

To summarize, Puppet has very different needs than Ruby, and providing full
access to Ruby puts too much unnecessary complexity in the configuration
file while limiting the future of the system by unnecessarily tying it to
Ruby.

> I have a simple program a number of people use, that has only required a
> few updates/enhancements because the configuration file is 'require'd into
> the main code and returns a perl hash-of-hashes.  It has allowed complex
> decisions and extensions in the config file where needed rather than
> hacking it into the main code.  I have another program where I've fought
> with various formats and parsers for the config, and always come up short.
> (That may be from ignorance, inexperience, and/or poor programming, but
> I spend an inappropriate amount of time extending the configuration when
> I'd prefer to improve the core).  I suspect config files and their
> human-to-program interface problems may be as complex a problem as the
> system administration tasks that cfengine and puppet attempt to solve.

While I admit that Puppet's language is not currently as expressive as it
could be, there are significant benefits to the simplicity of the language
that could not be achieved as easily through a full parser.

> I am sure the success of Emacs is due to lisp access to enhance,
> customize, and extend the editor (and I'm a vim user).  If the puppet
> config were valid Ruby code, it would free the admin in a similar way.
> I believe the reason big/comercial admin tools often lose to adhoc scripts
> is this lack of flexibility, and allowing Ruby to parse the config would
> go a long way towards combining a sophisticated cfengine-type tool with
> adhoc scripting.

I don't actually want to provide ad-hoc scripting, though; in fact, I
specifically want to forbid it.  Puppet should be a clean interface between
what you want and what gets done.  Once people can do ad-hoc scripting in
the config files, those config files are no longer portable, which pretty
much ruins it.

Also, notice how few sysadmins use Emacs -- while sysadmins who program tend
to assume that all sysadmins program, this is clearly not the case, and even
those who do program tend to do so in very limited ways.

> This is rather long for an introductory message, but I hope I've drawn a
> clear picture of my idea, and you'll consider the possibilities.

Yes; I thank you very much for the feedback, and I hope you'll consider
using Puppet even though it doesn't support full Ruby on the front end.  I
am working on some templating projects and other mechanisms to provide the
power you want without breaking some of the rules within Puppet, but I'm
adding power slowly.

-- 
Honest criticism is hard to take, particularly from a relative, a
friend, an acquaintance, or a stranger.      -- Franklin P. Jones
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com