[Puppet-dev] aggregating and a list syntax

[Luke Kanies]

Hi all,

Comments on this email are highly appreciated -- I'd really like to come up
with a solution to this soon, since I think it's very compelling and would
be a heavily used feature.

On IRC, we have discussed something we're calling "aggregation", which is
collecting a set of elements that any machine might have specified and applying
them to the machine making the 'aggregate' call.

There are many use cases, but the three that keep coming up are SSH public
keys (every machine has a public key, and each machine needs a copy of every
other machine's public key in a single ssh_known_hosts file), firewall ports
(publicly accessible servers need to be able to specify that they need holes
in the firewall), and DHCP addresses (individual machines need to be able to
specify that they need an IP).

In each of these cases, a set of nodes are contacting the central server and
compiling their configurations, but these specific instances are not being
sent to them -- that is, a web server would not receive a firewall port
configuration, it would only compile one on the server.  The role of
aggregation is to get those compiled specifications onto the right machine.

This is much more complicated to explain than it seems that it should be. :/

There are two main concepts that we want to be able to specify in
aggregation:  The element type being aggregated (e.g., sshkey, dynamicip)
and any criteria used for selection (e.g., only hosts on this LAN).  We
might consider class membership (only sshkey elements defined on a host in
the Nashville data center) to be a different type of criteria, but it'd be
preferable if it were the same.

So, there are something like three potential syntaxes bouncing around.
First is the too-simple:

    aggregate sshkey

Just list the element type, with no ability to specify different criteria.
This has the huge benefit of being simple for me to implement.

Then David Lutterkort mentioned that he thinks of this aggregation need as
being similar to the existing search syntax in requirements and such:

    service { nagios:
        require => file["/etc/nagios.conf"]
    }

He mentioned expanding that syntax so that it maybe supported arbitrary
criteria:

    service { nagios:
        require => file[name =~ "/etc/nagios"]
    }

While I like the idea of having a general lookup syntax, it confuses the
current syntax (which only supports a single string between the brackets),
and it just feels wrong.  Frankly, though, the use of square brackets at all
still feels wrong (they're a hold-over from the early Puppet syntax, which I
won't abuse you with).

So, I'm thinking of making a distinction:  When used as an lvalue (that is,
a statement in its own right), the '<type> { <stuff> }' syntax is used for
declaration and can only have '=>' separators, but when used as an rvalue
(that is, when used as the argument to an assignment or function), the same
syntax is used for querying, and would support operators instead of '=>':

    service { nagios:
        require => file { name =~ %r{/etc/nagios} }
    }

    aggregate(dynamicip { ip =~ %r{192.168.0} })

    aggregate(firewallport { tag == nashvilledmz })

This would provide operators and such in Puppet for the first time, and I
would plan on (eventually) adding regular expressions and such, along with
(eventually) or/and operators and such.  I know that the Puppet language is
insanely simplistic right now and it would be nice if it were a bit more
powerful; this might be a step towards that.

Incidentally, this would make the current syntax just syntactic sugar:

    # Current syntax...
    service { nagios:
        require => file["/etc/nagios.conf"]
    }

    # ...equivalent to...
    service { nagios:
        require => file { name == "/etc/nagios.conf" }
    }

What do people think of this syntax?

If this syntax were provided, then I would only need to add an 'aggregate'
function to the language, which would be relatively easy (compared to the
search stuff, anyway).

There are a couple other points worth noting:

1) The current syntax is parsed by the parser but is otherwise left alone.
That is, it's the library that looks up an object by type and name.  This
would move that lookup into the parser, and I'm not sure how that would
work, frankly.

2) I have a 'tag' parameter up there as an example -- all Puppet elements
get automatically tagged with the classes and nodes that they're specified
on, so you could look up object up by those tags, rather than using class
membership.  I don't know if that would be sufficient, but it's a start.  We
could maybe add 'inclass' and 'isnode' test functions or something; I dunno.


-- 
Never esteem anything as of advantage to you that will make you break your word
or lose your self-respect.      -- Marcus Aurelius Antoninus
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com