""" Use Consul K/V as a Pillar source with values parsed as YAML :depends: - python-consul In order to use an consul server, a profile must be created in the master configuration file: .. code-block:: yaml my_consul_config: consul.host: 127.0.0.1 consul.port: 8500 consul.token: b6376760-a8bb-edd5-fcda-33bc13bfc556 consul.scheme: http consul.consistency: default consul.dc: dev consul.verify: True All parameters are optional. The ``consul.token`` requires python-consul >= 0.4.7. If you have a multi-datacenter Consul cluster you can map your ``pillarenv`` entries to your data centers by providing a dictionary of mappings in ``consul.dc`` field: .. code-block:: yaml my_consul_config: consul.dc: dev: us-east-1 prod: us-west-1 In the example above we specifying static mapping between Pillar environments and data centers: the data for ``dev`` and ``prod`` Pillar environments will be fetched from ``us-east-1`` and ``us-west-1`` datacenter respectively. In fact when ``consul.dc`` is set to dictionary keys are processed as regular expressions (that can capture named parameters) and values are processed as string templates as per PEP 3101. .. code-block:: yaml my_consul_config: consul.dc: ^dev-.*$: dev-datacenter ^(?P.*)-prod$: prod-datacenter-{region} This example maps all Pillar environments starting with ``dev-`` to ``dev-datacenter`` whereas Pillar environment like ``eu-prod`` will be mapped to ``prod-datacenter-eu``. Before evaluation patterns are sorted by length in descending order. If Pillar environment names correspond to data center names a single pattern can be used: .. code-block:: yaml my_consul_config: consul.dc: ^(?P.*)$: '{env}' After the profile is created, configure the external pillar system to use it. Optionally, a root may be specified. .. code-block:: yaml ext_pillar: - consul: my_consul_config ext_pillar: - consul: my_consul_config root=salt Using these configuration profiles, multiple consul sources may also be used: .. code-block:: yaml ext_pillar: - consul: my_consul_config - consul: my_other_consul_config Either the ``minion_id``, or the ``role``, or the ``environment`` grain may be used in the ``root`` path to expose minion-specific information stored in consul. .. code-block:: yaml ext_pillar: - consul: my_consul_config root=salt/%(minion_id)s - consul: my_consul_config root=salt/%(role)s - consul: my_consul_config root=salt/%(environment)s Minion-specific values may override shared values when the minion-specific root appears after the shared root: .. code-block:: yaml ext_pillar: - consul: my_consul_config root=salt-shared - consul: my_other_consul_config root=salt-private/%(minion_id)s If using the ``role`` or ``environment`` grain in the consul key path, be sure to define it using `/etc/salt/grains`, or similar: .. code-block:: yaml role: my-minion-role environment: dev It's possible to lock down where the pillar values are shared through minion targeting. Note that double quotes ``"`` are required around the target value and cannot be used inside the matching statement. See the section on Compound Matchers for more examples. .. code-block:: yaml ext_pillar: - consul: my_consul_config root=salt target="L@salt.example.com and G@osarch:x86_64" The data from Consul can be merged into a nested key in Pillar. .. code-block:: yaml ext_pillar: - consul: my_consul_config pillar_root=consul_data By default, keys containing YAML data will be deserialized before being merged into Pillar. This behavior can be disabled by setting ``expand_keys`` to ``false``. .. code-block:: yaml ext_pillar: - consul: my_consul_config expand_keys=false """ import logging import re import salt.utils.minions import salt.utils.yaml from salt.exceptions import CommandExecutionError from salt.utils.dictupdate import update as dict_merge try: import consul if not hasattr(consul, "__version__"): consul.__version__ = ( # Some packages has no version, and so this pillar crashes on access to it. "0.1" ) except ImportError: consul = None __virtualname__ = "consul" # Set up logging log = logging.getLogger(__name__) def __virtual__(): """ Only return if python-consul is installed """ return __virtualname__ if consul is not None else False def ext_pillar(minion_id, pillar, conf): # pylint: disable=W0613 """ Check consul for all data """ opts = {} temp = conf target_re = re.compile('target="(.*?)"') match = target_re.search(temp) if match: opts["target"] = match.group(1) temp = temp.replace(match.group(0), "") checker = salt.utils.minions.CkMinions(__opts__) _res = checker.check_minions(opts["target"], "compound") minions = _res["minions"] log.debug("Targeted minions: %r", minions) if minion_id not in minions: return {} root_re = re.compile(r"(?