« Back to home

Testing salt formulas and models with kitchen-salt


An overview how to implement salt-formula test's with kitchen-salt.


In this article I will talk about writing and testing https://github.com/salt-formulas[salt-formulas]. An curated collection of formulas behind salt-formulas community. The concepts however will fit to any salt formula.

If you are interested in https://github.com/salt-formulas[salt-formulas] read more at: https://salt-formulas.readthedocs.io/en/latest>[readthedocs salt-formulas].

To test formulas we will use https://github.com/saltstack/kitchen-salt[kitchen-salt] framework.

This particular article extends already published documentation https://salt-formulas.readthedocs.io/en/latest/develop/index.html#testing[salt-formulas/testing] however the additional posts will cover advanced configurations, latest features, verifiers as well as testing salt models/pillars.

Formula repository

To meet required formula repository structure, new formulas can be generated by using this cookiecutter template <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula>_.

Mind there is also a script to update existing formula repo to kitchen initial kitchen setup.

Salt Formula testing

Each formula contains Makefile with at least test target. Under tests directory are located resources for test execution.

Test target executes "smoke test" implemented by tests/run_tests.sh <https://github.com/salt-formulas/salt-formulas/blob/master/cookiecutter/salt-formula/%7B%7Bcookiecutter.service_name%7D%7D/tests/run_tests.sh>_ capable to fetch dependencies in python virtual environment by executing salt-call state.show_sls with provided tests/pillar data.

The purpose of the smoke test is to find syntax, typo issues and verify example pillar data against the formula.

Initial content of tests folder contains test pillars and a run_tests.sh as generated by cookiecutter <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula>_.

├── pillar
│   └── client_single.sls
│   └── server_single.sls
└── run_tests.sh

Create or update pillars in tests/pillar/\*.sls with test data.

Formula testing with Test Kitchen

Test Kitchen with "now" SaltStack official kitchen-salt provisioner plugin may be used for local development as well as CI scenario.

Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation. There is a .kitchen.yml in main directory that defines platforms to be tested and suites to execute on them.

Kitchen CI can spin instances locally or remote, based on used driver. For example .kitchen.yml may define a docker, openstack or vagrant driver.

For more, explore it's rich ecosystem of supported drivers/provisioners/verifiers/...

How it works

Kitchen spin an instances in Docker, Vagrant, OpenStack environment, etc. based on configured driver. Instance is configured as salt minion, where the configuration is defined by .kitchen.yml and tests/pillar/*.sls

Override your specific needs with .kitchen.<backend|local>.yml that you may load as: KITCHEN_LOCAL_YAML=.kitchen.<driver>.yml kitchen <action> <suite>.

Example: KITCHEN_LOCAL_YAML=.kitchen.local kitchen verify server-ubuntu-1404 -t tests/integration.

Test Kitchen then allows you execute several action to perform your testing under configured conditions:

  1. create, provision an test instance (VM, container)
  2. converge, run a provisioner driver (shell script, kitchen-salt, ...)
  3. verify, run a verification tool (inspec, test-infra, ...)
  4. destroy


A listing of scenarios to be executed:

$ kitchen list

Instance                    Driver   Provisioner  Verifier  Transport  Last Action

client-single-ubuntu-1404   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-ubuntu-1604   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-centos-71     Docker   SaltSolo     Inspec    Ssh        <Not Created>

Out of the box kitchen uses special verifier called Busser <https://github.com/test-kitchen/busser>_ used to recognize verification driver automatically by scripts/configs implemented in <repo>/test/integration. Busser then install, setup verifier(s) and execute.

As the verifier you can use InSpec, test-infra, Shell, Bats, ...). You specify directly

Example workflow:

# list instances and status
kitchen list

# manually execute integration tests
kitchen [test || [create|converge|verify|exec|login|destroy|...]] [instance] -t tests/integration

# use with provided Makefile (ie: within CI pipeline)
make kitchen

Verifying deployment

There is couple of verifier plugins that are shipped with Test Kitchen. They allow to run simple bash scripts and checking it's exit codes to run specific purpose based frameworks.

The Busser Verifier goes with test-kitchen by default. It is used to setup and run tests implemented in <repo>/test/integration. It guess and installs the particular driver to tested instance. By default InSpec is expected.

You may avoid to install busser framework if you configure specific verifier in .kitchen.yml:

        name: inspec

For default Inspec Verifier implement your scripts directly in <repo>/test/integration/<suite> directory with _spec.rb suffix.

If you would to write another verification scripts than InSpec store them in <repo>/tests/integration/<suite>/<verifier>. Busser <https://github.com/test-kitchen/busser> is a test setup and execution framework under test kitchen.

Implement integration tests under <repo>/tests/integration/<suite>/<verifier> directory with _spec.<verifier suffix> filename suffix.


InSpec is native validation framework for Test Kitchen and as such don't require usage of <verifier> folder. Thus the tests may by stored directly under <repo>/tests/integration/<suite>

Additional resources.

Example verification scripts under tests/integration folder of the formula:

├── integration
│   ├── default
│   │   └── default_testcase_spec.rb  # Written in InSpec
│   ├── backupmx
│   │   └── serverspec                # <Verifier framework>
│   │       └── backupmx_spec.rb      # Written in ServerSpec
│   ├── helpers
│   │   └── serverspec
│   │       └── spec_helper.rb
│   ├── relay
│   │   └── serverspec
│   │       └── relay_spec.rb
│   └── server
│       └── serverspec
│           ├── aliases_spec.rb
│           └── server_spec.rb
├── pillar
│   ├── backupmx.sls
│   ├── relay.sls
│   └── server.sls
└── run_tests.sh


Use latest stable kitchen-salt and kitchen-test. Supported version of kitchen-salt is available at kitchen-salt <https://github.com/salt-formulas/kitchen-salt> fork.


First you have to install ruby package manager gem.

Install required gems:

# Ruby side:
gem install <gem name from the list below>

# Isolated w/Bundler
gem install bundler

cat > Gemfile <-EOF
              source 'https://rubygems.org'

              gem 'rake'
              gem 'test-kitchen'
              gem 'kitchen-docker'
              gem 'kitchen-inspec'
              gem 'inspec'
              gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git'

bundle install [--path $PWD/.vendor/bundle]

# use with preffix 'bundle kitchen':
# bundle exec kitchen list

Create aliases:

cat > ~/.${SHELL}rc <-EOF
              alias bk='nocorrect bundle exec kitchen'
              alias kl='nocorrect bundle exec kitchen list'

See http://kitchen.ci/ for more details.

Install procedure

One may be satisfied installing ruby and gems system-wide right from OS package manager.

If you are an ruby/chef developer you will probably want to use ChefDK <https://downloads.chef.io/chefdk>.

For advanced users or the sake of complex environments you may use rbenv for user side ruby installation.

An example steps to install user side ruby and prerequisites:

# Use package manager to install rbenv and ruby-build
sudo apt-get install rbenv ruby-build

# list all available versions:
rbenv install -l

# install a Ruby version of your choice or pick latest
rbenv install $(rbenv install -l|grep -E '^[ ]*[0-9]\.[0-9]+'|tail -1)

# activate
rbenv local 2.4.0

# it's usually a good idea to update rubygems first
rbenv exec gem update --system

# install test kitchen
rbenv exec gem install bundler
rbenv exec gem install test-kitchen

Continue with the optional Gemfile in the formula main directory to fetch fine tuned dependencies. If you use Gemfile and Bundler for local dependencies prepend all command with rbenv exec bundler exec and possibly set an alias in your ~/.bashrc, etc.

cat >> ~/.${SHELL}rc <<-EOF
              alias rk="rbenv exec kitchen"
              alias bk="rbenv exec bundler exec kitchen"

With such alias set, you should be able to execute rbenv exec bundler exec make kitchen and see test results.

Sample configs

For advanced configs have a look at .kitchen*.yml examples in cookiecutter template <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula/%7B%7Bcookiecutter.service_name%7D%7D>_.


  name: docker
  hostname: opencontrail
  use_sudo: true

  name: salt_solo
  salt_install: bootstrap
  salt_bootstrap_url: https://bootstrap.saltstack.com
  salt_version: latest
  require_chef: false
  log_level: error
  formula: opencontrail
    noservices: True
    - name: linux
      repo: git
      source: https://github.com/salt-formulas/salt-formula-linux
        - linux
        - opencontrail
          - linux_repo_openstack
          - linux_repo_cassandra
          - linux_repo_opencontrail
          - linux_repo_mos
          - linux
          - opencontrail
          - opencontrail_juniper
          enabled: true
          name: opencontrail
    opencontrail_juniper.sls: {}
    linux_repo_mos.sls: tests/pillar/repo_mos8.sls
    linux_repo_cassandra.sls: tests/pillar/repo_cassandra.sls
    linux_repo_openstack.sls: tests/pillar/repo_openstack.sls
    linux_repo_opencontrail.sls: tests/pillar/repo_opencontrail.sls

  name: inspec
  sudo: true

  - name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
      image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
      platform: ubuntu


  - name: <%= ENV['SUITE'] || 'single' %>
        opencontrail.sls: tests/pillar/<%= ENV['SUITE'] || 'single' %>.sls

  - name: cluster
        opencontrail.sls: tests/pillar/cluster.sls

  - name: analytics
        opencontrail.sls: tests/pillar/analytics.sls

  - name: control
        opencontrail.sls: tests/pillar/control.sls

  - name: vendor-juniper
        - type: apt
          url: http://aptly.local/contrail
          key_url: http://aptly.local/public.gpg
          components: main
          distribution: trusty
        opencontrail.sls: tests/pillar/control.sls
              vendor: juniper

# vim: ft=yaml sw=2 ts=2 sts=2 tw=125

Continous Integration with Travis

Salt-formulas uses Travis CI to run smoke and integration tests. Add .travis.yml:

Sample configs


sudo: required
  - docker

  - pip install PyYAML
  - pip install virtualenv
  - |
    test -e Gemfile || cat <<EOF > Gemfile
    source 'https://rubygems.org'
    gem 'rake'
    gem 'test-kitchen'
    gem 'kitchen-docker'
    gem 'kitchen-inspec'
    gem 'inspec'
    gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git
  - bundle install

  - PLATFORM=trevorj/salty-whales:trusty
  - PLATFORM=trevorj/salty-whales:xenial
  - PLATFORM=trevorj/salty-whales:xenial-2016.3

  - set -o pipefail
  - make test | tail

  - bundle exec kitchen test -t tests/integration

# vim: ft=yaml sw=2 ts=2 sts=2 tw=125

Common practices


At some rare cases execution of given state in the formula is not possible or required. For these cases set grain noservices: True and wrap corresponding code as in the example below:

{%- if not grains.get('noservices', False) %}
mysql_database_{{ database_name }}:
  - name: {{ database_name }}
  - character_set: {{ database.get('encoding', 'utf8') }}
  - connection_user: {{ connection.user }}
  - connection_pass: {{ connection.password }}
  - connection_charset: {{ connection.charset }}
{%- endif %}

As the mysql database might not be available in the given test environment (travis/docker, etc..).

In .kitchen.yml we set grain noservices: True by default.

  noservices: True

** formula dependencies **

Formula dependencies might be specified in <formula repo>/metadata.yml

name: "galera"
version: "1.0"
source: "https://github.com/salt-formulas/salt-formula-galera"
- name: mysql
  source: "https://github.com/salt-formulas/salt-formula-mysql"

While using test-kitchen formula dependencies must be specified in .kitchen.yml as well. Dependencies may be installed from git, spm or even apt repository.

    - name: mysql
      repo: git
      source: https://github.com/salt-formulas/salt-formula-mysql.git
    - name: linux
      repo: git
      source: https://github.com/salt-formulas/salt-formula-linux.git

For convenience kitchen-salt will read metadata.yml of these dependencies and install their dependencies in case you omit them in .kitchen.yml.

** build matrix **

To simplify local CI we ship .kitchen.yml with limited number of platforms. (ie: latest ubuntu as a falback option if no ENV variable PLATFORM is specified)

However this is later extended on Travis CI while using ENV variables in build matrix.

.travis.yml snippet:

  - PLATFORM=trevorj/salty-whales:trusty
  - PLATFORM=trevorj/salty-whales:xenial

.kitchen.yml snippet:

  - name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
      image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
      platform: ubuntu

= Example forumlas

# vim: filetype=asciidoc


comments powered by Disqus