Testing salt formulas and models with kitchen-salt

How it works

Kitchen spin an instances in Docker, Vagrant, OpenStack environment, etc. based on configured driver. Instance is configured as salt minion, where the configuration is defined by .kitchen.yml and tests/pillar/*.sls

Override your specific needs with .kitchen.<backend|local>.yml that you may load as: KITCHEN_LOCAL_YAML=.kitchen.<driver>.yml kitchen <action> <suite>.

Example: KITCHEN_LOCAL_YAML=.kitchen.local kitchen verify server-ubuntu-1404 -t tests/integration.

Test Kitchen then allows you execute several action to perform your testing under configured conditions:

1. create, provision an test instance (VM, container)
2. converge, run a provisioner driver (shell script, kitchen-salt, ...)
3. verify, run a verification tool (inspec, test-infra, ...)
4. destroy

Usage

A listing of scenarios to be executed:

$ kitchen list

Instance                    Driver   Provisioner  Verifier  Transport  Last Action

client-single-ubuntu-1404   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-ubuntu-1604   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-centos-71     Docker   SaltSolo     Inspec    Ssh        <Not Created>

Out of the box kitchen uses special verifier called Busser <https://github.com/test-kitchen/busser>_ used to recognize verification driver automatically by scripts/configs implemented in <repo>/test/integration. Busser then install, setup verifier(s) and execute.

As the verifier you can use InSpec, test-infra, Shell, Bats, ...). You specify directly

Example workflow:

# list instances and status
kitchen list

# manually execute integration tests
kitchen [test || [create|converge|verify|exec|login|destroy|...]] [instance] -t tests/integration

# use with provided Makefile (ie: within CI pipeline)
make kitchen

Verifying deployment

There is couple of verifier plugins that are shipped with Test Kitchen. They allow to run simple bash scripts and checking it's exit codes to run specific purpose based frameworks.

The Busser Verifier goes with test-kitchen by default. It is used to setup and run tests implemented in <repo>/test/integration. It guess and installs the particular driver to tested instance. By default InSpec is expected.

You may avoid to install busser framework if you configure specific verifier in .kitchen.yml:

verifier:
        name: inspec

For default Inspec Verifier implement your scripts directly in <repo>/test/integration/<suite> directory with _spec.rb suffix.

If you would to write another verification scripts than InSpec store them in <repo>/tests/integration/<suite>/<verifier>. Busser <https://github.com/test-kitchen/busser> is a test setup and execution framework under test kitchen.

Implement integration tests under <repo>/tests/integration/<suite>/<verifier> directory with _spec.<verifier suffix> filename suffix.

InSpec

InSpec is native validation framework for Test Kitchen and as such don't require usage of <verifier> folder. Thus the tests may by stored directly under <repo>/tests/integration/<suite>

Additional resources.

• https://inspec.io
• https://github.com/chef/inspec
• https://github.com/chef/kitchen-inspec

Example verification scripts under tests/integration folder of the formula:

tests
├── integration
│   ├── default
│   │   └── default_testcase_spec.rb  # Written in InSpec
│   ├── backupmx
│   │   └── serverspec                # <Verifier framework>
│   │       └── backupmx_spec.rb      # Written in ServerSpec
│   ├── helpers
│   │   └── serverspec
│   │       └── spec_helper.rb
│   ├── relay
│   │   └── serverspec
│   │       └── relay_spec.rb
│   └── server
│       └── serverspec
│           ├── aliases_spec.rb
│           └── server_spec.rb
├── pillar
│   ├── backupmx.sls
│   ├── relay.sls
│   └── server.sls
└── run_tests.sh

Requirements

Use latest stable kitchen-salt and kitchen-test. Supported version of kitchen-salt is available at kitchen-salt <https://github.com/salt-formulas/kitchen-salt> fork.

TL;DR

First you have to install ruby package manager gem.

Install required gems:

# Ruby side:
gem install <gem name from the list below>

# Isolated w/Bundler
gem install bundler

cat > Gemfile <-EOF
              source 'https://rubygems.org'

              gem 'rake'
              gem 'test-kitchen'
              gem 'kitchen-docker'
              gem 'kitchen-inspec'
              gem 'inspec'
              gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git'
      EOF

bundle install [--path $PWD/.vendor/bundle]

# use with preffix 'bundle kitchen':
# bundle exec kitchen list

Create aliases:

cat > ~/.${SHELL}rc <-EOF
              alias bk='nocorrect bundle exec kitchen'
              alias kl='nocorrect bundle exec kitchen list'
EOF

See http://kitchen.ci/ for more details.

Install procedure

One may be satisfied installing ruby and gems system-wide right from OS package manager.

If you are an ruby/chef developer you will probably want to use ChefDK <https://downloads.chef.io/chefdk>.

For advanced users or the sake of complex environments you may use rbenv for user side ruby installation.

• https://github.com/rbenv/rbenv
• http://kitchen.ci/docs/getting-started/installing

An example steps to install user side ruby and prerequisites:

# Use package manager to install rbenv and ruby-build
sudo apt-get install rbenv ruby-build

# list all available versions:
rbenv install -l

# install a Ruby version of your choice or pick latest
rbenv install $(rbenv install -l|grep -E '^[ ]*[0-9]\.[0-9]+'|tail -1)

# activate
rbenv local 2.4.0

# it's usually a good idea to update rubygems first
rbenv exec gem update --system

# install test kitchen
rbenv exec gem install bundler
rbenv exec gem install test-kitchen

Continue with the optional Gemfile in the formula main directory to fetch fine tuned dependencies. If you use Gemfile and Bundler for local dependencies prepend all command with rbenv exec bundler exec and possibly set an alias in your ~/.bashrc, etc.

cat >> ~/.${SHELL}rc <<-EOF
              alias rk="rbenv exec kitchen"
              alias bk="rbenv exec bundler exec kitchen"
EOF

With such alias set, you should be able to execute rbenv exec bundler exec make kitchen and see test results.

Sample configs

For advanced configs have a look at .kitchen*.yml examples in cookiecutter template <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula/%7B%7Bcookiecutter.service_name%7D%7D>_.

.kitchen.yml

---
driver:
  name: docker
  hostname: opencontrail
  use_sudo: true

provisioner:
  name: salt_solo
  salt_install: bootstrap
  salt_bootstrap_url: https://bootstrap.saltstack.com
  salt_version: latest
  require_chef: false
  log_level: error
  formula: opencontrail
  grains:
    noservices: True
  dependencies:
    - name: linux
      repo: git
      source: https://github.com/salt-formulas/salt-formula-linux
  state_top:
    base:
      "*":
        - linux
        - opencontrail
  pillars:
    top.sls:
      base:
        "*":
          - linux_repo_openstack
          - linux_repo_cassandra
          - linux_repo_opencontrail
          - linux_repo_mos
          - linux
          - opencontrail
          - opencontrail_juniper
    linux.sls:
      linux:
        system:
          enabled: true
          name: opencontrail
    opencontrail_juniper.sls: {}
  pillars-from-files:
    linux_repo_mos.sls: tests/pillar/repo_mos8.sls
    linux_repo_cassandra.sls: tests/pillar/repo_cassandra.sls
    linux_repo_openstack.sls: tests/pillar/repo_openstack.sls
    linux_repo_opencontrail.sls: tests/pillar/repo_opencontrail.sls

verifier:
  name: inspec
  sudo: true

platforms:
  - name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
    driver_config:
      image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
      platform: ubuntu

suites:

  - name: <%= ENV['SUITE'] || 'single' %>
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/<%= ENV['SUITE'] || 'single' %>.sls

  - name: cluster
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/cluster.sls

  - name: analytics
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/analytics.sls

  - name: control
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/control.sls

  - name: vendor-juniper
    provisioner:
      vendor_repo:
        - type: apt
          url: http://aptly.local/contrail
          key_url: http://aptly.local/public.gpg
          components: main
          distribution: trusty
      pillars-from-files:
        opencontrail.sls: tests/pillar/control.sls
      pillars:
        opencontrail_juniper.sls:
          opencontrail:
            common:
              vendor: juniper


# vim: ft=yaml sw=2 ts=2 sts=2 tw=125

Sample configs

.travis.yml

sudo: required
services:
  - docker

# PREREQUISITES
install:
  - pip install PyYAML
  - pip install virtualenv
  - |
    test -e Gemfile || cat <<EOF > Gemfile
    source 'https://rubygems.org'
    gem 'rake'
    gem 'test-kitchen'
    gem 'kitchen-docker'
    gem 'kitchen-inspec'
    gem 'inspec'
    gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git
  - bundle install

# BUILD MATRIX
env:
  - PLATFORM=trevorj/salty-whales:trusty
  - PLATFORM=trevorj/salty-whales:xenial
  - PLATFORM=trevorj/salty-whales:xenial-2016.3

# SMOKE TEST
before_script:
  - set -o pipefail
  - make test | tail

# KITCHEN TEST
script:
  - bundle exec kitchen test -t tests/integration

# vim: ft=yaml sw=2 ts=2 sts=2 tw=125