mirror of
https://github.com/ansible-collections/community.general.git
synced 2025-07-26 14:41:23 -07:00
Docs how to test (2nd) (#24094)
* Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail
This commit is contained in:
John R Barker
GitHub
parent
bc22223d63
commit
ecbf8e933a
22 changed files with 866 additions and 558 deletions
235
docs/docsite/rst/dev_guide/testing_integration.rst
Normal file
235
docs/docsite/rst/dev_guide/testing_integration.rst
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
*****************
|
||||
Integration tests
|
||||
*****************
|
||||
|
||||
.. contents:: Topics
|
||||
|
||||
The Ansible integration Test system.
|
||||
|
||||
Tests for playbooks, by playbooks.
|
||||
|
||||
Some tests may require credentials. Credentials may be specified with `credentials.yml`.
|
||||
|
||||
Some tests may require root.
|
||||
|
||||
Quick Start
|
||||
===========
|
||||
|
||||
It is highly recommended that you install and activate the ``argcomplete`` python package.
|
||||
It provides tab completion in ``bash`` for the ``ansible-test`` test runner.
|
||||
|
||||
To get started quickly using Docker containers for testing,
|
||||
see [Tests in Docker containers](#tests-in-docker-containers).
|
||||
|
||||
Configuration
|
||||
=============
|
||||
|
||||
Making your own version of ``integration_config.yml`` can allow for setting some
|
||||
tunable parameters to help run the tests better in your environment. Some
|
||||
tests (e.g. cloud) will only run when access credentials are provided. For
|
||||
more information about supported credentials, refer to ``credentials.template``.
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
|
||||
The tests will assume things like hg, svn, and git are installed and in path.
|
||||
|
||||
(Complete list pending)
|
||||
|
||||
Non-destructive Tests
|
||||
=====================
|
||||
|
||||
These tests will modify files in subdirectories, but will not do things that install or remove packages or things
|
||||
outside of those test subdirectories. They will also not reconfigure or bounce system services.
|
||||
|
||||
.. note:: Running integration tests within Docker
|
||||
|
||||
To protect your system from any potential changes caused by integration tests, and to ensure the a sensible set of dependencies are available we recommend that you always run integration tests with the ``--docker`` option. See the `list of supported docker images <https://github.com/ansible/ansible/blob/devel/test/runner/completion/docker.txt>`_ for options.
|
||||
|
||||
.. note:: Avoiding pulling new Docker images:
|
||||
|
||||
Use the ``--docker-no-pull`` option to avoid pulling the latest container image. This is required when using custom local images that are not available for download.
|
||||
|
||||
Run as follows for all POSIX platform tests executed by our CI system::
|
||||
|
||||
test/runner/ansible-test integration --docker fedora25 -v posix/ci/
|
||||
|
||||
You can select specific tests as well, such as for individual modules::
|
||||
|
||||
test/runner/ansible-test integration -v ping
|
||||
|
||||
By installing ``argcomplete`` you can obtain a full list by doing::
|
||||
|
||||
test/runner/ansible-test integration <tab><tab>
|
||||
|
||||
Destructive Tests
|
||||
=================
|
||||
|
||||
These tests are allowed to install and remove some trivial packages. You will likely want to devote these
|
||||
to a virtual environment, such as Docker. They won't reformat your filesystem::
|
||||
|
||||
test/runner/ansible-test integration --docker fedora25 -v destructive/
|
||||
|
||||
Windows Tests
|
||||
=============
|
||||
|
||||
These tests exercise the ``winrm`` connection plugin and Windows modules. You'll
|
||||
need to define an inventory with a remote Windows 2008 or 2012 Server to use
|
||||
for testing, and enable PowerShell Remoting to continue.
|
||||
|
||||
Running these tests may result in changes to your Windows host, so don't run
|
||||
them against a production/critical Windows environment.
|
||||
|
||||
Enable PowerShell Remoting (run on the Windows host via Remote Desktop):
|
||||
Enable-PSRemoting -Force
|
||||
|
||||
Define Windows inventory::
|
||||
|
||||
cp inventory.winrm.template inventory.winrm
|
||||
${EDITOR:-vi} inventory.winrm
|
||||
|
||||
Run the Windows tests executed by our CI system::
|
||||
|
||||
test/runner/ansible-test windows-integration -v windows/ci/
|
||||
|
||||
Tests in Docker containers
|
||||
==========================
|
||||
|
||||
If you have a Linux system with Docker installed, running integration tests using the same Docker containers used by
|
||||
the Ansible continuous integration (CI) system is recommended.
|
||||
|
||||
.. note: Docker on non-Linux::
|
||||
|
||||
Using Docker Engine to run Docker on a non-Linux host is not recommended.
|
||||
Some tests may fail, depending on the image used for testing.
|
||||
Using the ``--docker-privileged`` option may resolve the issue.
|
||||
|
||||
Running Integration Tests
|
||||
-------------------------
|
||||
|
||||
To run all CI integration test targets for POSIX platforms in a Ubuntu 16.04 container::
|
||||
|
||||
test/runner/ansible-test integration -v posix/ci/ --docker
|
||||
|
||||
You can also run specific tests or select a different Linux distribution.
|
||||
For example, to run tests for the ``ping`` module on a Ubuntu 14.04 container::
|
||||
|
||||
test/runner/ansible-test integration -v ping --docker ubuntu1404
|
||||
|
||||
Container Images
|
||||
----------------
|
||||
|
||||
Python 2
|
||||
````````
|
||||
|
||||
Most container images are for testing with Python 2:
|
||||
|
||||
- centos6
|
||||
- centos7
|
||||
- fedora24
|
||||
- fedora25
|
||||
- opensuse42.1
|
||||
- opensuse42.2
|
||||
- ubuntu1204
|
||||
- ubuntu1404
|
||||
- ubuntu1604
|
||||
|
||||
Python 3
|
||||
````````
|
||||
|
||||
To test with Python 3 use the following images:
|
||||
|
||||
- ubuntu1604py3
|
||||
|
||||
Network Tests
|
||||
=============
|
||||
|
||||
This page details the specifics around testing Ansible Networking modules.
|
||||
|
||||
|
||||
.. important:: Network testing requirements for Ansible 2.4
|
||||
|
||||
Starting with Ansible 2.4, all network modules MUST include corresponding unit tests to defend functionality.
|
||||
The unit tests must be added in the same PR that includes the new network module, or extends functionality.
|
||||
Integration tests, although not required, are a welcome addition.
|
||||
How to do this is explained in the rest of this document.
|
||||
|
||||
|
||||
|
||||
Network integration tests can be ran by doing::
|
||||
|
||||
cd test/integration
|
||||
ANSIBLE_ROLES_PATH=targets ansible-playbook network-all.yaml
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
* To run the network tests you will need a number of test machines and suitably configured inventory file. A sample is included in ``test/integration/inventory.network``
|
||||
* As with the rest of the integration tests, they can be found grouped by module in ``test/integration/targets/MODULENAME/``
|
||||
|
||||
To filter a set of test cases set ``limit_to`` to the name of the group, generally this is the name of the module::
|
||||
|
||||
ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml -e "limit_to=eos_command"
|
||||
|
||||
|
||||
To filter a singular test case set the tags options to eapi or cli, set limit_to to the test group,
|
||||
and test_cases to the name of the test::
|
||||
|
||||
ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml --tags="cli" -e "limit_to=eos_command test_case=notequal"
|
||||
|
||||
|
||||
|
||||
Writing network integration tests
|
||||
---------------------------------
|
||||
|
||||
Test cases are added to roles based on the module being testing. Test cases
|
||||
should include both `cli` and `eapi` test cases. Cli test cases should be
|
||||
added to `test/integration/targets/modulename/tests/cli` and eapi tests should be added to
|
||||
`test/integration/targets/modulename/tests/eapi`.
|
||||
|
||||
In addition to positive testing, negative tests are required to ensure user friendly warnings & errors are generated, rather than backtraces, for example:
|
||||
|
||||
.. code-block: yaml
|
||||
|
||||
- name: test invalid subset (foobar)
|
||||
eos_facts:
|
||||
provider: "{{ cli }}"
|
||||
gather_subset:
|
||||
- "foobar"
|
||||
register: result
|
||||
ignore_errors: true
|
||||
|
||||
- assert:
|
||||
that:
|
||||
# Failures shouldn't return changes
|
||||
- "result.changed == false"
|
||||
# It's a failure
|
||||
- "result.failed == true"
|
||||
# Sensible Failure message
|
||||
- "'Subset must be one of' in result.msg"
|
||||
|
||||
|
||||
Conventions
|
||||
```````````
|
||||
|
||||
- Each test case should generally follow the pattern:
|
||||
|
||||
setup —> test —> assert —> test again (idempotent) —> assert —> teardown (if needed) -> done
|
||||
|
||||
This keeps test playbooks from becoming monolithic and difficult to
|
||||
troubleshoot.
|
||||
|
||||
- Include a name for each task that is not an assertion. (It's OK to add names
|
||||
to assertions too. But to make it easy to identify the broken task within a failed
|
||||
test, at least provide a helpful name for each task.)
|
||||
|
||||
- Files containing test cases must end in `.yaml`
|
||||
|
||||
|
||||
Adding a new Network Platform
|
||||
`````````````````````````````
|
||||
|
||||
A top level playbook is required such as `ansible/test/integration/eos.yaml` which needs to be references by `ansible/test/integration/network-all.yaml`
|
||||
|
||||
Where to find out more
|
||||
======================
|
||||
Loading…
Add table
Add a link
Reference in a new issue