ansible-collections/community.general
1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2025-10-23 20:44:00 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

Docs how to test (2nd) (#24094)

Browse source 
* Big testing doc refactor
* Combine all the testing documentation in to one place to make it easier to find
* Convert everything to RST
* Create testing_network guide
* Create testing landing page
* For each section detail "how to run" and "how to extend testing"
* More examples
* Lots more detail
This commit is contained in:
John R Barker 2017-04-28 09:08:26 +01:00 • committed by GitHub
commit ecbf8e933a
22 changed files with 866 additions and 558 deletions
Download patch file Download diff file Expand all files Collapse all files

140
test/README.rst
View file

@ -1,140 +0,0 @@
Testing Ansible
===============
How to run and create tests for the Ansible core engine and modules with ``ansible-test``.
Requirements
============
There are no special requirements for running ``ansible-test`` on Python 2.7 or later.
The ``argparse`` package is required for Python 2.6.
The requirements for each ``ansible-test`` command are covered later.
Setup
=====
#. Fork the `ansible/ansible <https://github.com/ansible/ansible/>`_ repository on Git Hub.
#. Clone your fork: ``git clone git@github.com:USERNAME/ansible.git``
#. Install the optional ``argcomplete`` package for tab completion (highly recommended):
#. ``pip install argcomplete``
#. ``activate-global-python-argcomplete``
#. Restart your shell to complete global activation.
#. Configure your environment to run from your clone (once per shell): ``. hacking/env-setup``
Test Environments
=================
Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing.
Local
-----
The ``--local`` option runs tests locally without the use of an isolated test environment.
This is the default behavior.
Recommended for ``compile`` tests.
See the `command requirements directory <runner/requirements/>`_ for the requirements for each ``ansible-test`` command.
Requirements files are named after their respective commands.
See also the `constraints <runner/requirements/constraints.txt>`_ applicable to all commands.
Use the ``--requirements`` option to automatically install ``pip`` requirements relevant to the command being used.
Docker
------
The ``--docker`` option runs tests in a docker container.
Recommended for ``integration`` tests.
This option accepts an optional docker container image.
See the `list of supported docker images <runner/completion/docker.txt>`_ for options.
Use the ``--docker-no-pull`` option to avoid pulling the latest container image.
This is required when using custom local images that are not available for download.
Tox
---
The ``--tox`` option run tests in a ``tox`` managed Python virtual environment.
Recommended for ``windows-integration`` and ``units`` tests.
The following Python versions are supported:
* 2.6
* 2.7
* 3.5
* 3.6
By default, test commands will run against all supported Python versions when using ``tox``.
Use the ``--python`` option to specify a single Python version to use for test commands.
Remote
------
The ``--remote`` option runs tests in a cloud hosted environment.
An API key is required to use this feature.
Recommended for integration tests.
See the `list of supported platforms and versions <runner/completion/remote.txt>`_ for additional details.
General Usage
=============
Tests are run with the ``ansible-test`` command.
Consult ``ansible-test --help`` for usage information not covered here.
Use the ``--explain`` option to see what commands will be executed without actually running them.
Running Tests
=============
There are four main categories of tests, each in their own directory.
* `compile <compile/>`_ - Python syntax checking for supported versions. Examples:
* ``ansible-test compile`` - Check syntax for all supported versions.
* ``ansible-test compile --python 3.5`` - Check only Python 3.5 syntax.
* `sanity <sanity/>`_ - Static code analysis and general purpose script-based tests. Examples:
* ``ansible-test sanity --tox --python 2.7`` - Run all sanity tests on Python 2.7 using ``tox``.
* ``ansible-test sanity --test pep8`` - Run the ``pep8`` test without ``tox``.
* `integration <integration/>`_ - Playbook based tests for modules and core engine functionality. Examples:
* ``ansible-test integration ping --docker`` - Run the ``ping`` module test using ``docker``.
* ``ansible-test windows-integration windows/ci/`` - Run all Windows tests covered by CI.
* `units <units/>`_ - API oriented tests using mock interfaces for modules and core engine functionality. Examples:
* ``ansible-test units --tox`` - Run all unit tests on all supported Python versions using ``tox``.
* ``ansible-test units --tox --python 2.7 test/units/vars/`` - Run specific tests on Python 2.7 using ``tox``.
Consult each of the test directories for additional details on usage and requirements.
Interactive Shell
=================
Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples:
* ``ansible-test shell --docker`` - Open a shell in the default docker container.
* ``ansible-test shell --tox --python 3.6`` - Open a shell in the Python 3.6 ``tox`` environment.
Code Coverage
=============
Add the ``--coverage`` option to any test command to collect code coverage data.
Reports can be generated in several different formats:
* ``ansible-test coverage report`` - Console report.
* ``ansible-test coverage html`` - HTML report.
* ``ansible-test coverage xml`` - XML report.
To clear data between test runs, use the ``ansible-test coverage erase`` command.

13
test/compile/README.rst
View file

@ -1,13 +0,0 @@
Compile Tests
=============
Compile tests check source files for valid syntax on all supported python versions:
- 2.6
- 2.7
- 3.5
- 3.6
Tests are run with ``ansible-test compile``.
All versions are tested unless the ``--python`` option is used.
All ``*.py`` files are tested unless specific files are specified.

223
test/integration/README.md
View file

@ -1,223 +0,0 @@
Integration tests
=================
The ansible integration system.
Tests for playbooks, by playbooks.
Some tests may require credentials. Credentials may be specified with `credentials.yml`.
Some tests may require root.
Quick Start
===========
It is highly recommended that you install and activate the `argcomplete` python package.
It provides tab completion in `bash` for the `ansible-test` test runner.
To get started quickly using Docker containers for testing,
see [Tests in Docker containers](#tests-in-docker-containers).
Configuration
=============
Making your own version of `integration_config.yml` can allow for setting some
tunable parameters to help run the tests better in your environment. Some
tests (e.g. cloud) will only run when access credentials are provided. For
more information about supported credentials, refer to `credentials.template`.
Prerequisites
=============
The tests will assume things like hg, svn, and git are installed and in path.
(Complete list pending)
Non-destructive Tests
=====================
These tests will modify files in subdirectories, but will not do things that install or remove packages or things
outside of those test subdirectories. They will also not reconfigure or bounce system services.
Run as follows for all POSIX platform tests executed by our CI system:
test/runner/ansible-test integration -v posix/ci/
You can select specific tests as well, such as for individual modules:
test/runner/ansible-test integration -v ping
Destructive Tests
=================
These tests are allowed to install and remove some trivial packages. You will likely want to devote these
to a virtual environment. They won't reformat your filesystem, however :)
test/runner/ansible-test integration -v destructive/
Cloud Tests
===========
Cloud tests exercise capabilities of cloud modules (e.g. ec2_key). These are
not 'tests run in the cloud' so much as tests that leverage the cloud modules
and are organized by cloud provider.
In order to run cloud tests, you must provide access credentials in a file
named `credentials.yml`. A sample credentials file named
`credentials.template` is available for syntax help.
Provide cloud credentials:
cp credentials.template credentials.yml
${EDITOR:-vi} credentials.yml
Run the tests:
make cloud
*WARNING* running cloud integration tests will create and destroy cloud
resources. Running these tests may result in additional fees associated with
your cloud account. Care is taken to ensure that created resources are
removed. However, it is advisable to inspect your AWS console to ensure no
unexpected resources are running.
Windows Tests
=============
These tests exercise the winrm connection plugin and Windows modules. You'll
need to define an inventory with a remote Windows 2008 or 2012 Server to use
for testing, and enable PowerShell Remoting to continue.
Running these tests may result in changes to your Windows host, so don't run
them against a production/critical Windows environment.
Enable PowerShell Remoting (run on the Windows host via Remote Desktop):
Enable-PSRemoting -Force
Define Windows inventory:
cp inventory.winrm.template inventory.winrm
${EDITOR:-vi} inventory.winrm
Run the Windows tests executed by our CI system:
test/runner/ansible-test windows-integration -v windows/ci/
Tests in Docker containers
==========================
If you have a Linux system with Docker installed, running integration tests using the same Docker containers used by
the Ansible continuous integration (CI) system is recommended.
> Using Docker Engine to run Docker on a non-Linux host is not recommended.
> Some tests may fail, depending on the image used for testing.
> Using the `--docker-privileged` option may resolve the issue.
## Running Integration Tests
To run all CI integration test targets for POSIX platforms in a Ubuntu 16.04 container:
test/runner/ansible-test integration -v posix/ci/ --docker
You can also run specific tests or select a different Linux distribution.
For example, to run tests for the `ping` module on a Ubuntu 14.04 container:
test/runner/ansible-test integration -v ping --docker ubuntu1404
## Container Images
### Python 2
Most container images are for testing with Python 2:
- centos6
- centos7
- fedora24
- fedora25
- opensuse42.1
- opensuse42.2
- ubuntu1204
- ubuntu1404
- ubuntu1604
### Python 3
To test with Python 3 use the following images:
- ubuntu1604py3
Network Tests
=============
**Note:** From Ansible 2.3, for any new Network Module to be accepted it must be accompanied by a corresponding test.
For further help with this please contact `gundalow` in `#ansible-devel` on FreeNode IRC.
```
$ ANSIBLE_ROLES_PATH=targets ansible-playbook network-all.yaml
```
*NOTE* To run the network tests you will need a number of test machines and sutabily configured inventory file, a sample is included in `test/integration/inventory.network`
*NOTE* As with the rest of the integration tests, they can be found grouped by module in `test/integration/targets/MODULENAME/`
To filter a set of test cases set `limit_to` to the name of the group, generally this is the name of the module:
```
$ ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml -e "limit_to=eos_command"
```
To filter a singular test case set the tags options to eapi or cli, set limit_to to the test group,
and test_cases to the name of the test:
```
$ ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml --tags="cli" -e "limit_to=eos_command test_case=notequal"
```
## Contributing Test Cases
Test cases are added to roles based on the module being testing. Test cases
should include both `cli` and `eapi` test cases. Cli test cases should be
added to `test/integration/targets/modulename/tests/cli` and eapi tests should be added to
`test/integration/targets/modulename/tests/eapi`.
In addition to positive testing, negative tests are required to ensure user friendly warnings & errors are generated, rather than backtraces, for example:
```yaml
- name: test invalid subset (foobar)
eos_facts:
provider: "{{ cli }}"
gather_subset:
- "foobar"
register: result
ignore_errors: true
- assert:
that:
# Failures shouldn't return changes
- "result.changed == false"
# It's a failure
- "result.failed == true"
# Sensible Failure message
- "'Subset must be one of' in result.msg"
```
### Conventions
- Each test case should generally follow the pattern:
>setup —> test —> assert —> test again (idempotent) —> assert —> -teardown (if needed) -> done
This keeps test playbooks from becoming monolithic and difficult to
troubleshoot.
- Include a name for each task that is not an assertion. (It's OK to add names
to assertions too. But to make it easy to identify the broken task within a failed
test, at least provide a helpful name for each task.)
- Files containing test cases must end in `.yaml`
### Adding a new Network Platform
A top level playbook is required such as `ansible/test/integration/eos.yaml` which needs to be references by `ansible/test/integration/network-all.yaml`

70
test/sanity/README.rst
View file

@ -1,70 +0,0 @@
Sanity Tests
============
Sanity tests are made up of scripts and tools used to perform static code analysis.
The primary purpose of these tests is to enforce Ansible coding standards and requirements.
Tests are run with ``ansible-test sanity``.
All available tests are run unless the ``--test`` option is used.
Available Tests
===============
Tests can be listed with ``ansible-test sanity --list-tests``.
This list is a combination of two different categories of tests.
Code Smell Tests
----------------
Miscellaneous `scripts <code-smell/>`_ used for enforcing coding standards and requirements, identifying trip hazards, etc.
These tests are listed and accessed by script name. There is no actual test named ``code-smell``.
All executable scripts added to the ``code-smell`` directory are automatically detected and executed by ``ansible-test``.
Scripts in the directory which fail can be skipped by adding them to `skip.txt <code-smell/skip.txt>`_.
This is useful for scripts which identify issues that have not yet been resolved in the code base.
Files tested are specific to the individual test scripts and are not affected by command line arguments.
Built-in Tests
--------------
These tests are integrated directly into ``ansible-test``.
All files relevant to each test are tested unless specific files are specified.
ansible-doc
~~~~~~~~~~~
Verifies that ``ansible-doc`` can parse module documentation on all supported python versions.
pep8
~~~~
Python static analysis for PEP 8 style guideline compliance.
pylint
~~~~~~
Python static analysis for common programming errors.
rstcheck
~~~~~~~~
Check reStructuredText files for syntax and formatting issues.
shellcheck
~~~~~~~~~~
Static code analysis for shell scripts using the excellent `shellcheck <https://www.shellcheck.net/>`_ tool.
validate-modules
~~~~~~~~~~~~~~~~
Analyze modules for common issues in code and documentation.
yamllint
~~~~~~~~
Check YAML files for syntax and formatting issues.

33
test/sanity/pep8/README.md
View file

@ -1,33 +0,0 @@
# PEP 8
[PEP 8](https://www.python.org/dev/peps/pep-0008/) style guidelines are enforced by
[pep8](https://pypi.python.org/pypi/pep8) on all python files in the repository by default.
## Current Rule Set
By default all files are tested using the current rule set.
All `pep8` tests are executed, except those listed in the [current ignore list](current-ignore.txt).
## Legacy Rule Set
Files which are listed in the [legacy file list](legacy-files.txt) are tested using the legacy rule set.
All `pep8` tests are executed, except those listed in the [current ignore list](current-ignore.txt) or
[legacy ignore list](legacy-ignore.txt).
> Files listed in the legacy file list which pass the current rule set will result in an error.
> This is intended to prevent regressions on style guidelines for files which pass the more stringent current rule set.
## Skipping Tests
Files listed in the [skip list](skip.txt) are not tested by `pep8`.
## Removed Files
Files which have been removed from the repository must be removed from the legacy file list and the skip list.
## Running Locally
The pep8 check can be run locally with:
./test/runner/ansible-test sanity --test pep8 [file-or-directory-path-to-check] ...

168
test/sanity/validate-modules/README.rst
View file

@ -1,168 +0,0 @@
validate-modules
================
Python program to help test or validate Ansible modules.
Originally developed by Matt Martz (@sivel)
Usage
~~~~~
.. code:: shell
cd /path/to/ansible/source
source hacking/env-setup
test/sanity/validate-modules/validate-modules /path/to/modules
Help
~~~~
.. code:: shell
usage: validate-modules [-h] [-w] [--exclude EXCLUDE] [--arg-spec]
[--base-branch BASE_BRANCH] [--format {json,plain}]
[--output OUTPUT]
modules [modules ...]
positional arguments:
modules Path to module or module directory
optional arguments:
-h, --help show this help message and exit
-w, --warnings Show warnings
--exclude EXCLUDE RegEx exclusion pattern
--arg-spec Analyze module argument spec
--base-branch BASE_BRANCH
Used in determining if new options were added
--format {json,plain}
Output format. Default: "plain"
--output OUTPUT Output location, use "-" for stdout. Default "-"
Codes
~~~~~~~
Errors
^^^^^^
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| code | sample message |
+=========+============================================================================================================================================+
| **1xx** | **Locations** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 101 | Interpreter line is not ``#!/usr/bin/python`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 102 | Interpreter line is not ``#!powershell`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 103 | Did not find a call to ``main()`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 104 | Call to ``main()`` not the last line |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 105 | GPLv3 license header not found |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 106 | Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 107 | Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **2xx** | **Imports** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 201 | Did not find a ``module_utils`` import |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 203 | ``requests`` import found, should use ``ansible.module_utils.urls`` instead |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 204 | ``boto`` import found, new modules should use ``boto3`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 205 | ``sys.exit()`` call found. Should be ``exit_json``/``fail_json`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 206 | ``WANT_JSON`` not found in module |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 207 | ``REPLACER_WINDOWS`` not found in module |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 208 | ``module_utils`` imports should import specific components, not ``*`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **3xx** | **Documentation** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 301 | No ``DOCUMENTATION`` provided |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 302 | ``DOCUMENTATION`` is not valid YAML |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 303 | ``DOCUMENTATION`` fragment missing |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 304 | Unknown ``DOCUMENTATION`` error |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 305 | Invalid ``DOCUMENTATION`` schema |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 306 | Module level ``version_added`` is not a valid version number |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 307 | Module level ``version_added`` is incorrect |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 308 | ``version_added`` for new option is not a valid version number |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 309 | ``version_added`` for new option is incorrect |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 310 | No ``EXAMPLES`` provided |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 311 | ``EXAMPLES`` is not valid YAML |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 312 | No ``RETURN`` documentation provided |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 313 | ``RETURN`` is not valid YAML |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 314 | No ``ANSIBLE_METADATA`` provided |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 315 | ``ANSIBLE_METADATA`` is not valid YAML |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 316 | Invalid ``ANSIBLE_METADATA`` schema |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 317 | option is marked as required but specifies a default. Arguments with a default should not be marked as required |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 318 | Module deprecated, but DOCUMENTATION.deprecated is missing |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **4xx** | **Syntax** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 401 | Python ``SyntaxError`` while parsing module |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 402 | Indentation contains tabs |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 403 | Type comparison using ``type()`` found. Use ``isinstance()`` instead |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **5xx** | **Naming** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 501 | Official Ansible modules must have a ``.py`` extension for python modules or a ``.ps1`` for powershell modules |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 502 | Ansible module subdirectories must contain an ``__init__.py`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 503 | Missing python documentation file |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
Warnings
^^^^^^^^
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| code | sample message |
+=========+============================================================================================================================================+
| **1xx** | **Locations** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 107 | Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` for legacy modules |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **2xx** | **Imports** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 208 | ``module_utils`` imports should import specific components for legacy module, not ``*`` |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 291 | Try/Except ``HAS_`` expression missing |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 292 | Did not find ``ansible.module_utils.basic`` import |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **3xx** | **Documentation** |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 312 | No ``RETURN`` documentation provided for legacy module |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 391 | Unknown pre-existing ``DOCUMENTATION`` error |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 392 | Pre-existing ``DOCUMENTATION`` fragment missing |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+

50
test/utils/docker/httptester/README.rst
View file

@ -1,50 +0,0 @@
httptester
==========
HTTP Testing endpoint which provides httpbin, nginx, SSL and SNI
capabilities, for providing a local HTTP endpoint for testing
Building
--------
Docker
~~~~~~
Both ways of building docker utilize the ``nginx:alpine`` image, but can
be customized for ``Fedora``, ``Red Hat``, ``CentOS``, ``Ubuntu``,
``Debian`` and other variants of ``Alpine``
When utilizing ``packer`` or configuring with ``ansible-playbook``
the services will not automtically start on launch, and will have to be
manually started using::
$ /services.sh
Such as when starting a docker container::
docker run -ti --rm -p 80:80 -p 443:443 --name httptester ansible/ansible:httptester /services.sh
docker build
^^^^^^^^^^^^
::
docker build -t ansible/ansible:httptester .
packer
^^^^^^
The packer build will use ``ansible-playbook`` to perform the
configuration, and will tag the image as ``ansible/ansible:httptester``
::
packer build packer.json
Ansible
~~~~~~~
::
ansible-playbook -i hosts -v httptester.yml