mirror of
https://github.com/ansible-collections/community.general.git
synced 2025-10-23 12:33:59 -07:00
Docs how to test (2nd) (#24094)
* Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail
This commit is contained in:
John R Barker
GitHub
parent
bc22223d63
commit
ecbf8e933a
22 changed files with 866 additions and 558 deletions
|
|
@ -1,70 +0,0 @@
|
|||
Sanity Tests
|
||||
============
|
||||
|
||||
Sanity tests are made up of scripts and tools used to perform static code analysis.
|
||||
The primary purpose of these tests is to enforce Ansible coding standards and requirements.
|
||||
|
||||
Tests are run with ``ansible-test sanity``.
|
||||
All available tests are run unless the ``--test`` option is used.
|
||||
|
||||
Available Tests
|
||||
===============
|
||||
|
||||
Tests can be listed with ``ansible-test sanity --list-tests``.
|
||||
|
||||
This list is a combination of two different categories of tests.
|
||||
|
||||
Code Smell Tests
|
||||
----------------
|
||||
|
||||
Miscellaneous `scripts <code-smell/>`_ used for enforcing coding standards and requirements, identifying trip hazards, etc.
|
||||
|
||||
These tests are listed and accessed by script name. There is no actual test named ``code-smell``.
|
||||
|
||||
All executable scripts added to the ``code-smell`` directory are automatically detected and executed by ``ansible-test``.
|
||||
|
||||
Scripts in the directory which fail can be skipped by adding them to `skip.txt <code-smell/skip.txt>`_.
|
||||
This is useful for scripts which identify issues that have not yet been resolved in the code base.
|
||||
|
||||
Files tested are specific to the individual test scripts and are not affected by command line arguments.
|
||||
|
||||
Built-in Tests
|
||||
--------------
|
||||
|
||||
These tests are integrated directly into ``ansible-test``.
|
||||
All files relevant to each test are tested unless specific files are specified.
|
||||
|
||||
ansible-doc
|
||||
~~~~~~~~~~~
|
||||
|
||||
Verifies that ``ansible-doc`` can parse module documentation on all supported python versions.
|
||||
|
||||
pep8
|
||||
~~~~
|
||||
|
||||
Python static analysis for PEP 8 style guideline compliance.
|
||||
|
||||
pylint
|
||||
~~~~~~
|
||||
|
||||
Python static analysis for common programming errors.
|
||||
|
||||
rstcheck
|
||||
~~~~~~~~
|
||||
|
||||
Check reStructuredText files for syntax and formatting issues.
|
||||
|
||||
shellcheck
|
||||
~~~~~~~~~~
|
||||
|
||||
Static code analysis for shell scripts using the excellent `shellcheck <https://www.shellcheck.net/>`_ tool.
|
||||
|
||||
validate-modules
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
Analyze modules for common issues in code and documentation.
|
||||
|
||||
yamllint
|
||||
~~~~~~~~
|
||||
|
||||
Check YAML files for syntax and formatting issues.
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
# PEP 8
|
||||
|
||||
[PEP 8](https://www.python.org/dev/peps/pep-0008/) style guidelines are enforced by
|
||||
[pep8](https://pypi.python.org/pypi/pep8) on all python files in the repository by default.
|
||||
|
||||
## Current Rule Set
|
||||
|
||||
By default all files are tested using the current rule set.
|
||||
All `pep8` tests are executed, except those listed in the [current ignore list](current-ignore.txt).
|
||||
|
||||
## Legacy Rule Set
|
||||
|
||||
Files which are listed in the [legacy file list](legacy-files.txt) are tested using the legacy rule set.
|
||||
All `pep8` tests are executed, except those listed in the [current ignore list](current-ignore.txt) or
|
||||
[legacy ignore list](legacy-ignore.txt).
|
||||
|
||||
> Files listed in the legacy file list which pass the current rule set will result in an error.
|
||||
> This is intended to prevent regressions on style guidelines for files which pass the more stringent current rule set.
|
||||
|
||||
## Skipping Tests
|
||||
|
||||
Files listed in the [skip list](skip.txt) are not tested by `pep8`.
|
||||
|
||||
## Removed Files
|
||||
|
||||
Files which have been removed from the repository must be removed from the legacy file list and the skip list.
|
||||
|
||||
## Running Locally
|
||||
|
||||
The pep8 check can be run locally with:
|
||||
|
||||
./test/runner/ansible-test sanity --test pep8 [file-or-directory-path-to-check] ...
|
||||
|
||||
|
|
@ -1,168 +0,0 @@
|
|||
validate-modules
|
||||
================
|
||||
|
||||
Python program to help test or validate Ansible modules.
|
||||
|
||||
Originally developed by Matt Martz (@sivel)
|
||||
|
||||
Usage
|
||||
~~~~~
|
||||
|
||||
.. code:: shell
|
||||
|
||||
cd /path/to/ansible/source
|
||||
source hacking/env-setup
|
||||
test/sanity/validate-modules/validate-modules /path/to/modules
|
||||
|
||||
Help
|
||||
~~~~
|
||||
|
||||
.. code:: shell
|
||||
|
||||
usage: validate-modules [-h] [-w] [--exclude EXCLUDE] [--arg-spec]
|
||||
[--base-branch BASE_BRANCH] [--format {json,plain}]
|
||||
[--output OUTPUT]
|
||||
modules [modules ...]
|
||||
|
||||
positional arguments:
|
||||
modules Path to module or module directory
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-w, --warnings Show warnings
|
||||
--exclude EXCLUDE RegEx exclusion pattern
|
||||
--arg-spec Analyze module argument spec
|
||||
--base-branch BASE_BRANCH
|
||||
Used in determining if new options were added
|
||||
--format {json,plain}
|
||||
Output format. Default: "plain"
|
||||
--output OUTPUT Output location, use "-" for stdout. Default "-"
|
||||
|
||||
Codes
|
||||
~~~~~~~
|
||||
|
||||
Errors
|
||||
^^^^^^
|
||||
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| code | sample message |
|
||||
+=========+============================================================================================================================================+
|
||||
| **1xx** | **Locations** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 101 | Interpreter line is not ``#!/usr/bin/python`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 102 | Interpreter line is not ``#!powershell`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 103 | Did not find a call to ``main()`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 104 | Call to ``main()`` not the last line |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 105 | GPLv3 license header not found |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 106 | Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 107 | Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **2xx** | **Imports** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 201 | Did not find a ``module_utils`` import |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 203 | ``requests`` import found, should use ``ansible.module_utils.urls`` instead |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 204 | ``boto`` import found, new modules should use ``boto3`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 205 | ``sys.exit()`` call found. Should be ``exit_json``/``fail_json`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 206 | ``WANT_JSON`` not found in module |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 207 | ``REPLACER_WINDOWS`` not found in module |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 208 | ``module_utils`` imports should import specific components, not ``*`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **3xx** | **Documentation** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 301 | No ``DOCUMENTATION`` provided |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 302 | ``DOCUMENTATION`` is not valid YAML |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 303 | ``DOCUMENTATION`` fragment missing |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 304 | Unknown ``DOCUMENTATION`` error |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 305 | Invalid ``DOCUMENTATION`` schema |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 306 | Module level ``version_added`` is not a valid version number |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 307 | Module level ``version_added`` is incorrect |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 308 | ``version_added`` for new option is not a valid version number |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 309 | ``version_added`` for new option is incorrect |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 310 | No ``EXAMPLES`` provided |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 311 | ``EXAMPLES`` is not valid YAML |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 312 | No ``RETURN`` documentation provided |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 313 | ``RETURN`` is not valid YAML |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 314 | No ``ANSIBLE_METADATA`` provided |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 315 | ``ANSIBLE_METADATA`` is not valid YAML |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 316 | Invalid ``ANSIBLE_METADATA`` schema |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 317 | option is marked as required but specifies a default. Arguments with a default should not be marked as required |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 318 | Module deprecated, but DOCUMENTATION.deprecated is missing |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **4xx** | **Syntax** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 401 | Python ``SyntaxError`` while parsing module |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 402 | Indentation contains tabs |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 403 | Type comparison using ``type()`` found. Use ``isinstance()`` instead |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **5xx** | **Naming** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 501 | Official Ansible modules must have a ``.py`` extension for python modules or a ``.ps1`` for powershell modules |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 502 | Ansible module subdirectories must contain an ``__init__.py`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 503 | Missing python documentation file |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
|
||||
Warnings
|
||||
^^^^^^^^
|
||||
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| code | sample message |
|
||||
+=========+============================================================================================================================================+
|
||||
| **1xx** | **Locations** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 107 | Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` for legacy modules |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **2xx** | **Imports** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 208 | ``module_utils`` imports should import specific components for legacy module, not ``*`` |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 291 | Try/Except ``HAS_`` expression missing |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 292 | Did not find ``ansible.module_utils.basic`` import |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| **3xx** | **Documentation** |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 312 | No ``RETURN`` documentation provided for legacy module |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 391 | Unknown pre-existing ``DOCUMENTATION`` error |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| 392 | Pre-existing ``DOCUMENTATION`` fragment missing |
|
||||
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
Loading…
Add table
Add a link
Reference in a new issue