Python interpreter discovery (#50163)

* Python interpreter discovery

* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter, append `_silent` to either value to suppress warnings.
* includes new doc utility method `get_versioned_doclink` to generate a major.minor versioned doclink against docs.ansible.com (or some other config-overridden URL)

* docs revisions for python interpreter discovery

(cherry picked from commit 5b53c0012ab7212304c28fdd24cb33fd8ff755c2)

* verify output on some distros, cleanup

docs/docsite/rst/index.rst

@@ -75,6 +75,7 @@ Ansible releases a new major release of Ansible approximately three to four time
    reference_appendices/config
    reference_appendices/YAMLSyntax
    reference_appendices/python_3_support
+   reference_appendices/interpreter_discovery
    reference_appendices/release_and_maintenance
    reference_appendices/test_strategies
    dev_guide/testing/sanity/index

docs/docsite/rst/porting_guides/porting_guide_2.8.rst

@@ -62,11 +62,85 @@ Beginning in version 2.8, Ansible will warn if a module expects a string, but a
 
 This behavior can be changed to be an error or to be ignored by setting the ``ANSIBLE_STRING_CONVERSION_ACTION`` environment variable, or by setting the ``string_conversion_action`` configuration in the ``defaults`` section of ``ansible.cfg``.
 
+
 Command line facts
 ------------------
 
 ``cmdline`` facts returned in system will be deprecated in favor of ``proc_cmdline``. This change handles special case where Kernel command line parameter contains multiple values with the same key.
 
+
+Python Interpreter Discovery
+============================
+
+In Ansible 2.7 and earlier, Ansible defaulted to ``usr/bin/python`` as the
+setting for ``ansible_python_interpreter``. If you ran Ansible against a system
+that installed Python with a different name or a different path, your playbooks
+would fail with ``/usr/bin/python: bad interpreter: No such file or directory``
+unless you either set ``ansible_python_interpreter`` to the correct value for
+that system or added a Python interpreter and any necessary dependencies at
+``usr/bin/python``.
+
+Starting in Ansible 2.8, Ansible searches for the correct path and executable
+name for Python on each target system, first in a lookup table of default
+Python interpreters for common distros, then in an ordered fallback list of
+possible Python interpreter names/paths.
+
+It's risky to rely on a Python interpreter set from the fallback list, because
+the interpreter may change on future runs. If an interpreter from
+higher in the fallback list gets installed (for example, as a side-effect of
+installing other packages), your original interpreter and its dependencies will
+no longer be used. For this reason, Ansible warns you when it uses a Python
+interpreter discovered from the fallback list. If you see this warning, the
+best solution is to explicitly set ``ansible_python_interpreter`` to the path
+of the correct interpreter for those target systems.
+
+You can still set ``ansible_python_interpreter`` to a specific path at any
+variable level (as a host variable, in vars files, in playbooks, etc.).
+If you prefer to use the Python interpreter discovery behavior, use
+one of the four new values for ``ansible_python_interpreter`` introduced in
+Ansible 2.8:
+
++---------------------------+-----------------------------------------------+
+| New value                 | Behavior                                      |
++===========================+===============================================+
+| | auto                    | | If a Python interpreter is discovered,      |
+| | (future default)        | | Ansible uses the discovered Python, even if |
+| |                         | | ``/usr/bin/python`` is also present. Warns  |
+| |                         | | when using the fallback list.               |
++---------------------------+-----------------------------------------------+
+| | **auto_legacy**         | | If a Python interpreter is discovered, and  |
+| | (Ansible 2.8 default)   | | ``/usr/bin/python`` is absent, Ansible      |
+| |                         | | uses the discovered Python. Warns when      |
+| |                         | | using the fallback list.                    |
+| |                         | |                                             |
+| |                         | | If a Python interpreter is discovered, and  |
+| |                         | | ``/usr/bin/python`` is present, Ansible     |
+| |                         | | uses ``/usr/bin/python`` and prints a       |
+| |                         | | deprecation warning about future default    |
+| |                         | | behavior. Warns when using the fallback     |
+| |                         | | list.                                       |
++---------------------------+-----------------------------------------------+
+| | auto_legacy_silent      | | Behaves like ``auto_legacy`` but suppresses |
+| |                         | | the deprecation and fallback-list warnings. |
++---------------------------+-----------------------------------------------+
+| | auto_silent             | | Behaves like ``auto`` but suppresses the    |
+| |                         | | fallback-list warning.                      |
++---------------------------+-----------------------------------------------+
+
+Starting with Ansible 2.12, Ansible will use the discovered Python interpreter
+by default, whether or not ``/usr/bin/python`` is also present. Until then,
+the default ``auto_legacy`` setting provides compatibility with
+previous versions of Ansible that always defaulted to ``/usr/bin/python``.
+
+If you installed Python and dependencies (``boto``, etc.) to
+``/usr/bin/python`` as a workaround on distros with a different default Python
+interpreter (for example, Ubuntu 16.04+, RHEL8, Fedora 23+), you have two
+options:
+
+  #. Move existing dependencies over to the default Python for each platform/distribution/version.
+  #. Use ``auto_legacy``. This setting lets Ansible find and use the workaround Python on hosts that have it, while also finding the correct default Python on newer hosts. But remember, the default will change in 4 releases.
+
+
 Command Line
 ============
 

docs/docsite/rst/reference_appendices/interpreter_discovery.rst

@@ -0,0 +1,51 @@
+.. _interpreter_discovery:
+
+Interpreter Discovery
+=====================
+
+Most Ansible modules that execute under a POSIX environment require a Python
+interpreter on the target host. Unless configured otherwise, Ansible will
+attempt to discover a suitable Python interpreter on each target host
+the first time a Python module is executed for that host.
+
+To control the discovery behavior:
+
+* for individual hosts and groups, use the ``ansible_python_interpreter`` inventory variable
+* globally, use the ``interpreter_python`` key in the ``[defaults]`` section of ``ansible.cfg``
+
+Use one of the following values:
+
+auto_legacy : (default in 2.8)
+  Detects the target OS platform, distribution, and version, then consults a
+  table listing the correct Python interpreter and path for each
+  platform/distribution/version. If an entry is found, and ``/usr/bin/python`` is absent, uses the discovered interpreter (and path). If an entry
+  is found, and ``/usr/bin/python`` is present, uses ``/usr/bin/python``
+  and issues a warning.
+  This exception provides temporary compatibility with previous versions of
+  Ansible that always defaulted to ``/usr/bin/python``, so if you have
+  installed Python and other dependencies at ``usr/bin/python`` on some hosts,
+  Ansible will find and use them with this setting.
+  If no entry is found, or the listed Python is not present on the
+  target host, searches a list of common Python interpreter
+  paths and uses the first one found; also issues a warning that future
+  installation of another Python interpreter could alter the one chosen.
+
+auto : (future default in 2.12)
+  Detects the target OS platform, distribution, and version, then consults a
+  table listing the correct Python interpreter and path for each
+  platform/distribution/version. If an entry is found, uses the discovered
+  interpreter.
+  If no entry is found, or the listed Python is not present on the
+  target host, searches a list of common Python interpreter
+  paths and uses the first one found; also issues a warning that future
+  installation of another Python interpreter could alter the one chosen.
+
+auto_legacy_silent
+  Same as ``auto_legacy``, but does not issue warnings.
+
+auto_silent
+  Same as ``auto``, but does not issue warnings.
+
+You can still set ``ansible_python_interpreter`` to a specific path at any
+variable level (for example, in host_vars, in vars files, in playbooks, etc.).
+Setting a specific path completely disables automatic interpreter discovery; Ansible always uses the path specified.

docs/docsite/rst/reference_appendices/python_3_support.rst

@@ -39,9 +39,12 @@ command via ``python3``.  For example:
 Using Python 3 on the managed machines with commands and playbooks
 ------------------------------------------------------------------
 
-* Set the ``ansible_python_interpreter`` configuration option to :command:`/usr/bin/python3`. The
-  ``ansible_python_interpreter`` configuration option is usually set as an inventory
-  variable associated with a host or group of hosts:
+* Ansible will automatically detect and use Python 3 on many platforms that ship with it. To explicitly configure a
+  Python 3 interpreter, set the ``ansible_python_interpreter`` inventory variable at a group or host level to the
+  location of a Python 3 interpreter, such as :command:`/usr/bin/python3`. The default interpreter path may also be
+  set in ``ansible.cfg``.
+
+.. seealso:: :ref:`interpreter_discovery` for more information.
 
 .. code-block:: ini
 

docs/docsite/rst/user_guide/modules.rst

@@ -29,6 +29,8 @@ like services, packages, or files (anything really), or handle executing system
        How to write your own modules
    :doc:`../dev_guide/developing_api`
        Examples of using modules with the Python API
+   :doc:`../reference_appendices/interpreter_discovery`
+       Configuring the right Python interpreter on target hosts
    `Mailing List <https://groups.google.com/group/ansible-project>`_
        Questions? Help? Ideas?  Stop by the list on Google Groups
    `irc.freenode.net <http://irc.freenode.net>`_