Become plugins (#50991)

* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now

lib/ansible/plugins/connection/__init__.py

@@ -6,19 +6,16 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import fcntl
-import gettext
 import os
 import shlex
 from abc import abstractmethod, abstractproperty
 from functools import wraps
 
 from ansible import constants as C
-from ansible.errors import AnsibleError
-from ansible.module_utils.six import string_types
 from ansible.module_utils._text import to_bytes, to_text
 from ansible.plugins import AnsiblePlugin
-from ansible.plugins.loader import shell_loader, connection_loader
 from ansible.utils.display import Display
+from ansible.plugins.loader import connection_loader, get_shell_plugin
 from ansible.utils.path import unfrackpath
 
 display = Display()
@@ -46,7 +43,7 @@ class ConnectionBase(AnsiblePlugin):
     has_pipelining = False
     has_native_async = False  # eg, winrm
     always_pipeline_modules = False  # eg, winrm
-    become_methods = C.BECOME_METHODS
+    has_tty = True  # for interacting with become plugins
     # When running over this connection type, prefer modules written in a certain language
     # as discovered by the specified file extension.  An empty string as the
     # language means any language.
@@ -66,11 +63,12 @@ class ConnectionBase(AnsiblePlugin):
 
         # All these hasattrs allow subclasses to override these parameters
         if not hasattr(self, '_play_context'):
+            # Backwards compat: self._play_context isn't really needed, using set_options/get_option
             self._play_context = play_context
         if not hasattr(self, '_new_stdin'):
             self._new_stdin = new_stdin
-        # Backwards compat: self._display isn't really needed, just import the global display and use that.
         if not hasattr(self, '_display'):
+            # Backwards compat: self._display isn't really needed, just import the global display and use that.
             self._display = display
         if not hasattr(self, '_connected'):
             self._connected = False
@@ -80,30 +78,17 @@ class ConnectionBase(AnsiblePlugin):
         self._connected = False
         self._socket_path = None
 
-        if shell is not None:
-            self._shell = shell
+        # helper plugins
+        self._shell = shell
 
-        # load the shell plugin for this action/connection
-        if play_context.shell:
-            shell_type = play_context.shell
-        elif hasattr(self, '_shell_type'):
-            shell_type = getattr(self, '_shell_type')
-        else:
-            shell_type = 'sh'
-            shell_filename = os.path.basename(self._play_context.executable)
-            try:
-                shell = shell_loader.get(shell_filename)
-            except Exception:
-                shell = None
-            if shell is None:
-                for shell in shell_loader.all():
-                    if shell_filename in shell.COMPATIBLE_SHELLS:
-                        break
-            shell_type = shell.SHELL_FAMILY
-
-        self._shell = shell_loader.get(shell_type)
+        # we always must have shell
         if not self._shell:
-            raise AnsibleError("Invalid shell type specified (%s), or the plugin for that shell type is missing." % shell_type)
+            self._shell = get_shell_plugin(shell_type=getattr(self, '_shell_type', None), executable=self._play_context.executable)
+
+        self.become = None
+
+    def set_become_plugin(self, plugin):
+        self.become = plugin
 
     @property
     def connected(self):
@@ -115,14 +100,6 @@ class ConnectionBase(AnsiblePlugin):
         '''Read-only property holding the connection socket path for this remote host'''
         return self._socket_path
 
-    def _become_method_supported(self):
-        ''' Checks if the current class supports this privilege escalation method '''
-
-        if self._play_context.become_method in self.become_methods:
-            return True
-
-        raise AnsibleError("Internal Error: this connection module does not support running commands via %s" % self._play_context.become_method)
-
     @staticmethod
     def _split_ssh_args(argstring):
         """
@@ -151,10 +128,6 @@ class ConnectionBase(AnsiblePlugin):
     def _connect(self):
         """Connect to the host we've been initialized with"""
 
-        # Check if PE is supported
-        if self._play_context.become:
-            self._become_method_supported()
-
     @ensure_connect
     @abstractmethod
     def exec_command(self, cmd, in_data=None, sudoable=True):
@@ -240,31 +213,6 @@ class ConnectionBase(AnsiblePlugin):
         """Terminate the connection"""
         pass
 
-    def check_become_success(self, b_output):
-        b_success_key = to_bytes(self._play_context.success_key)
-        for b_line in b_output.splitlines(True):
-            if b_success_key == b_line.rstrip():
-                return True
-        return False
-
-    def check_password_prompt(self, b_output):
-        if self._play_context.prompt is None:
-            return False
-        elif isinstance(self._play_context.prompt, string_types):
-            b_prompt = to_bytes(self._play_context.prompt).strip()
-            b_lines = b_output.splitlines()
-            return any(l.strip().startswith(b_prompt) for l in b_lines)
-        else:
-            return self._play_context.prompt(b_output)
-
-    def check_incorrect_password(self, b_output):
-        b_incorrect_password = to_bytes(gettext.dgettext(self._play_context.become_method, C.BECOME_ERROR_STRINGS[self._play_context.become_method]))
-        return b_incorrect_password and b_incorrect_password in b_output
-
-    def check_missing_password(self, b_output):
-        b_missing_password = to_bytes(gettext.dgettext(self._play_context.become_method, C.BECOME_MISSING_STRINGS[self._play_context.become_method]))
-        return b_missing_password and b_missing_password in b_output
-
     def connection_lock(self):
         f = self._play_context.connection_lockfd
         display.vvvv('CONNECTION: pid %d waiting for lock on %d' % (os.getpid(), f), host=self._play_context.remote_addr)
@@ -279,6 +227,39 @@ class ConnectionBase(AnsiblePlugin):
     def reset(self):
         display.warning("Reset is not implemented for this connection")
 
+    # NOTE: these password functions are all become specific, the name is
+    # confusing as it does not handle 'protocol passwords'
+    # DEPRECATED:
+    # These are kept for backwards compatiblity
+    # Use the methods provided by the become plugins instead
+    def check_become_success(self, b_output):
+        display.deprecated(
+            "Connection.check_become_success is deprecated, calling code should be using become plugins instead",
+            version="2.12"
+        )
+        return self.become.check_success(b_output)
+
+    def check_password_prompt(self, b_output):
+        display.deprecated(
+            "Connection.check_password_prompt is deprecated, calling code should be using become plugins instead",
+            version="2.12"
+        )
+        return self.become.check_password_prompt(b_output)
+
+    def check_incorrect_password(self, b_output):
+        display.deprecated(
+            "Connection.check_incorrect_password is deprecated, calling code should be using become plugins instead",
+            version="2.12"
+        )
+        return self.become.check_incorrect_password(b_output)
+
+    def check_missing_password(self, b_output):
+        display.deprecated(
+            "Connection.check_missing_password is deprecated, calling code should be using become plugins instead",
+            version="2.12"
+        )
+        return self.become.check_missing_password(b_output)
+
 
 class NetworkConnectionBase(ConnectionBase):
     """

lib/ansible/plugins/connection/buildah.py

@@ -63,7 +63,6 @@ class Connection(ConnectionBase):
     # String used to identify this Connection class from other classes
     transport = 'buildah'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS)
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

lib/ansible/plugins/connection/chroot.py

@@ -59,7 +59,7 @@ class Connection(ConnectionBase):
     # su currently has an undiagnosed issue with calculating the file
     # checksums (so copy, for instance, doesn't work right)
     # Have to look into that before re-enabling this
-    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))
+    has_tty = False
 
     default_user = 'root'
 

lib/ansible/plugins/connection/docker.py

@@ -62,7 +62,6 @@ class Connection(ConnectionBase):
 
     transport = 'docker'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS)
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

lib/ansible/plugins/connection/jail.py

@@ -56,8 +56,7 @@ class Connection(ConnectionBase):
     # Pipelining may work.  Someone needs to test by setting this to True and
     # having pipelining=True in their ansible.cfg
     has_pipelining = True
-
-    become_methods = frozenset(C.BECOME_METHODS)
+    has_tty = False
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

lib/ansible/plugins/connection/kubectl.py

@@ -198,7 +198,6 @@ class Connection(ConnectionBase):
     connection_options = CONNECTION_OPTIONS
     documentation = DOCUMENTATION
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS)
     transport_cmd = None
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):

lib/ansible/plugins/connection/libvirt_lxc.py

@@ -49,8 +49,8 @@ class Connection(ConnectionBase):
     # su currently has an undiagnosed issue with calculating the file
     # checksums (so copy, for instance, doesn't work right)
     # Have to look into that before re-enabling this
-    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))
     default_user = 'root'
+    has_tty = False
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

lib/ansible/plugins/connection/local.py

@@ -83,7 +83,7 @@ class Connection(ConnectionBase):
         )
         display.debug("done running command with Popen()")
 
-        if self._play_context.prompt and sudoable:
+        if self.become and self.become.expect_prompt() and sudoable:
             fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) | os.O_NONBLOCK)
             fcntl.fcntl(p.stderr, fcntl.F_SETFL, fcntl.fcntl(p.stderr, fcntl.F_GETFL) | os.O_NONBLOCK)
             selector = selectors.DefaultSelector()

lib/ansible/plugins/connection/lxc.py

@@ -54,7 +54,6 @@ class Connection(ConnectionBase):
 
     transport = 'lxc'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS)
     default_user = 'root'
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):

lib/ansible/plugins/connection/napalm.py

@@ -156,10 +156,8 @@ class Connection(NetworkConnectionBase):
 
     def _connect(self):
         if not HAS_NAPALM:
-            raise AnsibleError(
-                'Napalm is required to use the napalm connection type.\n'
-                'Please run pip install napalm'
-            )
+            raise AnsibleError('The "napalm" python library is required to use the napalm connection type.\n')
+
         super(Connection, self)._connect()
 
         if not self.connected:

lib/ansible/plugins/connection/netconf.py

@@ -274,7 +274,7 @@ class Connection(NetworkConnectionBase):
     def _connect(self):
         if not HAS_NCCLIENT:
             raise AnsibleError(
-                'ncclient is required to use the netconf connection type: %s.\n'
+                'The required "ncclient" python library is required to use the netconf connection type: %s.\n'
                 'Please run pip install ncclient' % to_native(NCCLIENT_IMP_ERR)
             )
 

lib/ansible/plugins/connection/paramiko_ssh.py

@@ -415,7 +415,7 @@ class Connection(ConnectionBase):
 
         try:
             chan.exec_command(cmd)
-            if self._play_context.prompt:
+            if self.become and self.become.expect_prompt():
                 passprompt = False
                 become_sucess = False
                 while not (become_sucess or passprompt):

lib/ansible/plugins/connection/psrp.py

@@ -218,7 +218,6 @@ class Connection(ConnectionBase):
 
     transport = 'psrp'
     module_implementation_preferences = ('.ps1', '.exe', '')
-    become_methods = ['runas']
     allow_executable = False
     has_pipelining = True
     allow_extras = True

lib/ansible/plugins/connection/qubes.py

@@ -61,7 +61,6 @@ class Connection(ConnectionBase):
     # String used to identify this Connection class from other classes
     transport = 'qubes'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS)
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

lib/ansible/plugins/connection/ssh.py

@@ -443,7 +443,6 @@ class Connection(ConnectionBase):
 
     transport = 'ssh'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS).difference(['runas'])
 
     def __init__(self, *args, **kwargs):
         super(Connection, self).__init__(*args, **kwargs)
@@ -708,11 +707,11 @@ class Connection(ConnectionBase):
             suppress_output = False
 
             # display.debug("Examining line (source=%s, state=%s): '%s'" % (source, state, display_line))
-            if self._play_context.prompt and self.check_password_prompt(b_line):
+            if self.become.expect_prompt() and self.check_password_prompt(b_line):
                 display.debug("become_prompt: (source=%s, state=%s): '%s'" % (source, state, display_line))
                 self._flags['become_prompt'] = True
                 suppress_output = True
-            elif self._play_context.success_key and self.check_become_success(b_line):
+            elif self.become.success and self.check_become_success(b_line):
                 display.debug("become_success: (source=%s, state=%s): '%s'" % (source, state, display_line))
                 self._flags['become_success'] = True
                 suppress_output = True
@@ -811,11 +810,12 @@ class Connection(ConnectionBase):
 
         state = states.index('ready_to_send')
         if to_bytes(self.get_option('ssh_executable')) in cmd and sudoable:
-            if self._play_context.prompt:
+            prompt = getattr(self.become, 'prompt', None)
+            if prompt:
                 # We're requesting escalation with a password, so we have to
                 # wait for a password prompt.
                 state = states.index('awaiting_prompt')
-                display.debug(u'Initial state: %s: %s' % (states[state], self._play_context.prompt))
+                display.debug(u'Initial state: %s: %s' % (states[state], prompt))
             elif self._play_context.become and self._play_context.success_key:
                 # We're requesting escalation without a password, so we have to
                 # detect success/failure before sending any initial data.

lib/ansible/plugins/connection/winrm.py

@@ -176,7 +176,6 @@ class Connection(ConnectionBase):
 
     transport = 'winrm'
     module_implementation_preferences = ('.ps1', '.exe', '')
-    become_methods = ['runas']
     allow_executable = False
     has_pipelining = True
     allow_extras = True
@@ -207,10 +206,6 @@ class Connection(ConnectionBase):
         self._winrm_user = self.get_option('remote_user')
         self._winrm_pass = self._play_context.password
 
-        self._become_method = self._play_context.become_method
-        self._become_user = self._play_context.become_user
-        self._become_pass = self._play_context.become_pass
-
         self._winrm_port = self.get_option('port')
 
         self._winrm_scheme = self.get_option('scheme')

lib/ansible/plugins/connection/zone.py

@@ -47,7 +47,7 @@ class Connection(ConnectionBase):
 
     transport = 'zone'
     has_pipelining = True
-    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))
+    has_tty = False
 
     def __init__(self, play_context, new_stdin, *args, **kwargs):
         super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)