Become plugins (#50991)

* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now

lib/ansible/plugins/become/su.py

@@ -0,0 +1,158 @@
+# -*- coding: utf-8 -*-
+# Copyright: (c) 2018, Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+DOCUMENTATION = """
+    become: su
+    short_description: Substitute User
+    description:
+        - This become plugins allows your remote/login user to execute commands as another user via the su utility.
+    author: ansible (@core)
+    version_added: "2.8"
+    options:
+        become_user:
+            description: User you 'become' to execute the task
+            default: root
+            ini:
+              - section: privilege_escalation
+                key: become_user
+              - section: su_become_plugin
+                key: user
+            vars:
+              - name: ansible_become_user
+              - name: ansible_su_user
+            env:
+              - name: ANSIBLE_BECOME_USER
+              - name: ANSIBLE_SU_USER
+        become_exe:
+            description: Su executable
+            default: su
+            ini:
+              - section: privilege_escalation
+                key: become_exe
+              - section: su_become_plugin
+                key: executable
+            vars:
+              - name: ansible_become_exe
+              - name: ansible_su_exe
+            env:
+              - name: ANSIBLE_BECOME_EXE
+              - name: ANSIBLE_SU_EXE
+        become_flags:
+            description: Options to pass to su
+            default: ''
+            ini:
+              - section: privilege_escalation
+                key: become_flags
+              - section: su_become_plugin
+                key: flags
+            vars:
+              - name: ansible_become_flags
+              - name: ansible_su_flags
+            env:
+              - name: ANSIBLE_BECOME_FLAGS
+              - name: ANSIBLE_SU_FLAGS
+        become_pass:
+            description: Password to pass to su
+            required: False
+            vars:
+              - name: ansible_become_password
+              - name: ansible_become_pass
+              - name: ansible_su_pass
+            env:
+              - name: ANSIBLE_BECOME_PASS
+              - name: ANSIBLE_SU_PASS
+            ini:
+              - section: su_become_plugin
+                key: password
+        prompt_l10n:
+            description:
+                - List of localized strings to match for prompt detection
+                - If empty we'll use the built in one
+            default: []
+            ini:
+              - section: su_become_plugin
+                key: localized_prompts
+            vars:
+              - name: ansible_su_prompt_l10n
+            env:
+              - name: ANSIBLE_SU_PROMPT_L10N
+"""
+
+import re
+
+from ansible.module_utils._text import to_bytes
+from ansible.module_utils.six.moves import shlex_quote
+from ansible.plugins.become import BecomeBase
+
+
+class BecomeModule(BecomeBase):
+
+    name = 'su'
+
+    # messages for detecting prompted password issues
+    fail = ('Authentication failure',)
+
+    SU_PROMPT_LOCALIZATIONS = [
+        'Password',
+        '암호',
+        'パスワード',
+        'Adgangskode',
+        'Contraseña',
+        'Contrasenya',
+        'Hasło',
+        'Heslo',
+        'Jelszó',
+        'Lösenord',
+        'Mật khẩu',
+        'Mot de passe',
+        'Parola',
+        'Parool',
+        'Pasahitza',
+        'Passord',
+        'Passwort',
+        'Salasana',
+        'Sandi',
+        'Senha',
+        'Wachtwoord',
+        'ססמה',
+        'Лозинка',
+        'Парола',
+        'Пароль',
+        'गुप्तशब्द',
+        'शब्दकूट',
+        'సంకేతపదము',
+        'හස්පදය',
+        '密码',
+        '密碼',
+        '口令',
+    ]
+
+    def check_password_prompt(self, b_output):
+        ''' checks if the expected passwod prompt exists in b_output '''
+
+        prompts = self.get_option('prompt_l10n') or self.SU_PROMPT_LOCALIZATIONS
+        b_password_string = b"|".join((br'(\w+\'s )?' + to_bytes(p)) for p in prompts)
+        # Colon or unicode fullwidth colon
+        b_password_string = b_password_string + to_bytes(u' ?(:|：) ?')
+        b_su_prompt_localizations_re = re.compile(b_password_string, flags=re.IGNORECASE)
+        return bool(b_su_prompt_localizations_re.match(b_output))
+
+    def build_become_command(self, cmd, shell):
+        super(BecomeModule, self).build_become_command(cmd, shell)
+
+        # Prompt handling for ``su`` is more complicated, this
+        # is used to satisfy the connection plugin
+        self.prompt = True
+
+        if not cmd:
+            return cmd
+
+        exe = self.get_option('become_exe') or self.name
+        flags = self.get_option('become_flags') or ''
+        user = self.get_option('become_user') or ''
+        success_cmd = self._build_success_command(cmd, shell)
+
+        return "%s %s %s -c %s" % (exe, flags, user, shlex_quote(success_cmd))