Transition inventory into plugins (#23001)

* draft new inventory plugin arch, yaml sample

 - split classes, moved out of init
 - extra debug statements
 - allow mulitple invenotry files
 - dont add hosts more than once
 - simplified host vars
 - since now we can have multiple, inventory_dir/file needs to be per host
 - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
 - centralized localhost handling
 - added plugin docs
 - leaner meaner inventory (split to data + manager)
 - moved noop vars plugin
 - added 'postprocessing' inventory plugins
 - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
 - grouphost_vars loading as inventory plugin (postprocessing)
 - playbook_dir allways full path
 - use bytes for file operations
 - better handling of empty/null sources
 - added test target that skips networking modules
 - now var manager loads play group/host_vars independant from inventory
 - centralized play setup repeat code
 - updated changelog with inv features
 - asperioribus verbis spatium album
 - fixed dataloader to new sig
 - made yaml plugin more resistant to bad data
 - nicer error msgs
 - fixed undeclared group detection
 - fixed 'ungrouping'
 - docs updated s/INI/file/ as its not only format
 - made behaviour of var merge a toggle
 - made 'source over group' path follow existing rule for var precedence
 - updated add_host/group from strategy
 - made host_list a plugin and added it to defaults
 - added advanced_host_list as example variation
 - refactored 'display' to be availbe by default in class inheritance
 - optimized implicit handling as per @pilou's feedback
 - removed unused code and tests
 - added inventory cache and vbox plugin now uses it
 - added _compose method for variable expressions in plugins
 - vbox plugin now uses 'compose'
 - require yaml extension for yaml
 - fix for plugin loader to always add original_path, even when not using all()
 - fix py3 issues
 - added --inventory as clearer option
 - return name when stringifying host objects
 - ajdust checks to code moving

* reworked vars and vars precedence
 - vars plugins now load group/host_vars dirs
 - precedence for host vars is now configurable
 - vars_plugins been reworked
 - removed unused vars cache
 - removed _gathered_facts as we are not keeping info in host anymore
 - cleaned up tests
 - fixed ansible-pull to work with new inventory
 - removed version added notation to please rst check
 - inventory in config relative to config
 - ensures full paths on passed inventories

* implicit localhost connection local

lib/ansible/executor/play_iterator.py

@@ -22,11 +22,9 @@ __metaclass__ = type
 import fnmatch
 
 from ansible import constants as C
-from ansible.errors import AnsibleError
 from ansible.module_utils.six import iteritems
 from ansible.playbook.block import Block
 from ansible.playbook.task import Task
-from ansible.playbook.role_include import IncludeRole
 
 
 boolean = C.mk_boolean
@@ -205,10 +203,6 @@ class PlayIterator:
         start_at_matched = False
         for host in inventory.get_hosts(self._play.hosts):
             self._host_states[host.name] = HostState(blocks=self._blocks)
-            # if the host's name is in the variable manager's fact cache, then set
-            # its _gathered_facts flag to true for smart gathering tests later
-            if host.name in variable_manager._fact_cache and variable_manager._fact_cache.get(host.name).get('module_setup', False):
-                host._gathered_facts = True
             # if we're looking to start at a specific task, iterate through
             # the tasks for this host until we find the specified task
             if play_context.start_at_task is not None and not start_at_done:
@@ -265,7 +259,6 @@ class PlayIterator:
             display.debug("host %s is done iterating, returning" % host.name)
             return (s, None)
 
-        old_s = s
         (s, task) = self._get_next_task_from_state(s, host=host, peek=peek)
 
         if not peek:
@@ -310,16 +303,12 @@ class PlayIterator:
 
                     if (gathering == 'implicit' and implied) or \
                        (gathering == 'explicit' and boolean(self._play.gather_facts)) or \
-                       (gathering == 'smart' and implied and not host._gathered_facts):
+                       (gathering == 'smart' and implied and not (variable_manager._fact_cache.get(host.name,{}).get('module_setup', False))):
                         # The setup block is always self._blocks[0], as we inject it
                         # during the play compilation in __init__ above.
                         setup_block = self._blocks[0]
                         if setup_block.has_tasks() and len(setup_block.block) > 0:
                             task = setup_block.block[0]
-                            if not peek:
-                                # mark the host as having gathered facts, because we're
-                                # returning the setup task to be executed
-                                host.set_gathered_facts(True)
                 else:
                     # This is the second trip through ITERATING_SETUP, so we clear
                     # the flag and move onto the next block in the list while setting