Stricter module documentation validation (#22353)

Raise the bar for module `DOCUMENTAION`
This validator update was used to find the issues in https://github.com/ansible/ansible/pull/22297/files

**Validation**
* Updated Validation and docs to enforce more (items fixed in https://github.com/ansible/ansible/pull/22297/files)
* Use `suboptions` to document complex options 
* Validate module name
* Validate deprecated modules have correct ANSIBLE_METADATA

**Module Documentation Generation**
* Document `suboptions:` Example https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
* Tidy up HTML generation (valid HTML, no empty lists, etc)
 
**Documentation**
* Clarify the steps for deprecating a module
* Use correct RST headings
* Document `suboptions:` (options)
* Document `contains:` (returns)


**Details**
The aim is to get this (and corresponding module updates) complete by the time `devel` becomes `2.4`, as this allows us to raise the bar for new modules

Example `suboptions` https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718

The aim is to get this PR integrated into `devel` *before* we branch `stable-2.3`, this will allows us to:
* Raise the bar for new modules in 2.4
* Ensure the generated module documentation for 2.3 and higher is improved, important as we will be doing versioned docs moving forward.

test/sanity/validate-modules/schema.py

@@ -16,40 +16,84 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from voluptuous import ALLOW_EXTRA, Any, Required, Schema
+from voluptuous import PREVENT_EXTRA, Any, Required, Schema
+
+suboption_schema = Schema(
+    {
+        Required('description'): Any(basestring, [basestring]),
+        'required': bool,
+        'choices': list,
+        'aliases': Any(basestring, list),
+        'version_added': Any(basestring, float),
+        'default': Any(None, basestring, float, int, bool, list, dict),
+        # Note: Types are strings, not literal bools, such as True or False
+        'type': Any(None, "bool")
+    },
+    extra=PREVENT_EXTRA
+)
 
 option_schema = Schema(
     {
         Required('description'): Any(basestring, [basestring]),
         'required': bool,
         'choices': list,
-        'aliases': list,
-        'version_added': Any(basestring, float)
-    },
-    extra=ALLOW_EXTRA
-)
-
-doc_schema = Schema(
-    {
-        Required('module'): basestring,
-        'short_description': basestring,
-        'description': Any(basestring, [basestring]),
+        'aliases': Any(basestring, list),
         'version_added': Any(basestring, float),
-        'author': Any(None, basestring, [basestring]),
-        'notes': Any(None, [basestring]),
-        'requirements': [basestring],
-        'options': Any(None, dict),
-        'extends_documentation_fragment': Any(basestring, [basestring])
+        'default': Any(None, basestring, float, int, bool, list, dict),
+        'suboptions': Any(None, {basestring: suboption_schema,}),
+        # Note: Types are strings, not literal bools, such as True or False
+        'type': Any(None, "bool")
     },
-    extra=ALLOW_EXTRA
+    extra=PREVENT_EXTRA
 )
 
-metadata_schema = Schema(
-    {
-        Required('status'): [Any('stableinterface', 'preview', 'deprecated',
-                                 'removed')],
-        Required('version'): '1.0',
-        Required('supported_by'): Any('core', 'community', 'unmaintained',
-                                      'committer')
-    }
-)
+def doc_schema(module_name):
+    if module_name.startswith('_'):
+        module_name = module_name[1:]
+    return Schema(
+        {
+            Required('module'): module_name,
+            'deprecated': basestring,
+            Required('short_description'): basestring,
+            Required('description'): Any(basestring, [basestring]),
+            Required('version_added'): Any(basestring, float),
+            Required('author'): Any(None, basestring, [basestring]),
+            'notes': Any(None, [basestring]),
+            'requirements': [basestring],
+            'todo': Any(None, basestring, [basestring]),
+            'options': Any(None, {basestring: option_schema}),
+            'extends_documentation_fragment': Any(basestring, [basestring])
+        },
+        extra=PREVENT_EXTRA
+    )
+
+def metadata_schema(deprecated):
+    valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
+    if deprecated:
+        valid_status = Any('deprecated')
+
+    return Schema(
+        {
+            Required('status'): [valid_status],
+            Required('version'): '1.0',
+            Required('supported_by'): Any('core', 'community', 'unmaintained',
+                                          'committer')
+        }
+    )
+
+
+
+# Things to add soon
+####################
+# 1) Validate RETURN, including `contains` if `type: complex`
+#    This will improve documentation, though require fair amount of module tidyup
+
+# Possible Future Enhancements
+##############################
+
+# 1) Don't allow empty options for choices, aliases, etc
+# 2) If type: bool ensure choices isn't set - perhaps use Exclusive
+# 3) both version_added should be quoted floats
+# 4) Use Recursive Schema: https://github.com/alecthomas/voluptuous/issues/128 though don't allow two layers
+
+#  Tool that takes JSON and generates RETURN skeleton (needs to support complex structures)