Allow terraform module to specify complex variable structures (#4797)

* Adding capability to specify complex variables type to terraform

* Terrform variable types are mapped to ansible veriable types

* Currently handles Dict, List, Str, Int, Bool types

* Updated the documentation accordingly

* Updated with an example.

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Wonder how that missed the PEP8 checks :).

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Adding the changelog fragment

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Adding ``integer_types`` from ``module_utils``

Simplified the ``integer_types``,  ``str`` and ``float`` value population through ``json.dumps()``.  Now the strings can have special characters which can break the module execution.

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Felix Fontein <felix@fontein.de>

* * Changed to approach to make the code more readble and simple to understand.

    * Maintaining the original for loop for the top_level variables. Therefore the rocess_conplex_args() now only handle second level variables when the type() is either Dict or List.

    * Json dumps are used only for the low level variables. Terraform CLI had issues interpreting escape sequecences from json.dumps()

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* adding boolean explicitly, although boolean is a subclass of integer, adding this for self documentation pupose and the clarity of the code.

* fixing the doc strings

* Update terraform.py

Fixing docstrings

* * Introducing format_args funtion to simplify formatting each argument type for top_level and lower level.

* Terraform Lists of strings, numbers, objects and lists are supported.

* Adding COMMAND: to the fail_json msg, for plan failures to help troubleshoot command line arguments.

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* * Adding full terraform command to fail_json() when the terrafor plan fails
    * Fixing a spelling mistake.

* plan_command if a list, stringifying the list

* * Fixing the new line for the change fragments

* Removed CR (\r) from the output messages. Now output lines carry only LF (\n), not CRLF (\r\n).

* Added integration testing for complex variables.

* Restructured integration testing code to be more expandable.

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>

* double-quotes are not properly escaped in shell, and python string
escaping are nullified the way terraform handle second tier string
variables (within terraform).

* changing all the task actions to FQCN format.

* integration testing now includes:

1. Top level strings containing, special shell characters, spaces,
   double-quotes.

2. Second level strings containing, special shell characters, spaces,
   double-quotes repeating double-quotes to ensure proper regex
substitution.

* Adding colon ':' to string test casses.

* Added complex_vars to switch between the old and the new variable
interpretations.
Updated the documentations to reflect the changes.
Updated the examples.
Handling '\' as well with the escape sequence.

* Added tests for the new escape sequences.
Added multilines tests.

* Restructuring the documente strings to a shorter string.
Argument_spec changed to 'bool'

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>

tests/integration/targets/terraform/files/complex_variables/main.tf

@@ -0,0 +1,35 @@
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+resource "null_resource" "mynullresource" {
+  triggers = {
+    # plain dictionaries
+    dict_name = var.dictionaries.name
+    dict_age  = var.dictionaries.age
+
+    # list of dicrs
+    join_dic_name = join(",", var.list_of_objects.*.name)
+
+    # list-of-strings
+    join_list = join(",", var.list_of_strings.*)
+
+    # testing boolean
+    name = var.boolean ? var.dictionaries.name : var.list_of_objects[0].name
+
+    # top level string
+    sample_string_1 = var.string_type
+
+    # nested lists
+    num_from_matrix = var.list_of_lists[1][2]
+  }
+
+}
+
+output "string_type" {
+  value = var.string_type
+}
+
+output "multiline_string" {
+  value = var.multiline_string
+}

tests/integration/targets/terraform/files/complex_variables/variables.tf

@@ -0,0 +1,62 @@
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+variable "dictionaries" {
+  type = object({
+    name = string
+    age  = number
+  })
+  description = "Same as ansible Dict"
+  default = {
+    age = 1
+    name = "value"
+  }
+}
+
+variable "list_of_strings" {
+  type        = list(string)
+  description = "list of strings"
+  validation {
+    condition     = (var.list_of_strings[1] == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "list_of_objects" {
+  type = list(object({
+    name = string
+    age  = number
+  }))
+  validation {
+    condition     = (var.list_of_objects[1].name == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "boolean" {
+  type        = bool
+  description = "boolean"
+
+}
+
+variable "string_type" {
+  type = string
+  validation {
+    condition     = (var.string_type == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "multiline_string" {
+  type = string
+  validation {
+    condition     = (var.multiline_string == "one\ntwo\n")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "list_of_lists" {
+  type = list(list(any))
+  default = [ [ 1 ], [1, 2, 3], [3] ]
+}

tests/integration/targets/terraform/tasks/complex_variables.yml

@@ -0,0 +1,60 @@
+---
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+- name: Create terraform project directory (complex variables)
+  ansible.builtin.file:
+    path: "{{ terraform_project_dir }}/complex_vars"
+    state: directory
+    mode: 0755
+
+- name: copy terraform files to work space
+  ansible.builtin.copy:
+    src: "complex_variables/{{ item }}"
+    dest: "{{ terraform_project_dir }}/complex_vars/{{ item }}"
+  with_items:
+    - main.tf
+    - variables.tf
+
+# This task would test the various complex variable structures of the with the
+# terraform null_resource
+- name: test complex variables
+  community.general.terraform:
+    project_path: "{{ terraform_project_dir }}/complex_vars"
+    binary_path: "{{ terraform_binary_path }}"
+    force_init: yes
+    complex_vars: true
+    variables:
+      dictionaries:
+        name: "kosala"
+        age: 99
+      list_of_strings:
+        - "kosala"
+        - 'cli specials"&$%@#*!(){}[]:"" \\'
+        - "xxx"
+        - "zzz"
+      list_of_objects:
+        - name: "kosala"
+          age: 99
+        - name: 'cli specials"&$%@#*!(){}[]:"" \\'
+          age: 0.1
+        - name: "zzz"
+          age: 9.789
+        - name: "lll"
+          age: 1000
+      boolean: true
+      string_type: 'cli specials"&$%@#*!(){}[]:"" \\'
+      multiline_string: |
+        one
+        two
+      list_of_lists:
+        - [ 1 ]
+        - [ 11, 12, 13 ]
+        - [ 2 ]
+        - [ 3 ]
+    state: present
+  register: terraform_init_result
+
+- assert:
+    that: terraform_init_result is not failed

tests/integration/targets/terraform/tasks/main.yml

@@ -9,17 +9,17 @@
 - name: Check for existing Terraform in path
   block:
   - name: Check if terraform is present in path
-    command: "command -v terraform"
+    ansible.builtin.command: "command -v terraform"
     register: terraform_binary_path
     ignore_errors: true
 
   - name: Check Terraform version
-    command: terraform version
+    ansible.builtin.command: terraform version
     register: terraform_version_output
     when: terraform_binary_path.rc == 0
 
   - name: Set terraform version
-    set_fact:
+    ansible.builtin.set_fact:
       terraform_version_installed: "{{ terraform_version_output.stdout | regex_search('(?!Terraform.*v)([0-9]+\\.[0-9]+\\.[0-9]+)') }}"
     when: terraform_version_output.changed
 
@@ -30,7 +30,7 @@
   block:
 
   - name: Install Terraform
-    debug:
+    ansible.builtin.debug:
       msg: "Installing terraform {{ terraform_version }}, found: {{ terraform_version_installed | default('no terraform binary found') }}."
 
   - name: Ensure unzip is present
@@ -39,7 +39,7 @@
       state: present
 
   - name: Install Terraform binary
-    unarchive:
+    ansible.builtin.unarchive:
       src: "{{ terraform_url }}"
       dest: "{{ remote_tmp_dir }}"
       mode: 0755
@@ -52,22 +52,16 @@
 # path from the 'Check if terraform is present in path' task, and lastly, the fallback path.
 
 - name: Set path to terraform binary
-  set_fact:
+  ansible.builtin.set_fact:
     terraform_binary_path: "{{ terraform_binary_path.stdout or remote_tmp_dir ~ '/terraform' }}"
 
-- name: Create terraform project directory
-  file:
-    path: "{{ terraform_project_dir }}/{{ item['name'] }}"
-    state: directory
-    mode: 0755
-  loop: "{{ terraform_provider_versions }}"
-  loop_control:
-    index_var: provider_index
-
 - name: Loop over provider upgrade test tasks
-  include_tasks: test_provider_upgrade.yml
+  ansible.builtin.include_tasks: test_provider_upgrade.yml
   vars:
     tf_provider: "{{ terraform_provider_versions[provider_index] }}"
   loop: "{{ terraform_provider_versions }}"
   loop_control:
     index_var: provider_index
+
+- name: Test Complex Varibles
+  ansible.builtin.include_tasks: complex_variables.yml

tests/integration/targets/terraform/tasks/test_provider_upgrade.yml

@@ -3,6 +3,15 @@
 # GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
 # SPDX-License-Identifier: GPL-3.0-or-later
 
+- name: Create terraform project directory (provider upgrade)
+  file:
+    path: "{{ terraform_project_dir }}/{{ item['name'] }}"
+    state: directory
+    mode: 0755
+  loop: "{{ terraform_provider_versions }}"
+  loop_control:
+    index_var: provider_index
+
 - name: Output terraform provider test project
   ansible.builtin.template:
     src: templates/provider_test/main.tf.j2