Adds support for all Consul 0.8 ACL rule scopes (#25800)

* Added in support for 'agent' and 'node' types.

* Tidies and moves `consul_acl` module closer to PEP8 compliance.

* Switched from using byspoke code to handle py2/3 string issues to using `to_text`.

* Made changes suggested by jrandall in https://github.com/ansible/ansible/pull/23467#pullrequestreview-34021967.

* Refactored consul_acl to support scopes with no pattern (and therefore a different HCL defintion).

* Corrects whitespace in Consul ACL HCL representation.

* Fixes Consul ACL to return the HCL equivalent JSON (according to the Consul docs) for the set ACLs.

* Repositioned import to align with Ansible standard (!= PEP8 standard).

* Adds Python 2.6 compatibility.

* Fixes PEP8 issues.

* Removes consul_acl.py as it now passes PEP8.

* Follows advice in the "Documenting Your Module" guide and moves imports up from the bottom.

* Tidies consul_acl module documentation.

* Updates link to guide about Consul ACLs.

* Removes new line spaces from error message string.

* Provide better error message if user forgets to associate a value to a Consul ACL rule.

* Minor refactoring of Consul ACL module.

* Fixes bug that was breaking idempotence in Consul ACL module.

* Detects redefinition of same rule.

* Adds test to check the Consul ACL module can set rules for all supported scopes.

* Fixes return when updating an ACL.

* Clean up of Consul ACL integration test file.

* Verify correct changes to existing Consul ACL rule.

* Adds tests for idempotence.

* Splits Consul ACL tests into cohesive modules.

* Adds test for deleting Consul ACLs.

* Test that Consul ACL module can set all rule scopes.

* Fixes issues surrounding the creation of ACLs.

Thanks for the comments by manos in https://github.com/ansible/ansible/pull/25800#issuecomment-310137889.

* Stops Consul ACL's name being "forgotten" if ACL updated by token.

* Fixes incorrect assignment when a Consul ACL is deleted.

* Fixes value of `changed` when Consul ACL is removed.

* Fixes tests for Consul ACL.

* Adds interal documentation.

* Refactors to separate update and create (also makes it possible to unit test this module).

* Improves documentation.

* Completes RETURN documentation for Consul ACL module.

* Fixes issue with equality checking for `None` in ACL Consul.

* Fixes Python 2 issue with making a decision based on `str` type.

* Fixes inequality check bug in Python 2.

* Adds tests for setting ACL with token.

* Adds support for creating an ACL with a given token.

* Outputs operation performed on Consul ACL when changed.

* Fixs issue with test for creating a Consul ACL with rules.

* Corrects property used to set ACL token in python-consul library.

* Fixes tear-down issue in test that creates a Consul ACL using a token.

test/integration/consul.yml

@@ -16,7 +16,6 @@
       register: consul_running
 
   roles:
-
     - {role: test_consul_service,
        when: not consul_running.failed is defined}
 
@@ -30,7 +29,6 @@
        when: not consul_running.failed is defined}
 
   tasks:
-
     - name: setup services with passing check for consul inventory test
       consul:
         service_name: nginx
@@ -42,7 +40,6 @@
           - dev
           - master
 
-
     - name: setup failing service for inventory test
       consul:
         service_name: nginx
@@ -69,7 +66,6 @@
         rules:
           - key: ''
             policy: write
-      register: inventory_token
 
     - name: add metadata for the node through kv_store
       consul_kv: "key=ansible/metadata/dc1/consul-1 value='{{metadata_json}}'"

test/integration/roles/test_consul_acl/tasks/create-acl-with-rules.yml

@@ -0,0 +1,77 @@
+---
+
+- name: create an ACL with rules
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+    rules:
+      - event: "bbq"
+        policy: write
+      - key: "foo"
+        policy: read
+      - key: "private"
+        policy: deny
+      - keyring: write
+      - node: "hgs4"
+        policy: write
+      - operator: read
+      - query: ""
+        policy: write
+      - service: "consul"
+        policy: write
+      - session: "standup"
+        policy: write
+  register: created_acl
+
+- name: verify created ACL's rules
+  assert:
+    that:
+      - created_acl.changed
+      - created_acl.operation == "create"
+      - created_acl.token | length == 36
+      - (created_acl.rules | json_query("event.bbq.policy")) == "write"
+      - (created_acl.rules | json_query("key.foo.policy")) == "read"
+      - (created_acl.rules | json_query("key.private.policy")) == "deny"
+      - (created_acl.rules | json_query("keyring")) == "write"
+      - (created_acl.rules | json_query("node.hgs4.policy")) == "write"
+      - (created_acl.rules | json_query("operator")) == "read"
+      - (created_acl.rules | json_query('query."".policy')) == "write"
+      - (created_acl.rules | json_query("service.consul.policy")) == "write"
+      - (created_acl.rules | json_query("session.standup.policy")) == "write"
+
+- name: create same ACL
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+    rules:
+      - event: "bbq"
+        policy: write
+      - key: "foo"
+        policy: read
+      - key: "private"
+        policy: deny
+      - keyring: write
+      - node: "hgs4"
+        policy: write
+      - operator: read
+      - query: ""
+        policy: write
+      - service: "consul"
+        policy: write
+      - session: "standup"
+        policy: write
+  register: doubly_created_acl
+
+- name: verify idempotence when creating ACL
+  assert:
+    that:
+      - not doubly_created_acl.changed
+
+- name: clean up
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ doubly_created_acl.token }}"
+    state: absent

test/integration/roles/test_consul_acl/tasks/create-acl-with-token.yml

@@ -0,0 +1,41 @@
+---
+
+- name: create an ACL with a given token
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+    token: "{{ test_consul_acl_token_id }}"
+    rules:
+      - key: "foo"
+        policy: write
+  register: created_acl
+
+- name: verify ACL created with given token
+  assert:
+    that:
+      - created_acl.changed
+      - created_acl.operation == "create"
+      - created_acl.token == test_consul_acl_token_id
+
+- name: re-create ACL with the token
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ test_consul_acl_token_id }}"
+    rules:
+      - key: "foo"
+        policy: write
+  register: doubly_created_acl
+
+- name: verify idempotence when creating ACL with same token
+  assert:
+    that:
+      - not doubly_created_acl.changed
+
+- name: clean up
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    state: absent

test/integration/roles/test_consul_acl/tasks/create-acl-without-rules.yml

@@ -0,0 +1,35 @@
+---
+
+- name: create a new ACL without rules
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+  register: created_ruleless_acl
+
+- name: verify ACL created without rules
+  assert:
+    that:
+      - created_ruleless_acl.changed
+      - created_ruleless_acl.operation == "create"
+      - created_ruleless_acl.token | length == 36
+      - created_ruleless_acl.rules == {}
+
+- name: create same rule-less ACL
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+  register: doubly_created_ruleless_acl
+
+- name: verify idempotence when creating ruleless ACL tokens
+  assert:
+    that:
+      - not doubly_created_ruleless_acl.changed
+
+- name: clean up
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ doubly_created_ruleless_acl.token }}"
+    state: absent

test/integration/roles/test_consul_acl/tasks/main.yml

@@ -1,42 +1,11 @@
-- name: create a new acl token
-  consul_acl:
-    mgmt_token: '{{mgmt_token}}'
-    host: '{{acl_host}}'
-    name: 'New ACL'
-  register: new_ruleless
+---
 
-- name: verify ruleless key created
-  assert:
-    that:
-      - new_ruleless.token | length == 36
-      - new_ruleless.name == 'New ACL'
+- import_tasks: create-acl-without-rules.yml
 
-- name: add rules to an acl token
-  consul_acl:
-    mgmt_token: '{{mgmt_token}}'
-    host: '{{acl_host}}'
-    name: 'With rule'
-    rules:
-      - key: 'foo'
-        policy: read
-      - key: 'private/foo'
-        policy: deny
-  register: with_rules
+- import_tasks: create-acl-with-rules.yml
 
-- name: verify rules created
-  assert:
-    that:
-      - with_rules.token | length == 36
-      - with_rules.name == 'With rule'
-      - with_rules.rules | match('.*"foo".*')
-      - with_rules.rules | search(pattern='private/foo')
+- import_tasks: create-acl-with-token.yml
 
-- name: clear up
-  consul_acl:
-    mgmt_token: '{{mgmt_token}}'
-    host: '{{acl_host}}'
-    token: '{{item}}'
-    state: absent
-  with_items:
-    - '{{new_ruleless.token}}'
-    - '{{with_rules.token}}'
+- import_tasks: update-acl.yml
+
+- import_tasks: remove-acl.yml

test/integration/roles/test_consul_acl/tasks/remove-acl.yml

@@ -0,0 +1,37 @@
+---
+
+- name: create an ACL
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+  register: created_acl
+
+- name: remove the ACL
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    state: absent
+  register: removed_acl
+
+# TODO: This does little to actually verify that the ACL has been removed
+- name: verify ACL has been removed
+  assert:
+    that:
+      - removed_acl.changed
+      - removed_acl.operation == "remove"
+      - removed_acl.token | length == 36
+
+- name: remove the ACL again
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    state: absent
+  register: doubly_removed_acl
+
+- name: verify idempotence when deleting an ACL
+  assert:
+    that:
+      - not doubly_removed_acl.changed

test/integration/roles/test_consul_acl/tasks/update-acl.yml

@@ -0,0 +1,71 @@
+---
+
+- name: create an ACL
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    name: "{{ test_consul_acl_token_name }}"
+    rules:
+      - key: "foo"
+        policy: read
+  register: created_acl
+
+- name: update ACL's rules
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    rules:
+      - key: "foo"
+        policy: write
+      - key: "moo"
+        policy: deny
+  register: updated_acl
+
+- name: verify updated ACL's rules
+  assert:
+    that:
+      - updated_acl.changed
+      - updated_acl.operation == "update"
+      - updated_acl.token | length == 36
+      - (updated_acl.rules | json_query("key.foo.policy")) == "write"
+      - (updated_acl.rules | json_query("key.moo.policy")) == "deny"
+
+- name: update already updated rule
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    rules:
+      - key: "foo"
+        policy: write
+      - key: "moo"
+        policy: deny
+  register: doubly_updated_acl
+
+- name: verify idempotence when setting rules
+  assert:
+    that:
+      - not doubly_updated_acl.changed
+
+- name: update to remove all ACL's rules
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    rules: []
+  register: updated_acl
+
+- name: verify ACL has no rules
+  assert:
+    that:
+      - updated_acl.changed
+      - updated_acl.token | length == 36
+      - updated_acl.rules == {}
+
+- name: clean up
+  consul_acl:
+    host: "{{ acl_host }}"
+    mgmt_token: "{{ mgmt_token }}"
+    token: "{{ created_acl.token }}"
+    state: absent

test/integration/roles/test_consul_acl/vars/main.yml

@@ -0,0 +1,4 @@
+---
+
+test_consul_acl_token_name: example-token
+test_consul_acl_token_id: 60DEC4BC-DD47-4F4E-A95A-19D639407D2C