add etcd3 lookup plugin (#127)

* add etcd3 lookup plugin

* retire version_added tag

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fix YAML syntax in example

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove python shebang as it is useless in Ansible lookup module

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update plugins/lookup/etcd3.py typo

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fixes:
- replaced LookupBase._display by ansible.utils.display.Display
- add regex to retrieve host and port from ETCDCTL_ENDPOINTS env
- add env support for user, password, timeout

* fixes:
- use short form for types
- update doc section with envs
- catch exceptions between etcd3 api calls

* etcd3 lookup pass ansible sanity checks
introduce ansible integration tests for etcd3 lookup

* extract etcd3 setup from existing etcd3 module integration test

* fix etcd3 module/lookup integration tests

* fixes:
- fix port option in docstring
- raise connecttion error
- fix display format issues
- fix ETCDCTL_ENDPOINTS regex
adds:
- basic unit tests

* fix sanity issues

* add etcd3 lookup plugin

* retire version_added tag

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fix YAML syntax in example

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove python shebang as it is useless in Ansible lookup module

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update plugins/lookup/etcd3.py typo

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fixes:
- replaced LookupBase._display by ansible.utils.display.Display
- add regex to retrieve host and port from ETCDCTL_ENDPOINTS env
- add env support for user, password, timeout

* fixes:
- use short form for types
- update doc section with envs
- catch exceptions between etcd3 api calls

* etcd3 lookup pass ansible sanity checks
introduce ansible integration tests for etcd3 lookup

* extract etcd3 setup from existing etcd3 module integration test

* fix etcd3 module/lookup integration tests

* fixes:
- fix port option in docstring
- raise connecttion error
- fix display format issues
- fix ETCDCTL_ENDPOINTS regex
adds:
- basic unit tests

* fix sanity issues

* changes:
- replace kwargs lookups with get_option()
- add 'entpoint' option for correct handling of ETCDCTL_ENDPOINTS env
- code simplification

* fix etcd3 lookup unit test:
replace LookupModule instanciation with lookup_loader

* fix sanity checks

* etcd3 changes:
- docstring documentation fixes/updates
- create etcd3 cnx object with a get_option() loop instead of copying 'private' class object
- set 'endpoints' option mutually exclusive with 'host' and 'port' (raises an AnsibleError exception)

* etcd3 changes:
- added ANSIBLE_METADATA,
- added default value for 'endpoints' option,
- removed defaults for options 'host' and 'port',
- fixed docstring links,
- added 'notes' and 'seealso' sections in doctring
- updated options code handling to reflect docstring's updates

* etcd3 changes:
- fix descriptions for endpoints, host, and port options
- update notes sections
- fix reference to etcd lookup plugin in seealso section
- fix return docstring
- remove useless logging
- obfuscates password in connection logging

* more pythonic lookup on dict keys

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update password obfuscation

Co-Authored-By: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>

tests/integration/targets/lookup_etcd3/aliases

@@ -0,0 +1,8 @@
+shippable/posix/group1
+destructive
+needs/file/tests/utils/constraints.txt
+needs/target/setup_etcd3
+skip/aix
+skip/osx
+skip/freebsd
+skip/python2.6  # lookups are controller only, and we no longer support Python 2.6 on the controller

tests/integration/targets/lookup_etcd3/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+
+  etcd3_prefix: '/keyprefix/'
+  etcd3_singlekey: '/singlekeypath'

tests/integration/targets/lookup_etcd3/dependencies.yml

@@ -0,0 +1,6 @@
+---
+- hosts: localhost
+  tasks:
+    - name: Setup etcd3
+      import_role:
+        name: setup_etcd3

tests/integration/targets/lookup_etcd3/meta/main.yml

@@ -0,0 +1,2 @@
+dependencies:
+  - setup_pkg_mgr

tests/integration/targets/lookup_etcd3/runme.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+set -eux
+
+ANSIBLE_ROLES_PATH=../ \
+    ansible-playbook dependencies.yml -v "$@"
+
+ANSIBLE_ROLES_PATH=../ \
+    ansible-playbook test_lookup_etcd3.yml -v "$@"

tests/integration/targets/lookup_etcd3/tasks/main.yml

@@ -0,0 +1,22 @@
+---
+# lookup_etcd3 integration tests
+# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+- name: put key/values with an etcd prefix
+  etcd3:
+    key: "{{ etcd3_prefix }}foo{{ item }}"
+    value: "bar{{ item }}"
+    state: present
+  loop:
+  - 1
+  - 2
+  - 3
+
+- name: put a single key/values in etcd
+  etcd3:
+    key: "{{ etcd3_singlekey }}"
+    value: "foobar"
+    state: present
+
+- import_tasks: tests.yml

tests/integration/targets/lookup_etcd3/tasks/tests.yml

@@ -0,0 +1,26 @@
+---
+# lookup_etcd3 integration tests
+# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+- block:
+  - name: 'Fetch secrets using "etcd3" lookup'
+    set_fact:
+      etcdoutkey1: "{{ lookup('community.general.etcd3', etcd3_prefix, prefix=True) }}"
+      etcdoutkey2: "{{ lookup('community.general.etcd3', etcd3_singlekey) }}"
+      key_inexistent: "{{ lookup('community.general.etcd3', 'inexistent_key') }}"
+
+  - name: 'Check etcd values'
+    assert:
+      msg: 'unexpected etcd3 values'
+      that:
+      - etcdoutkey1 is sequence
+      - etcdoutkey1 | length() == 3
+      - etcdoutkey1[0].value == 'bar1'
+      - etcdoutkey1[1].value == 'bar2'
+      - etcdoutkey1[2].value == 'bar3'
+      - etcdoutkey2 is sequence
+      - etcdoutkey2 | length() == 2
+      - etcdoutkey2.value == 'foobar'
+      - key_inexistent is sequence
+      - key_inexistent | length() == 0

tests/integration/targets/lookup_etcd3/test_lookup_etcd3.yml

@@ -0,0 +1,6 @@
+---
+- hosts: localhost
+  tasks:
+    - name: Test lookup etcd3
+      import_role:
+        name: lookup_etcd3

tests/integration/targets/setup_etcd3/defaults/main.yml

@@ -0,0 +1,16 @@
+---
+# setup etcd3 for integration tests on module/lookup
+# (c) 2017,  Jean-Philippe Evrard <jean-philippe@evrard.me>
+# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
+#
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+# # Copyright: (c) 2018, Ansible Project
+#
+etcd3_ver: "v3.2.14"
+etcd3_download_server: "https://storage.googleapis.com/etcd"
+#etcd3_download_server: "https://github.com/coreos/etcd/releases/download"
+etcd3_download_url: "{{ etcd3_download_server }}/{{ etcd3_ver }}/etcd-{{ etcd3_ver }}-linux-amd64.tar.gz"
+etcd3_download_location: /tmp/etcd-download-test
+etcd3_path: "{{ etcd3_download_location }}/etcd-{{ etcd3_ver }}-linux-amd64"
+
+etcd3_pip_module: etcd3>=0.12

tests/integration/targets/setup_etcd3/meta/main.yml

@@ -0,0 +1,2 @@
+dependencies:
+  - setup_pkg_mgr

tests/integration/targets/setup_etcd3/tasks/main.yml

@@ -0,0 +1,112 @@
+---
+# setup etcd3 for integration tests on module/lookup
+# (c) 2017,  Jean-Philippe Evrard <jean-philippe@evrard.me>
+# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# ============================================================
+
+# setup etcd3 for supported distros
+- block:
+
+  - name: python 2
+    set_fact:
+      python_suffix: ""
+    when: ansible_python_version is version('3', '<')
+
+  - name: python 3
+    set_fact:
+      python_suffix: "-py3"
+    when: ansible_python_version is version('3', '>=')
+
+  - include_vars: '{{ item }}'
+    with_first_found:
+      - files:
+          - '{{ ansible_distribution }}-{{ ansible_distribution_major_version }}{{ python_suffix }}.yml'
+          - '{{ ansible_distribution }}-{{ ansible_distribution_version }}{{ python_suffix }}.yml'
+          - '{{ ansible_os_family }}-{{ ansible_distribution_major_version }}{{ python_suffix }}.yml'
+          - '{{ ansible_os_family }}{{ python_suffix }}.yml'
+          - 'default{{ python_suffix }}.yml'
+          - 'default.yml'
+        paths: '../vars'
+
+  - name: Upgrade setuptools python2 module
+    pip:
+      name: setuptools<45
+      extra_args: --upgrade
+      state: present
+    when: python_suffix == ''
+
+  - name: Install etcd3 python modules
+    pip:
+      name: "{{ etcd3_pip_module }}"
+      extra_args: --only-binary grpcio
+      state: present
+
+  # Check if re-installing etcd3 is required
+  - name: Check if etcd3ctl exists for re-use.
+    shell: "ETCDCTL_API=3 {{ etcd3_path }}/etcdctl --endpoints=localhost:2379 get foo"
+    args:
+      executable: /bin/bash
+    changed_when: false
+    failed_when: false
+    register: _testetcd3ctl
+
+  - block:
+    # Installing etcd3
+    - name: If can't reuse, prepare download folder
+      file:
+        path: "{{ etcd3_download_location }}"
+        state: directory
+      register: _etcddownloadexists
+      when:
+        - _testetcd3ctl.rc != 0
+
+    - name: Delete download folder if already exists (to start clean)
+      file:
+        path: "{{ etcd3_download_location }}"
+        state: absent
+      when:
+        - _etcddownloadexists is not changed
+
+    - name: Recreate download folder if purged
+      file:
+        path: "{{ etcd3_download_location }}"
+        state: directory
+      when:
+        - _etcddownloadexists is not changed
+
+    - name: Download etcd3
+      unarchive:
+        src: "{{ etcd3_download_url }}"
+        dest: "{{ etcd3_download_location }}"
+        remote_src: yes
+
+    # Running etcd3 and kill afterwards if it wasn't running before.
+    - name: Run etcd3
+      shell: "{{ etcd3_path }}/etcd &"
+      register: _etcd3run
+      changed_when: true
+
+#    - name: kill etcd3
+#      command: "pkill etcd"
+
+    when:
+      - _testetcd3ctl.rc != 0
+
+  when:
+  - ansible_distribution | lower ~ "-" ~ ansible_distribution_major_version | lower != 'centos-6'

tests/integration/targets/setup_etcd3/vars/RedHat-7.yml

@@ -0,0 +1 @@
+etcd3_pip_module: etcd3<0.12

tests/integration/targets/setup_etcd3/vars/Suse-py3.yml

@@ -0,0 +1,3 @@
+# SuSE's python 3.6.10 comes with six 1.11.0 as distutil
+# we restrict to etcd3 < 0.11 to avoid pip to try to upgrade six
+etcd3_pip_module: 'etcd3<0.11'

tests/integration/targets/setup_etcd3/vars/Suse.yml

@@ -0,0 +1,3 @@
+# SuSE's python 3.6.10 comes with six 1.11.0 as distutil
+# we restrict to etcd3 < 0.11 to avoid pip to try to upgrade six
+etcd3_pip_module: 'etcd3<0.11'

tests/integration/targets/setup_etcd3/vars/default.yml

@@ -0,0 +1,2 @@
+---
+# default should don't touch anything