add etcd3 lookup plugin (#127)

* add etcd3 lookup plugin

* retire version_added tag

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fix YAML syntax in example

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove python shebang as it is useless in Ansible lookup module

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update plugins/lookup/etcd3.py typo

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fixes:
- replaced LookupBase._display by ansible.utils.display.Display
- add regex to retrieve host and port from ETCDCTL_ENDPOINTS env
- add env support for user, password, timeout

* fixes:
- use short form for types
- update doc section with envs
- catch exceptions between etcd3 api calls

* etcd3 lookup pass ansible sanity checks
introduce ansible integration tests for etcd3 lookup

* extract etcd3 setup from existing etcd3 module integration test

* fix etcd3 module/lookup integration tests

* fixes:
- fix port option in docstring
- raise connecttion error
- fix display format issues
- fix ETCDCTL_ENDPOINTS regex
adds:
- basic unit tests

* fix sanity issues

* add etcd3 lookup plugin

* retire version_added tag

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fix YAML syntax in example

Co-Authored-By: Felix Fontein <felix@fontein.de>

* typo fixes

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove python shebang as it is useless in Ansible lookup module

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update plugins/lookup/etcd3.py typo

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fixes:
- replaced LookupBase._display by ansible.utils.display.Display
- add regex to retrieve host and port from ETCDCTL_ENDPOINTS env
- add env support for user, password, timeout

* fixes:
- use short form for types
- update doc section with envs
- catch exceptions between etcd3 api calls

* etcd3 lookup pass ansible sanity checks
introduce ansible integration tests for etcd3 lookup

* extract etcd3 setup from existing etcd3 module integration test

* fix etcd3 module/lookup integration tests

* fixes:
- fix port option in docstring
- raise connecttion error
- fix display format issues
- fix ETCDCTL_ENDPOINTS regex
adds:
- basic unit tests

* fix sanity issues

* changes:
- replace kwargs lookups with get_option()
- add 'entpoint' option for correct handling of ETCDCTL_ENDPOINTS env
- code simplification

* fix etcd3 lookup unit test:
replace LookupModule instanciation with lookup_loader

* fix sanity checks

* etcd3 changes:
- docstring documentation fixes/updates
- create etcd3 cnx object with a get_option() loop instead of copying 'private' class object
- set 'endpoints' option mutually exclusive with 'host' and 'port' (raises an AnsibleError exception)

* etcd3 changes:
- added ANSIBLE_METADATA,
- added default value for 'endpoints' option,
- removed defaults for options 'host' and 'port',
- fixed docstring links,
- added 'notes' and 'seealso' sections in doctring
- updated options code handling to reflect docstring's updates

* etcd3 changes:
- fix descriptions for endpoints, host, and port options
- update notes sections
- fix reference to etcd lookup plugin in seealso section
- fix return docstring
- remove useless logging
- obfuscates password in connection logging

* more pythonic lookup on dict keys

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Update password obfuscation

Co-Authored-By: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>
This commit is contained in:
Eric Belhomme 2020-04-25 10:58:06 +02:00 committed by GitHub
commit 695eed943b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
17 changed files with 510 additions and 0 deletions

View file

@ -0,0 +1,8 @@
shippable/posix/group1
destructive
needs/file/tests/utils/constraints.txt
needs/target/setup_etcd3
skip/aix
skip/osx
skip/freebsd
skip/python2.6 # lookups are controller only, and we no longer support Python 2.6 on the controller

View file

@ -0,0 +1,4 @@
---
etcd3_prefix: '/keyprefix/'
etcd3_singlekey: '/singlekeypath'

View file

@ -0,0 +1,6 @@
---
- hosts: localhost
tasks:
- name: Setup etcd3
import_role:
name: setup_etcd3

View file

@ -0,0 +1,2 @@
dependencies:
- setup_pkg_mgr

View file

@ -0,0 +1,9 @@
#!/usr/bin/env bash
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
set -eux
ANSIBLE_ROLES_PATH=../ \
ansible-playbook dependencies.yml -v "$@"
ANSIBLE_ROLES_PATH=../ \
ansible-playbook test_lookup_etcd3.yml -v "$@"

View file

@ -0,0 +1,22 @@
---
# lookup_etcd3 integration tests
# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
- name: put key/values with an etcd prefix
etcd3:
key: "{{ etcd3_prefix }}foo{{ item }}"
value: "bar{{ item }}"
state: present
loop:
- 1
- 2
- 3
- name: put a single key/values in etcd
etcd3:
key: "{{ etcd3_singlekey }}"
value: "foobar"
state: present
- import_tasks: tests.yml

View file

@ -0,0 +1,26 @@
---
# lookup_etcd3 integration tests
# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
- block:
- name: 'Fetch secrets using "etcd3" lookup'
set_fact:
etcdoutkey1: "{{ lookup('community.general.etcd3', etcd3_prefix, prefix=True) }}"
etcdoutkey2: "{{ lookup('community.general.etcd3', etcd3_singlekey) }}"
key_inexistent: "{{ lookup('community.general.etcd3', 'inexistent_key') }}"
- name: 'Check etcd values'
assert:
msg: 'unexpected etcd3 values'
that:
- etcdoutkey1 is sequence
- etcdoutkey1 | length() == 3
- etcdoutkey1[0].value == 'bar1'
- etcdoutkey1[1].value == 'bar2'
- etcdoutkey1[2].value == 'bar3'
- etcdoutkey2 is sequence
- etcdoutkey2 | length() == 2
- etcdoutkey2.value == 'foobar'
- key_inexistent is sequence
- key_inexistent | length() == 0

View file

@ -0,0 +1,6 @@
---
- hosts: localhost
tasks:
- name: Test lookup etcd3
import_role:
name: lookup_etcd3

View file

@ -0,0 +1,16 @@
---
# setup etcd3 for integration tests on module/lookup
# (c) 2017, Jean-Philippe Evrard <jean-philippe@evrard.me>
# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
#
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
# # Copyright: (c) 2018, Ansible Project
#
etcd3_ver: "v3.2.14"
etcd3_download_server: "https://storage.googleapis.com/etcd"
#etcd3_download_server: "https://github.com/coreos/etcd/releases/download"
etcd3_download_url: "{{ etcd3_download_server }}/{{ etcd3_ver }}/etcd-{{ etcd3_ver }}-linux-amd64.tar.gz"
etcd3_download_location: /tmp/etcd-download-test
etcd3_path: "{{ etcd3_download_location }}/etcd-{{ etcd3_ver }}-linux-amd64"
etcd3_pip_module: etcd3>=0.12

View file

@ -0,0 +1,2 @@
dependencies:
- setup_pkg_mgr

View file

@ -0,0 +1,112 @@
---
# setup etcd3 for integration tests on module/lookup
# (c) 2017, Jean-Philippe Evrard <jean-philippe@evrard.me>
# 2020, SCC France, Eric Belhomme <ebelhomme@fr.scc.com>
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
# ============================================================
# setup etcd3 for supported distros
- block:
- name: python 2
set_fact:
python_suffix: ""
when: ansible_python_version is version('3', '<')
- name: python 3
set_fact:
python_suffix: "-py3"
when: ansible_python_version is version('3', '>=')
- include_vars: '{{ item }}'
with_first_found:
- files:
- '{{ ansible_distribution }}-{{ ansible_distribution_major_version }}{{ python_suffix }}.yml'
- '{{ ansible_distribution }}-{{ ansible_distribution_version }}{{ python_suffix }}.yml'
- '{{ ansible_os_family }}-{{ ansible_distribution_major_version }}{{ python_suffix }}.yml'
- '{{ ansible_os_family }}{{ python_suffix }}.yml'
- 'default{{ python_suffix }}.yml'
- 'default.yml'
paths: '../vars'
- name: Upgrade setuptools python2 module
pip:
name: setuptools<45
extra_args: --upgrade
state: present
when: python_suffix == ''
- name: Install etcd3 python modules
pip:
name: "{{ etcd3_pip_module }}"
extra_args: --only-binary grpcio
state: present
# Check if re-installing etcd3 is required
- name: Check if etcd3ctl exists for re-use.
shell: "ETCDCTL_API=3 {{ etcd3_path }}/etcdctl --endpoints=localhost:2379 get foo"
args:
executable: /bin/bash
changed_when: false
failed_when: false
register: _testetcd3ctl
- block:
# Installing etcd3
- name: If can't reuse, prepare download folder
file:
path: "{{ etcd3_download_location }}"
state: directory
register: _etcddownloadexists
when:
- _testetcd3ctl.rc != 0
- name: Delete download folder if already exists (to start clean)
file:
path: "{{ etcd3_download_location }}"
state: absent
when:
- _etcddownloadexists is not changed
- name: Recreate download folder if purged
file:
path: "{{ etcd3_download_location }}"
state: directory
when:
- _etcddownloadexists is not changed
- name: Download etcd3
unarchive:
src: "{{ etcd3_download_url }}"
dest: "{{ etcd3_download_location }}"
remote_src: yes
# Running etcd3 and kill afterwards if it wasn't running before.
- name: Run etcd3
shell: "{{ etcd3_path }}/etcd &"
register: _etcd3run
changed_when: true
# - name: kill etcd3
# command: "pkill etcd"
when:
- _testetcd3ctl.rc != 0
when:
- ansible_distribution | lower ~ "-" ~ ansible_distribution_major_version | lower != 'centos-6'

View file

@ -0,0 +1 @@
etcd3_pip_module: etcd3<0.12

View file

@ -0,0 +1,3 @@
# SuSE's python 3.6.10 comes with six 1.11.0 as distutil
# we restrict to etcd3 < 0.11 to avoid pip to try to upgrade six
etcd3_pip_module: 'etcd3<0.11'

View file

@ -0,0 +1,3 @@
# SuSE's python 3.6.10 comes with six 1.11.0 as distutil
# we restrict to etcd3 < 0.11 to avoid pip to try to upgrade six
etcd3_pip_module: 'etcd3<0.11'

View file

@ -0,0 +1,2 @@
---
# default should don't touch anything