Refactor gitlab modules (#51141)

* gitlab_group: refactor module

* gitlab_user: refactor module

* gitlab_group, gitlab_user; pylint

* gitlab_project: refactor module

* gitlab_group, gitlab_project, gitlab_user: Enchance modules

- Add generic loop to update object
- Enchance return messages
- PyLint

* gitlab_runner: refactor module

* gitlab_hooks: refactor module

* gitlab_deploy_key: refactor module

* gitlab_group: enchance module and documentation

- Enchange function arguments
- Add check_mode break
- Rewrite module documentation

* gitlab_hook: enchance module and documentation

- Rewrite documentation
- Enchance function parameters
- Rename functions

* gitlab_project: enchance module and documentation

- Rewrite documentation
- Enchance function parameters
- Add try/except on project creation

* gitlab_runner: enchance module and documentation

- Rewrite documentation
- Fix Copyright
- Enchance function arguments
- Add check_mode break
- Add missing function: deletion

* gitlab_user: enchance module and documentation

- Rewrite documentation
- Enchance function parameters
- Add check_mode break
- Add try/except on user creation

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project,
gitlab_runner, gitlab_user: Fix residual bugs

- Fix Copyright
- Fix result messages
- Add missing check_mode break

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project, gitlab_runner, gitlab_user: pylint

* gitlab_runner: Add substitution function for 'cmp' in python3

* unit-test: remove deprecated gitlab module tests

- gitlab_deploy_key
- gitlab_hooks
- gitlab_project

Actually, they can't be reused because of the modification of the way that the module communicate with the Gitlab instance. It doesn't make direct call to the API, now it use a python library that do the job. So using a pytest mocker to test the module won't work.

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project, gitlab_runner, gitlab_user: add copyright

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project, gitlab_runner, gitlab_user: Support old parameters format

* module_utils Gitlab: Edit copyright

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project,
gitlab_runner, gitlab_user: Unifying module inputs

- Rename verify_ssl into validate_certs to match standards
- Remove unused alias parameters
- Unify parameters type and requirement
- Reorder list order

* gitlab_deploy_key, gitlab_group, gitlab_hooks, gitlab_project, gitlab_runner, gitlab_user: Unifying module outputs

- Use standard output parameter "msg" instead of "return"
- Use snail_case for return values instead of camelCase

* validate-module: remove sanity ignore

* BOTMETA: remove gitlab_* test

- This tests need to be completely rewriten because of the refactoring
of these modules
- TodoList Community Wiki was updated

* gitlab_user: Fix group identifier

* gitlab_project: Fix when group was empty

* gitlab_deploy_key: edit return msg

* module_utils gitlab: fall back to user namespace is project not found

* gitlab modules: Add units tests

* unit test: gitlab module fake current user

* gitlab_user: fix access_level verification

* gitlab unit tests: use decoration instead of with statement

* unit tests: gitlab module skip python 2.6

* unit tests: gitlab module skip library import if python 2.6

* gitlab unit tests: use builtin unittest class

* gitlab unit tests: use custom test class

* unit test: gitlab module lint

* unit tests: move gitlab utils

* unit test: gitlab fix imports

* gitlab_module: edit requirement

python-gitlab library require python >= 2.7

* gitlab_module: add myself as author

* gitlab_modules: add python encoding tag

* gitlab_modules: keep consistency between variable name "validate_certs"

* gitlab_modules: enchance documentation

* gitlab_runner: fix syntax error in documentation

* gitlab_module: use basic_auth module_utils and add deprecation warning

* gitlab_module: documentation corrections

* gitlab_module: python lint

* gitlab_module: deprecate options and aliases for ansible 2.10

* gitlab_group: don't use 'local_action' is documentation example

* gitlab_module: correct return messages

* gitlab_module: use module_util 'missing_required_lib' when python library is missing

* gitlab_module: fix typo in function name.

* gitlab_modules: unify return msg on check_mode

* gitlab_modules: don't use deprecated options in examples
This commit is contained in:
Guillaume Martinez 2019-02-07 20:40:14 +01:00 committed by Dag Wieers
parent a682a0292d
commit 959939b866
17 changed files with 2874 additions and 1847 deletions

View file

@ -1,133 +1,137 @@
#!/usr/bin/python
# (c) 2015, Werner Dijkerman (ikben@werner-dijkerman.nl)
# -*- coding: utf-8 -*-
# Copyright: (c) 2019, Guillaume Martinez (guillaume.lunik@gmail.com)
# Copyright: (c) 2015, Werner Dijkerman (ikben@werner-dijkerman.nl)
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
DOCUMENTATION = '''
---
module: gitlab_project
short_description: Creates/updates/deletes Gitlab Projects
description:
- When the project does not exist in Gitlab, it will be created.
- When the project does exists and state=absent, the project will be deleted.
- When changes are made to the project, the project will be updated.
- When the project does not exist in Gitlab, it will be created.
- When the project does exists and state=absent, the project will be deleted.
- When changes are made to the project, the project will be updated.
version_added: "2.1"
author: "Werner Dijkerman (@dj-wasabi)"
author:
- Werner Dijkerman (@dj-wasabi)
- Guillaume Martinez (@Lunik)
requirements:
- pyapi-gitlab python module
- python >= 2.7
- python-gitlab python module
extends_documentation_fragment:
- auth_basic
options:
server_url:
description:
- Url of Gitlab server, with protocol (http or https).
required: true
validate_certs:
description:
- When using https if SSL certificate needs to be verified.
type: bool
default: 'yes'
aliases:
- verify_ssl
login_user:
description:
- Gitlab user name.
login_password:
description:
- Gitlab password for login_user
login_token:
description:
- Gitlab token for logging in.
group:
description:
- The name of the group of which this projects belongs to.
- When not provided, project will belong to user which is configured in 'login_user' or 'login_token'
- When provided with username, project will be created for this user. 'login_user' or 'login_token' needs admin rights.
name:
description:
- The name of the project
required: true
path:
description:
- The path of the project you want to create, this will be server_url/<group>/path
- If not supplied, name will be used.
server_url:
description:
description:
- An description for the project.
issues_enabled:
description:
- Whether you want to create issues or not.
- Possible values are true and false.
type: bool
default: 'yes'
merge_requests_enabled:
description:
- If merge requests can be made or not.
- Possible values are true and false.
type: bool
default: 'yes'
wiki_enabled:
description:
- If an wiki for this project should be available or not.
- Possible values are true and false.
type: bool
default: 'yes'
snippets_enabled:
description:
- If creating snippets should be available or not.
- Possible values are true and false.
type: bool
default: 'yes'
public:
description:
- If the project is public available or not.
- Setting this to true is same as setting visibility_level to 20.
- Possible values are true and false.
type: bool
default: 'no'
visibility_level:
description:
- Private. visibility_level is 0. Project access must be granted explicitly for each user.
- Internal. visibility_level is 10. The project can be cloned by any logged in user.
- Public. visibility_level is 20. The project can be cloned without any authentication.
- Possible values are 0, 10 and 20.
default: 0
import_url:
description:
- Git repository which will be imported into gitlab.
- Gitlab server needs read access to this git repository.
type: bool
default: 'no'
state:
description:
- create or delete project.
- Possible values are present and absent.
default: "present"
choices: ["present", "absent"]
- The URL of the Gitlab server, with protocol (i.e. http or https).
required: true
type: str
login_user:
description:
- Gitlab user name.
type: str
login_password:
description:
- Gitlab password for login_user
type: str
api_token:
description:
- Gitlab token for logging in.
type: str
aliases:
- login_token
group:
description:
- Id or The full path of the group of which this projects belongs to.
type: str
name:
description:
- The name of the project
required: true
type: str
path:
description:
- The path of the project you want to create, this will be server_url/<group>/path
- If not supplied, name will be used.
type: str
description:
description:
- An description for the project.
type: str
issues_enabled:
description:
- Whether you want to create issues or not.
- Possible values are true and false.
type: bool
default: yes
merge_requests_enabled:
description:
- If merge requests can be made or not.
- Possible values are true and false.
type: bool
default: yes
wiki_enabled:
description:
- If an wiki for this project should be available or not.
- Possible values are true and false.
type: bool
default: yes
snippets_enabled:
description:
- If creating snippets should be available or not.
- Possible values are true and false.
type: bool
default: yes
visibility:
description:
- Private. Project access must be granted explicitly for each user.
- Internal. The project can be cloned by any logged in user.
- Public. The project can be cloned without any authentication.
default: private
type: str
choices: ["private", "internal", "public"]
aliases:
- visibility_level
import_url:
description:
- Git repository which will be imported into gitlab.
- Gitlab server needs read access to this git repository.
required: false
type: str
state:
description:
- create or delete project.
- Possible values are present and absent.
default: present
type: str
choices: ["present", "absent"]
'''
EXAMPLES = '''
- name: Delete Gitlab Project
gitlab_project:
server_url: http://gitlab.example.com
api_url: https://gitlab.example.com/
api_token: "{{ access_token }}"
validate_certs: False
login_token: WnUzDsxjy8230-Dy_k
name: my_first_project
state: absent
delegate_to: localhost
- name: Create Gitlab Project in group Ansible
gitlab_project:
server_url: https://gitlab.example.com
api_url: https://gitlab.example.com/
validate_certs: True
login_user: dj-wasabi
login_password: MySecretPassword
api_username: dj-wasabi
api_password: "MySecretPassword"
name: my_first_project
group: ansible
issues_enabled: False
@ -138,257 +142,277 @@ EXAMPLES = '''
delegate_to: localhost
'''
RETURN = '''# '''
RETURN = '''
msg:
description: Success or failure message
returned: always
type: str
sample: "Success"
result:
description: json parsed response from the server
returned: always
type: dict
error:
description: the error message returned by the Gitlab API
returned: failed
type: str
sample: "400: path is already in use"
project:
description: API object
returned: always
type: dict
'''
import os
import traceback
GITLAB_IMP_ERR = None
try:
import gitlab
HAS_GITLAB_PACKAGE = True
except Exception:
GITLAB_IMP_ERR = traceback.format_exc()
HAS_GITLAB_PACKAGE = False
from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils.api import basic_auth_argument_spec
from ansible.module_utils.basic import AnsibleModule, missing_required_lib
from ansible.module_utils._text import to_native
from ansible.module_utils.gitlab import findGroup, findProject
class GitLabProject(object):
def __init__(self, module, git):
def __init__(self, module, gitlab_instance):
self._module = module
self._gitlab = git
self._gitlab = gitlab_instance
self.projectObject = None
def createOrUpdateProject(self, project_exists, group_name, import_url, arguments):
is_user = False
group_id = self.getGroupId(group_name)
if not group_id:
group_id = self.getUserId(group_name)
is_user = True
'''
@param project_name Name of the project
@param namespace Namespace Object (User or Group)
@param options Options of the project
'''
def createOrUpdateProject(self, project_name, namespace, options):
changed = False
if project_exists:
# Edit project
return self.updateProject(group_name, arguments)
# Because we have already call userExists in main()
if self.projectObject is None:
project = self.createProject(namespace, {
'name': project_name,
'path': options['path'],
'description': options['description'],
'issues_enabled': options['issues_enabled'],
'merge_requests_enabled': options['merge_requests_enabled'],
'wiki_enabled': options['wiki_enabled'],
'snippets_enabled': options['snippets_enabled'],
'visibility': options['visibility'],
'import_url': options['import_url']})
changed = True
else:
# Create project
changed, project = self.updateProject(self.projectObject, {
'name': project_name,
'description': options['description'],
'issues_enabled': options['issues_enabled'],
'merge_requests_enabled': options['merge_requests_enabled'],
'wiki_enabled': options['wiki_enabled'],
'snippets_enabled': options['snippets_enabled'],
'visibility': options['visibility']})
self.projectObject = project
if changed:
if self._module.check_mode:
self._module.exit_json(changed=True)
return self.createProject(is_user, group_id, import_url, arguments)
self._module.exit_json(changed=True, msg="Successfully created or updated the project %s" % project_name)
def createProject(self, is_user, user_id, import_url, arguments):
if is_user:
result = self._gitlab.createprojectuser(user_id=user_id, import_url=import_url, **arguments)
else:
group_id = user_id
result = self._gitlab.createproject(namespace_id=group_id, import_url=import_url, **arguments)
if not result:
self._module.fail_json(msg="Failed to create project %r" % arguments['name'])
return result
def deleteProject(self, group_name, project_name):
if self.existsGroup(group_name):
project_owner = group_name
else:
project_owner = self._gitlab.currentuser()['username']
search_results = self._gitlab.searchproject(search=project_name)
for result in search_results:
owner = result['namespace']['name']
if owner == project_owner:
return self._gitlab.deleteproject(result['id'])
def existsProject(self, group_name, project_name):
if self.existsGroup(group_name):
project_owner = group_name
else:
project_owner = self._gitlab.currentuser()['username']
search_results = self._gitlab.searchproject(search=project_name)
for result in search_results:
owner = result['namespace']['name']
if owner == project_owner:
return True
return False
def existsGroup(self, group_name):
if group_name is not None:
# Find the group, if group not exists we try for user
for group in self._gitlab.getall(self._gitlab.getgroups):
if group['name'] == group_name:
return True
user_name = group_name
user_data = self._gitlab.getusers(search=user_name)
for data in user_data:
if 'id' in user_data:
return True
return False
def getGroupId(self, group_name):
if group_name is not None:
# Find the group, if group not exists we try for user
for group in self._gitlab.getall(self._gitlab.getgroups):
if group['name'] == group_name:
return group['id']
def getProjectId(self, group_name, project_name):
if self.existsGroup(group_name):
project_owner = group_name
else:
project_owner = self._gitlab.currentuser()['username']
search_results = self._gitlab.searchproject(search=project_name)
for result in search_results:
owner = result['namespace']['name']
if owner == project_owner:
return result['id']
def getUserId(self, user_name):
user_data = self._gitlab.getusers(search=user_name)
for data in user_data:
if 'id' in data:
return data['id']
return self._gitlab.currentuser()['id']
def to_bool(self, value):
if value:
return 1
else:
return 0
def updateProject(self, group_name, arguments):
project_changed = False
project_name = arguments['name']
project_id = self.getProjectId(group_name, project_name)
project_data = self._gitlab.getproject(project_id=project_id)
for arg_key, arg_value in arguments.items():
project_data_value = project_data[arg_key]
if isinstance(project_data_value, bool) or project_data_value is None:
to_bool = self.to_bool(project_data_value)
if to_bool != arg_value:
project_changed = True
continue
else:
if project_data_value != arg_value:
project_changed = True
if project_changed:
if self._module.check_mode:
self._module.exit_json(changed=True)
return self._gitlab.editproject(project_id=project_id, **arguments)
try:
project.save()
except Exception as e:
self._module.fail_json(msg="Failed update project: %s " % e)
return True
else:
return False
'''
@param namespace Namespace Object (User or Group)
@param arguments Attributs of the project
'''
def createProject(self, namespace, arguments):
if self._module.check_mode:
return True
arguments['namespace_id'] = namespace.id
try:
project = self._gitlab.projects.create(arguments)
except (gitlab.exceptions.GitlabCreateError) as e:
self._module.fail_json(msg="Failed to create project: %s " % to_native(e))
return project
'''
@param project Project Object
@param arguments Attributs of the project
'''
def updateProject(self, project, arguments):
changed = False
for arg_key, arg_value in arguments.items():
if arguments[arg_key] is not None:
if getattr(project, arg_key) != arguments[arg_key]:
setattr(project, arg_key, arguments[arg_key])
changed = True
return (changed, project)
def deleteProject(self):
if self._module.check_mode:
return True
project = self.projectObject
return project.delete()
'''
@param namespace User/Group object
@param name Name of the project
'''
def existsProject(self, namespace, path):
# When project exists, object will be stored in self.projectObject.
project = findProject(self._gitlab, namespace.full_path + '/' + path)
if project:
self.projectObject = project
return True
return False
def deprecation_warning(module):
deprecated_aliases = ['login_token']
module.deprecate("Aliases \'{aliases}\' are deprecated".format(aliases='\', \''.join(deprecated_aliases)), 2.10)
def main():
argument_spec = basic_auth_argument_spec()
argument_spec.update(dict(
server_url=dict(type='str', required=True, removed_in_version=2.10),
login_user=dict(type='str', no_log=True, removed_in_version=2.10),
login_password=dict(type='str', no_log=True, removed_in_version=2.10),
api_token=dict(type='str', no_log=True, aliases=["login_token"]),
group=dict(type='str'),
name=dict(type='str', required=True),
path=dict(type='str'),
description=dict(type='str'),
issues_enabled=dict(type='bool', default=True),
merge_requests_enabled=dict(type='bool', default=True),
wiki_enabled=dict(type='bool', default=True),
snippets_enabled=dict(default=True, type='bool'),
visibility=dict(type='str', default="private", choices=["internal", "private", "public"], aliases=["visibility_level"]),
import_url=dict(type='str'),
state=dict(type='str', default="present", choices=["absent", "present"]),
))
module = AnsibleModule(
argument_spec=dict(
server_url=dict(required=True),
validate_certs=dict(required=False, default=True, type='bool', aliases=['verify_ssl']),
login_user=dict(required=False, no_log=True),
login_password=dict(required=False, no_log=True),
login_token=dict(required=False, no_log=True),
group=dict(required=False),
name=dict(required=True),
path=dict(required=False),
description=dict(required=False),
issues_enabled=dict(default=True, type='bool'),
merge_requests_enabled=dict(default=True, type='bool'),
wiki_enabled=dict(default=True, type='bool'),
snippets_enabled=dict(default=True, type='bool'),
public=dict(default=False, type='bool'),
visibility_level=dict(default="0", choices=["0", "10", "20"]),
import_url=dict(required=False),
state=dict(default="present", choices=["present", 'absent']),
),
supports_check_mode=True
argument_spec=argument_spec,
mutually_exclusive=[
['api_url', 'server_url'],
['api_username', 'login_user'],
['api_password', 'login_password'],
['api_username', 'api_token'],
['api_password', 'api_token'],
['login_user', 'login_token'],
['login_password', 'login_token']
],
required_together=[
['api_username', 'api_password'],
['login_user', 'login_password'],
],
required_one_of=[
['api_username', 'api_token', 'login_user', 'login_token']
],
supports_check_mode=True,
)
if not HAS_GITLAB_PACKAGE:
module.fail_json(msg="Missing required gitlab module (check docs or install with: pip install pyapi-gitlab")
deprecation_warning(module)
server_url = module.params['server_url']
verify_ssl = module.params['validate_certs']
login_user = module.params['login_user']
login_password = module.params['login_password']
login_token = module.params['login_token']
group_name = module.params['group']
api_url = module.params['api_url']
validate_certs = module.params['validate_certs']
api_user = module.params['api_username']
api_password = module.params['api_password']
gitlab_url = server_url if api_url is None else api_url
gitlab_user = login_user if api_user is None else api_user
gitlab_password = login_password if api_password is None else api_password
gitlab_token = module.params['api_token']
group_identifier = module.params['group']
project_name = module.params['name']
project_path = module.params['path']
description = module.params['description']
project_description = module.params['description']
issues_enabled = module.params['issues_enabled']
merge_requests_enabled = module.params['merge_requests_enabled']
wiki_enabled = module.params['wiki_enabled']
snippets_enabled = module.params['snippets_enabled']
public = module.params['public']
visibility_level = module.params['visibility_level']
visibility = module.params['visibility']
import_url = module.params['import_url']
state = module.params['state']
# We need both login_user and login_password or login_token, otherwise we fail.
if login_user is not None and login_password is not None:
use_credentials = True
elif login_token is not None:
use_credentials = False
else:
module.fail_json(msg="No login credentials are given. Use login_user with login_password, or login_token")
if not HAS_GITLAB_PACKAGE:
module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
try:
gitlab_instance = gitlab.Gitlab(url=gitlab_url, ssl_verify=validate_certs, email=gitlab_user, password=gitlab_password,
private_token=gitlab_token, api_version=4)
gitlab_instance.auth()
except (gitlab.exceptions.GitlabAuthenticationError, gitlab.exceptions.GitlabGetError) as e:
module.fail_json(msg="Failed to connect to Gitlab server: %s" % to_native(e))
except (gitlab.exceptions.GitlabHttpError) as e:
module.fail_json(msg="Failed to connect to Gitlab server: %s. \
Gitlab remove Session API now that private tokens are removed from user API endpoints since version 10.2." % to_native(e))
# Set project_path to project_name if it is empty.
if project_path is None:
project_path = project_name.replace(" ", "_")
# Gitlab API makes no difference between upper and lower cases, so we lower them.
project_name = project_name.lower()
project_path = project_path.lower()
if group_name is not None:
group_name = group_name.lower()
gitlab_project = GitLabProject(module, gitlab_instance)
# Lets make an connection to the Gitlab server_url, with either login_user and login_password
# or with login_token
try:
if use_credentials:
git = gitlab.Gitlab(host=server_url, verify_ssl=verify_ssl)
git.login(user=login_user, password=login_password)
else:
git = gitlab.Gitlab(server_url, token=login_token, verify_ssl=verify_ssl)
except Exception as e:
module.fail_json(msg="Failed to connect to Gitlab server: %s " % to_native(e))
if group_identifier:
group = findGroup(gitlab_instance, group_identifier)
if group is None:
module.fail_json(msg="Failed to create project: group %s doesn't exists" % group_identifier)
# Check if user is authorized or not before proceeding to any operations
# if not, exit from here
auth_msg = git.currentuser().get('message', None)
if auth_msg is not None and auth_msg == '401 Unauthorized':
module.fail_json(msg='User unauthorized',
details="User is not allowed to access Gitlab server "
"using login_token. Please check login_token")
# Validate if project exists and take action based on "state"
project = GitLabProject(module, git)
project_exists = project.existsProject(group_name, project_name)
# Creating the project dict
arguments = {"name": project_name,
"path": project_path,
"description": description,
"issues_enabled": project.to_bool(issues_enabled),
"merge_requests_enabled": project.to_bool(merge_requests_enabled),
"wiki_enabled": project.to_bool(wiki_enabled),
"snippets_enabled": project.to_bool(snippets_enabled),
"public": project.to_bool(public),
"visibility_level": int(visibility_level)}
if project_exists and state == "absent":
project.deleteProject(group_name, project_name)
module.exit_json(changed=True, result="Successfully deleted project %s" % project_name)
namespace = gitlab_instance.namespaces.get(group.id)
project_exists = gitlab_project.existsProject(namespace, project_path)
else:
if state == "absent":
module.exit_json(changed=False, result="Project deleted or does not exist")
user = gitlab_instance.users.list(username=gitlab_instance.user.username)[0]
namespace = gitlab_instance.namespaces.get(user.id)
project_exists = gitlab_project.existsProject(namespace, project_path)
if state == 'absent':
if project_exists:
gitlab_project.deleteProject()
module.exit_json(changed=True, msg="Successfully deleted project %s" % project_name)
else:
if project.createOrUpdateProject(project_exists, group_name, import_url, arguments):
module.exit_json(changed=True, result="Successfully created or updated the project %s" % project_name)
else:
module.exit_json(changed=False)
module.exit_json(changed=False, msg="Project deleted or does not exists")
if state == 'present':
if gitlab_project.createOrUpdateProject(project_name, namespace, {
"path": project_path,
"description": project_description,
"issues_enabled": issues_enabled,
"merge_requests_enabled": merge_requests_enabled,
"wiki_enabled": wiki_enabled,
"snippets_enabled": snippets_enabled,
"visibility": visibility,
"import_url": import_url}):
module.exit_json(changed=True, msg="Successfully created or updated the project %s" % project_name, project=gitlab_project.projectObject._attrs)
else:
module.exit_json(changed=False, msg="No need to update the project %s" % project_name, project=gitlab_project.projectObject._attrs)
if __name__ == '__main__':