ansible-collections/community.mysql
1
0
Fork 0
mirror of https://github.com/ansible-collections/community.mysql.git synced 2025-07-27 23:21:27 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

[PR #490/6970aef backport][stable-2] Integrations tests : Use containers (#519)

Browse source 
* Integrations tests : Use containers for more control and verify that versions match expectation (#490)

* Add healthcheck to MariaDB before starting the tests
This prevent the first test to fail because the db isn't ready yet.

* Add % instead of the default 'localhost' since we use remote connection

Previously, everything was on localhost. Now ansible-test is in a
venv and the db is in a container. The db see the IP address from the
podman host (10.88.0.2)

* Add mysql_client to the controller

* Prepare controller with Podman/Docker Network

We use the Podman/Docker network gateway address to communicate between
container. I haven't tested Docker. I would have preferred to use a pod
but only Podman support it and ansible-test only support the
--docker-network option.

* Generalize mysql and mariadb version based on container name

This way we can split db_engine and db_version and simplify tests.
Also this is mandatory to use the matrix.db_engine_version as the
image name for our services containers.

* Fix replication due to usage of gateway_addr instead of localhost

* Refactor setup_mysql into setup_controller

* Fix server_id in GHA

GHA lack a way to pass option to docker's command. Also server_id is
not read as a environment variable. So I'm forced to use a config file.

* Refactor test_mysql_user to work with other host than localhost

* Refactor way tests info are passed from sed to file with lookup

The idea is to avoid modifying test targets from the workflow to prevent
ansible-test to think every tests needs to be run.

* Refactor test to use the db_version from setup_controller

* Add temporary files to .gitignore

* Add back docker healthcheck on services

I saw in the GHA logs that it perform an healtcheck ! So I hope this
will work.

* Refactor the way server_id is set for replicas

The simple way is to add '--server-id 2' after the name of the image of
the container. But GHA doesn't let us do that. The idea of mount a file
from our repo doesn't work because the repo is check out later in the
workflow and I failed to find a pre-job hook. Then I realized that this
MySQL option is dynamic! So we will set that in the test target!

* Add IF NOT EXISTS to prevent misleading error on retry

* Cut python 3.11 not supported by ansible-test yet

* Add option to run only a single target

* Disable replication with channel tests entirely for MySQL 5.7

* Activate Mysql 8 and Mariadb into GitHub Action Workflow

* Document run_all_tests.py

* Fix replication settings

sh don't know 'echo -e', so we use bash instead.
Also, we need to wait for the container to be healthy before trying to
restart it. Otherwise that could corrupt it.

* Add more descriptive tests names

* Use mysql_host var name instead of gateway_addr in tests

* Refactor user@<gateway ip addr> into user@%

* Workaround for plugin role that fails with any MariaDB versions

* Ensure replicas are healthy before rebooting them

* Add a virtualenv for ansible-test used locally

* Cut column-statistics disabling

Thanks to our test-container, we now use the correspond mysql-client.
So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we
use mysql-client-8.

* Fix test matrix

Python version should be quoted, otherwise 3.10 become 3.1
We can skip 2.14 and devel with Python3.8
We can skip devel with Python 3.9
We can skip MariaDB 10.4 with mysql-client-10.6
Add tests for MariaDB 10.6, 10.7 and 10.8

* Fix queries for roles

* Add filter for issues resolved in newer version of mysqlclient

* Add names to tests

* Cut tests for incompatible MySQL 8 and pymysql 0.7.11

* Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok)

* Change docker-image workflow to work on all images using matrix

* Add support for version of mysqlclient

* Fix verify database version

Sometimes, version_full contains trailing information (-log). To prevent
issues it's best to concatenate major and minor version.

* Cut filter for tests now that the right connector is used

* Add clean up in "always" phase of the block

Because our tests use --retry-on-error, and the first thing the test
does is to try to create the database. We must cleanup otherwise if
there is a retry, it will throw a misleading "database already exists"
error.

* Disable tests using pymysql 1.0.2

Many tests are failing but this must be fixed in the plugins in a future
PR.

* Cut test MySQL 8 with incompatible pymysql 0.7.11

It fails to connect with error about cryptography unsupported

* Fix missing cffi package to connect to MySQL 8 using Python 3.9

* Split Docker image workflow to rebuild only changed Dockerfile

My goal is not to save the planet but to make it work. Currently
docker/setup-buildx-action@v2 often fails. You have to rerun the
workflow multiple times until it succeed. When you do that with the
matrix with 15 containers, you never get to the point where they all
built successfully. Having separate workflows makes rerun the failing
build easier.

* Add option to let containers alive at end of testing

* Migrate tests documentations in their own file

* Document usage of continue_on_errors

* Add support for systems with unsupported python set as default

* Add create podman network for system missing it.

We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't
find in which release the default network changed and maybe it's
defined in the Linux distribution. So in doubt I always attempt to
create the network.

* Add full path to image to prevent podman asking which registry to use

* Add options to enforce recreate containers even if already exists

* Add deletion of anonymous volumes associated with the container

* Change shebang from python to python3 to avoid confusion with python2

This script is a python3 script.

* Add disk and RAM requirements

* Cut the 3 from python command to follow shebang recommendations

https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html

* Reformat file path

Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>

* Move utility task files in their own folder

* Add called workflow file in the GHA hooks

Without this, the containers are not rebuilt when you modify the file
built-docker-image.yml.

* Rollback to github.repository in container image name

This time I think I understood. We publish in the
github.repository_owner's namespace. In my case it's laurent-indermuehle
and in case of upstream it's ansible-collections. A proof of that:
https://github.com/orgs/ansible-collections/packages <- here there is
one attempt I did in february to push my branch to the upstream.
So, our tests containers will be visible to the whole community, not
just community.mysql.

---------

Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
(cherry picked from commit 6970aef8f6)

* Add changelog fragment

* Disable tests that doesn't work on stable-1

It's shameful to disable tests, but they didn't makes much sense
anyway.

* Fix error message being different on stable-1 than v3

* Disable tests against MariaDB 10.5 and 10.6

Does version have never been tested on stable-1 and many tests fails.
This commit is contained in:
Laurent Indermühle 2023-03-29 08:56:52 +02:00 • committed by GitHub
commit 530e6c3d9c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
122 changed files with 3786 additions and 2639 deletions
Download patch file Download diff file Expand all files Collapse all files

2
tests/integration/targets/test_mysql_replication/defaults/main.yml
View file

@ -1,6 +1,6 @@
mysql_user: root
mysql_password: msandbox
mysql_host: 127.0.0.1
mysql_host: '{{ gateway_addr }}'
mysql_primary_port: 3307
mysql_replica1_port: 3308
mysql_replica2_port: 3309

2
tests/integration/targets/test_mysql_replication/meta/main.yml
View file

@ -1,3 +1,3 @@
---
dependencies:
- setup_mysql
- setup_controller

0
tests/integration/targets/test_mysql_replication/tasks/issue-265.yml Normal file
View file

18
tests/integration/targets/test_mysql_replication/tasks/issue-28.yml
View file

@ -1,7 +1,4 @@
---
- name: alias mysql command to include default options
set_fact:
mysql_command: "mysql -u{{ mysql_user }} -p{{ mysql_password }} -P{{ mysql_primary_port }} --protocol=tcp"
- name: set fact tls_enabled
command: "{{ mysql_command }} \"-e SHOW VARIABLES LIKE 'have_ssl';\""
@ -13,7 +10,7 @@
mysql_parameters: &mysql_params
login_user: '{{ mysql_user }}'
login_password: '{{ mysql_password }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
login_port: '{{ mysql_primary_port }}'
when: tls_enabled
block:
@ -29,6 +26,7 @@
mysql_user:
<<: *mysql_params
name: '{{ user_name_1 }}'
host_all: true
state: absent
ignore_errors: yes
@ -46,7 +44,7 @@
mode: getprimary
login_user: '{{ user_name_1 }}'
login_password: '{{ user_password_1 }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
login_port: '{{ mysql_primary_port }}'
ca_cert: /tmp/cert.pem
register: result
@ -55,19 +53,21 @@
- assert:
that:
- result is failed
when: connector_name is search('pymysql')
when:
- connector_name == 'pymysql'
- assert:
that:
- result is succeeded
when: connector_name is not search('pymysql')
when:
- connector_name != 'pymysql'
- name: attempt connection with newly created user ignoring hostname
mysql_replication:
mode: getprimary
login_user: '{{ user_name_1 }}'
login_password: '{{ user_password_1 }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
login_port: '{{ mysql_primary_port }}'
ca_cert: /tmp/cert.pem
check_hostname: no
@ -82,5 +82,5 @@
mysql_user:
<<: *mysql_params
name: '{{ user_name_1 }}'
host: 127.0.0.1
host: '{{ gateway_addr }}'
state: absent

3
tests/integration/targets/test_mysql_replication/tasks/main.yml
View file

@ -15,7 +15,8 @@
# Tests of channel parameter:
- import_tasks: mysql_replication_channel.yml
when:
- install_type == 'mysql' # FIXME: mariadb introduces FOR CHANNEL in 10.7
- db_engine == 'mysql' # FIXME: mariadb introduces FOR CHANNEL in 10.7
- mysql8022_and_higher == true # FIXME: mysql 5.7 should work, but our tets fails, why?
# Tests of resetprimary mode:
- import_tasks: mysql_replication_resetprimary_mode.yml

3
tests/integration/targets/test_mysql_replication/tasks/mysql_replication_channel.yml
View file

@ -1,3 +1,4 @@
---
# Copyright: (c) 2019, Andrew Klychkov (@Andersson007) <aaklychkov@mail.ru>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
@ -5,7 +6,7 @@
mysql_params: &mysql_params
login_user: '{{ mysql_user }}'
login_password: '{{ mysql_password }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
block:
# Get primary log file and log pos:

168
tests/integration/targets/test_mysql_replication/tasks/mysql_replication_initial.yml
View file

@ -1,3 +1,4 @@
---
# Copyright: (c) 2019, Andrew Klychkov (@Andersson007) <aaklychkov@mail.ru>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
@ -5,16 +6,9 @@
mysql_params: &mysql_params
login_user: '{{ mysql_user }}'
login_password: '{{ mysql_password }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
block:
- name: find out the database version
mysql_info:
<<: *mysql_params
login_port: '{{ mysql_primary_port }}'
filter: version
register: db
- name: Set mysql8022_and_higher
set_fact:
mysql8022_and_higher: false
@ -23,21 +17,31 @@
set_fact:
mysql8022_and_higher: true
when:
- db.version.major > 8 or (db.version.major == 8 and db.version.minor > 0) or (db.version.major == 8 and db.version.minor == 0 and db.version.release >= 22)
- install_type == 'mysql'
- db_engine == 'mysql'
- db_version is version('8.0.22', '>=')
- name: alias mysql command to include default options
set_fact:
mysql_command: "mysql -u{{ mysql_user }} -p{{ mysql_password }} --protocol=tcp"
# Preparation:
# We use iF NOT EXISTS because the GITHUB Action:
# "ansible-community/ansible-test-gh-action" uses "--retry-on-error".
# If test_mysql_replication fails, test will run again an without the IF
# NOT EXISTS, we see "Error 1396 (HY000): Operation CREATE USER failed..."
# which is misleading.
- name: Create user for mysql replication
shell: "echo \"CREATE USER '{{ replication_user }}'@'localhost' IDENTIFIED WITH mysql_native_password BY '{{ replication_pass }}'; GRANT REPLICATION SLAVE ON *.* TO '{{ replication_user }}'@'localhost';\" | {{ mysql_command }} -P{{ mysql_primary_port }}"
when: install_type == 'mysql'
shell:
"echo \"CREATE USER IF NOT EXISTS \
'{{ replication_user }}'@'{{ mysql_host }}' \
IDENTIFIED WITH mysql_native_password BY '{{ replication_pass }}'; \
GRANT REPLICATION SLAVE ON *.* TO \
'{{ replication_user }}'@'{{ mysql_host }}';\" | {{ mysql_command }}"
when: db_engine == 'mysql'
- name: Create user for mariadb replication
shell: "echo \"CREATE USER '{{ replication_user }}'@'localhost' IDENTIFIED BY '{{ replication_pass }}'; GRANT REPLICATION SLAVE ON *.* TO '{{ replication_user }}'@'localhost';\" | {{ mysql_command }} -P{{ mysql_primary_port }}"
when: install_type == 'mariadb'
shell:
"echo \"CREATE USER IF NOT EXISTS \
'{{ replication_user }}'@'{{ mysql_host }}' \
IDENTIFIED BY '{{ replication_pass }}'; \
GRANT REPLICATION SLAVE ON *.* TO \
'{{ replication_user }}'@'{{ mysql_host }}';\" | {{ mysql_command }}"
when: db_engine == 'mariadb'
- name: Create test database
mysql_db:
@ -47,13 +51,31 @@
name: '{{ test_db }}'
- name: Dump all databases from the primary
shell: 'mysqldump -u{{ mysql_user }} -p{{ mysql_password }} -h{{ mysql_host }} --protocol=tcp -P{{ mysql_primary_port }} --all-databases --ignore-table=mysql.innodb_index_stats --ignore-table=mysql.innodb_table_stats --master-data=2 > {{ dump_path }}'
shell:
cmd: >-
mysqldump
-u{{ mysql_user }}
-p{{ mysql_password }}
-h{{ mysql_host }}
-P{{ mysql_primary_port }}
--protocol=tcp
--all-databases
--ignore-table=mysql.innodb_index_stats
--ignore-table=mysql.innodb_table_stats
--master-data=2
> {{ dump_path }}
- name: Restore the dump to replica1
shell: '{{ mysql_command }} -P{{ mysql_replica1_port }} < {{ dump_path }}'
shell:
cmd: >-
{{ mysql_command_wo_port }}
-P{{ mysql_replica1_port }} < {{ dump_path }}
- name: Restore the dump to replica2
shell: '{{ mysql_command }} -P{{ mysql_replica2_port }} < {{ dump_path }}'
shell:
cmd: >-
{{ mysql_command_wo_port }}
-P{{ mysql_replica2_port }} < {{ dump_path }}
# Test getprimary mode:
- name: Get primary status
@ -63,11 +85,12 @@
mode: getprimary
register: mysql_primary_status
- assert:
- name: Assert that primary is in expected state
assert:
that:
- mysql_primary_status.Is_Primary == true
- mysql_primary_status.Position != 0
- mysql_primary_status is not changed
- mysql_primary_status.Is_Primary == true
- mysql_primary_status.Position != 0
- mysql_primary_status is not changed
# Test startreplica fails without changeprimary first. This needs fail_on_error
- name: Start replica (using deprecated startslave choice) and fail because primary is not specified; failing on error as requested
@ -79,9 +102,10 @@
register: result
ignore_errors: yes
- assert:
- name: Assert that startreplica is failed
assert:
that:
- result is failed
- result is failed
# Test startreplica doesn't fail if fail_on_error: no
- name: Start replica and fail without propagating it to ansible as we were asked not to
@ -92,9 +116,10 @@
fail_on_error: no
register: result
- assert:
- name: Assert that startreplica succeeded
assert:
that:
- result is not failed
- result is not failed
# Test startreplica doesn't fail if there is no fail_on_error.
# This is suboptimal because nothing happens, but it's the old behavior.
@ -105,9 +130,10 @@
mode: startreplica
register: result
- assert:
- name: Assert that start replica succeeded again
assert:
that:
- result is not failed
- result is not failed
# Test changeprimary mode:
# primary_ssl_ca will be set as '' to check the module's behaviour for #23976,
@ -126,10 +152,11 @@
primary_ssl_ca: ''
register: result
- assert:
- name: Assert that changeprimmary is changed and return expected query
assert:
that:
- result is changed
- result.queries == ["CHANGE MASTER TO MASTER_HOST='{{ mysql_host }}',MASTER_USER='{{ replication_user }}',MASTER_PASSWORD='********',MASTER_PORT={{ mysql_primary_port }},MASTER_LOG_FILE='{{ mysql_primary_status.File }}',MASTER_LOG_POS={{ mysql_primary_status.Position }},MASTER_SSL_CA=''"]
- result is changed
- result.queries == ["CHANGE MASTER TO MASTER_HOST='{{ mysql_host }}',MASTER_USER='{{ replication_user }}',MASTER_PASSWORD='********',MASTER_PORT={{ mysql_primary_port }},MASTER_LOG_FILE='{{ mysql_primary_status.File }}',MASTER_LOG_POS={{ mysql_primary_status.Position }},MASTER_SSL_CA=''"]
# Test startreplica mode:
- name: Start replica
@ -139,10 +166,11 @@
mode: startreplica
register: result
- assert:
- name: Assert that startreplica is changed and returns expected query
assert:
that:
- result is changed
- result.queries == ["START SLAVE"] or result.queries == ["START REPLICA"]
- result is changed
- result.queries == ["START SLAVE"] or result.queries == ["START REPLICA"]
# Test getreplica mode:
- name: Get replica status using deprecated getslave choice
@ -152,34 +180,36 @@
mode: getslave
register: replica_status
- assert:
- name: Assert that getreplica returns expected values for MySQL older than 8.0.22 and Mariadb
assert:
that:
- replica_status.Is_Replica == true
- replica_status.Master_Host == '{{ mysql_host }}'
- replica_status.Exec_Master_Log_Pos == mysql_primary_status.Position
- replica_status.Master_Port == {{ mysql_primary_port }}
- replica_status.Last_IO_Errno == 0
- replica_status.Last_IO_Error == ''
- replica_status is not changed
- replica_status.Is_Replica == true
- replica_status.Master_Host == '{{ mysql_host }}'
- replica_status.Exec_Master_Log_Pos == mysql_primary_status.Position
- replica_status.Master_Port == {{ mysql_primary_port }}
- replica_status.Last_IO_Errno == 0
- replica_status.Last_IO_Error == ''
- replica_status is not changed
when: mysql8022_and_higher == false
- assert:
- name: Assert that getreplica returns expected values for MySQL newer than 8.0.22
assert:
that:
- replica_status.Is_Slave == true
- replica_status.Source_Host == '{{ mysql_host }}'
- replica_status.Exec_Source_Log_Pos == mysql_primary_status.Position
- replica_status.Source_Port == {{ mysql_primary_port }}
- replica_status.Last_IO_Errno == 0
- replica_status.Last_IO_Error == ''
- replica_status is not changed
- replica_status.Is_Slave == true
- replica_status.Source_Host == '{{ mysql_host }}'
- replica_status.Exec_Source_Log_Pos == mysql_primary_status.Position
- replica_status.Source_Port == {{ mysql_primary_port }}
- replica_status.Last_IO_Errno == 0
- replica_status.Last_IO_Error == ''
- replica_status is not changed
when: mysql8022_and_higher == true
# Create test table and add data to it:
- name: Create test table
shell: "echo \"CREATE TABLE {{ test_table }} (id int);\" | {{ mysql_command }} -P{{ mysql_primary_port }} {{ test_db }}"
shell: "echo \"CREATE TABLE {{ test_table }} (id int);\" | {{ mysql_command_wo_port }} -P{{ mysql_primary_port }} {{ test_db }}"
- name: Insert data
shell: "echo \"INSERT INTO {{ test_table }} (id) VALUES (1), (2), (3); FLUSH LOGS;\" | {{ mysql_command }} -P{{ mysql_primary_port }} {{ test_db }}"
shell: "echo \"INSERT INTO {{ test_table }} (id) VALUES (1), (2), (3); FLUSH LOGS;\" | {{ mysql_command_wo_port }} -P{{ mysql_primary_port }} {{ test_db }}"
- name: Small pause to be sure the bin log, which was flushed previously, reached the replica
ansible.builtin.wait_for:
@ -195,19 +225,18 @@
# mysql_primary_status.Position is not actual and it has been changed by the prev step,
# so replica_status.Exec_Master_Log_Pos must be different:
- assert:
- name: Assert that getreplica Log_Pos is different for MySQL older than 8.0.22 and MariaDB
assert:
that:
- replica_status.Exec_Master_Log_Pos != mysql_primary_status.Position
- replica_status.Exec_Master_Log_Pos != mysql_primary_status.Position
when: mysql8022_and_higher == false
- assert:
- name: Assert that getreplica Log_Pos is different for MySQL newer than 8.0.22
assert:
that:
- replica_status.Exec_Source_Log_Pos != mysql_primary_status.Position
- replica_status.Exec_Source_Log_Pos != mysql_primary_status.Position
when: mysql8022_and_higher == true
- shell: pip show pymysql | awk '/Version/ {print $2}'
register: pymysql_version
- name: Start replica that is already running
mysql_replication:
<<: *mysql_params
@ -217,7 +246,8 @@
register: result
# mysqlclient 2.0.1 always return "changed"
- assert:
- name: Assert that startreplica is not changed
assert:
that:
- result is not changed
when:
@ -231,10 +261,11 @@
mode: stopslave
register: result
- assert:
- name: Assert that stopreplica is changed and returns expected query
assert:
that:
- result is changed
- result.queries == ["STOP SLAVE"] or result.queries == ["STOP REPLICA"]
- result is changed
- result.queries == ["STOP SLAVE"] or result.queries == ["STOP REPLICA"]
- name: Pause for 2 seconds to let the replication stop
ansible.builtin.wait_for:
@ -250,7 +281,8 @@
fail_on_error: true
register: result
- assert:
- name: Assert that stopreplica is not changed
assert:
that:
- result is not changed
when:

2
tests/integration/targets/test_mysql_replication/tasks/mysql_replication_primary_delay.yml
View file

@ -5,7 +5,7 @@
mysql_params: &mysql_params
login_user: '{{ mysql_user }}'
login_password: '{{ mysql_password }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
block:

2
tests/integration/targets/test_mysql_replication/tasks/mysql_replication_resetprimary_mode.yml
View file

@ -5,7 +5,7 @@
mysql_params: &mysql_params
login_user: '{{ mysql_user }}'
login_password: '{{ mysql_password }}'
login_host: 127.0.0.1
login_host: '{{ mysql_host }}'
block: