* Fix explanation about containers images
* Add definitive URI to the containers images
* Document that new images must be set as public
* Add makefile options possible values
* Document that any mysql and mariadb tag can be use
* Add computation of docker_image path
* Refactor pre-command to separate commands for cleaner GHA output
* Refactor to use GHA test matrix
* Cut docker_image from documentation since it's now automatic
* Document how to use run_all_test.py to display the test matrix
* Temp: Add path to images in my fork to validate integrations tests
* Draft: Add a mariadb container
* Add playbook to test connection to the server
* Add healthcheck to MariaDB before starting the tests
This prevent the first test to fail because the db isn't ready yet.
* Add default file for root necessary since using venv instead of docker
* Add % instead of the default 'localhost' since we use remote connection
Previously, everything was on localhost. Now ansible-test is in a
venv and the db is in a container. The db see the IP address from the
podman host (10.88.0.2)
* Add ansible-test integration inventory to .gitignore
* Revert to old workflow to use ansible-test --venv
It seams that that ansible-test-gh-action doesn't handle this option:
https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497
* Cut target filtering
* Fix comparison
We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't
test this easily, I decided to simplify the test.
* Add path to default-file
/root doesn't exist with --venv
* Fix workflow unknown option container_name
* Attempt GHA communication between container using "docker host network"
https://docs.github.com/en/actions/using-containerized-services/about-service-containers
I re-revert the workflow to use the new custom action. But I'm not sure
it will works because I don't know how the container for ansible-test
is started and if it will have access to the services containers.
* Cut anchors currently unsupported by GHA
* Disable healthcheck
I want to first prove that this setup is possible before adding safety
* Disable sanity, units and matrix to speed up tests in GHA
* Further disable tests to speed up
* Add mysql_client to the controller
* Install mysql_client the correct way
* Fix package name and missing apt cache
* Prepare controller with Podman/Docker Network
We use the Podman/Docker network gateway address to communicate between
container. I haven't tested Docker. I would have preferred to use a pod
but only Podman support it and ansible-test only support the
--docker-network option.
* Swap MariaDB with MySQL
* De-duplicate the mysql_command alias
* Generalize mysql and mariadb version based on container name
This way we can split db_engine and db_version and simplify tests.
Also this is mandatory to use the matrix.db_engine_version as the
image name for our services containers.
* Cut docker healthcheck unsupported by GHA
* Fix replication server_id already in use
* Add static test with replication containers
* Fix database not selected
* Fix replication due to usage of gateway_addr instead of localhost
* Simplify version computation
* Linting
* Refactor setup_mysql into setup_controller
* Fix test_mysql_role
* Fix server_id in GHA
GHA lack a way to pass option to docker's command. Also server_id is
not read as a environment variable. So I'm forced to use a config file.
* Add back a package to connect to MySQL 8+
* Linting
* Refactor test_mysql_user to work with other host than localhost
* Refactor way tests info are passed from sed to file with lookup
The idea is to avoid modifying test targets from the workflow to prevent
ansible-test to think every tests needs to be run.
* Fix missing var
* Refactor test to use the db_version from setup_controller
* Add temporary files to .gitignore
* Fix volume path
* Fix volume path by adding a final /
* Fix volume path using $(pwd)
* Fix volume path using github.workspace var
* Cut files from gitignore because it prevents ansible-test to copy them
* Fix pre-test-cmd missing separators
* Cut the newline added by lookup 'file'
* Fix tailing newline by not created it in the first place
* Disable tests to concentrate on the \n and quote issue with my files
* Fix trailing newline and quote in db_engine_version
* Re-enable integration tests to validate db_engine_version is fixed
* lint
* Cut unused file
* Fix pre-test-cmd paste in wrong context
* Re-enable service containers
* Add back docker healthcheck on services
I saw in the GHA logs that it perform an healtcheck ! So I hope this
will work.
* Add tmate to debug the server_id in replicas
* Attempt to fix "invalid syntax"
* Enclose command in quotes
* Refactor the way server_id is set for replicas
The simple way is to add '--server-id 2' after the name of the image of
the container. But GHA doesn't let us do that. The idea of mount a file
from our repo doesn't work because the repo is check out later in the
workflow and I failed to find a pre-job hook. Then I realized that this
MySQL option is dynamic! So we will set that in the test target!
* Re-activate all tests
* Cut useless task
* Use same variable as other target for consistency
* Linting
* Update version tested
* Add options to the makefile
* Add same variables as other target for consistency
* Add IF NOT EXISTS to prevent misleading error on retry
* Cut python 3.11 not supported by ansible-test yet
* Attempt to set log-bin into docker
* Reformat for readability
* Document that full version is mandatory
* Fix newline
* Github complain it doesn't find python 3.1 !!!
* Add option to run only a single target
* Fix mysqlclient not supporting Python 3.9
* Enhance installation of mysql_client
Initially I wanted to install mysql-client-5.7 to test mysql server 5.7
but this package is not available for Ubuntu 18+.
I keep those changes because it allow us to specify the name of the
package based on the Ubuntu version.
* Linting
* Add unique name to simplify debugging
* Fix mysql_dump for MySQL 5.7 and MariaDB when using mysqldump 8
* Add unique name to simplify debugging
* Deduplicate tasks
* Lining
* Add python script to recreate the test matrix from github workflow file
* Fix dump with mysqldump 8 against mysql 5.7
* Disable test for replication with chanel for mysql 5.7
* Add better task name
* Fix exclusion function
* Disable replication with channel tests entirely for MySQL 5.7
* Activate Mysql 8 and Mariadb into GitHub Action Workflow
* Cut Ansible since we can't change what the user have on his computer
* Add running make command for all tests of the matrix
* Add unique test names
* Document run_all_tests.py
* Add unique test names
* Add tmate to experiment with docker healthcheck
* Fix replication settings
sh don't know 'echo -e', so we use bash instead.
Also, we need to wait for the container to be healthy before trying to
restart it. Otherwise that could corrupt it.
* Add TODO verify that the version of mysql/mariadb is correct
* Add more descriptive tests names
* Use mysql_host var name instead of gateway_addr in tests
* Refactor user@<gateway ip addr> into user@%
* Fix healthcheck in GHA
* Disable tests that fails only on MariaDB
* Refactor to remove useless variables
* Workaround for plugin role that fails with any MariaDB versions
* Fix Python 3.10 beein run as 3.1
* Ensure replicas are healthy before rebooting them
* Enable all tests
* Add a virtualenv for ansible-test used locally
* Simplify connector_name variables
* Add PoC using custom ansible-test containers
* Fix docker_container variable name
* Cut forgotten comment
* Fix error when using local registry by using quay.io
* Change tag of test-containers to latest
* Fix ansible-test unknown option
I copied blindly https://github.com/ansible-collections/community.sops/blob/main/.github/workflows/ansible-test.yml#L195
and forgot what ansible-test was expecting
* Cut column-statistics disabling
Thanks to our test-container, we now use the correspond mysql-client.
So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we
use mysql-client-8.
* Add manual test matrix (MariaDB 10.6, 10.7 and 10.8 missing)
* Fix test matrix
Python version should be quoted, otherwise 3.10 become 3.1
We can skip 2.14 and devel with Python3.8
We can skip devel with Python 3.9
We can skip MariaDB 10.4 with mysql-client-10.6
Add tests for MariaDB 10.6, 10.7 and 10.8
* Reduce number of tests and adapt containers images
* Fix queries for roles
* Add filter for issues resolved in newer version of mysqlclient
* Add names to tests
* Fix assertion for mariadb
* Linting
* Cut tests for incompatible MySQL 8 and pymysql 0.7.11
* Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok)
* Cut playbook that are now handled by the test-containers
* Change timeout from 10 to 30 seconds to let mysql/mariadb restart
* Add connector information to the returned values
I need to know what python library was used. I had a container with
both mysqlclient and pymysql installed and tests used a different
connector that what is advertised by the title of integration tests.
We need to prevent that otherwise our tests are worth nothing.
* Add a verify stage at setup of test to assert all version are correct
* Attempt to build and publish an image on ghcr.io
* Add latest release of actions and with a context
* Add trigger on workflow file edit
* Fix env not recognized in the 'on' clause
* Add latest tag
* Fix insufficient context
* Add missing slash
* Cut addition of tag 'latest' as GHA does it automatically
* Add ghcr.io image for mariadb10.3 python3.8 mysqlclient2.0.1
* Change docker-image workflow to work on all images using matrix
* Fix workflow title
* Add support for version of mysqlclient
* Fix context path
* Workaround failed to push ghcr.io
Error was: failed to copy: io: read/write on closed pipe
* Add back all tests using ghcr.io images
* Cut unused images
* Fix verify database version
Sometimes, version_full contains trailing information (-log). To prevent
issues it's best to concatenate major and minor version.
* Fix verify for mysqlclient second name MySQLdb
* Rename variable for consistency
* Fix container name
* Add tag 'latest' to images
* Cut filter for tests now that the right connector is used
* Fix test of mysql/mariadb version in use
* Fix python version lookup
* Add clean up in "always" phase of the block
Because our tests use --retry-on-error, and the first thing the test
does is to try to create the database. We must cleanup otherwise if
there is a retry, it will throw a misleading "database already exists"
error.
* Document TODO
* Disable tests using pymysql 1.0.2
Many tests are failing but this must be fixed in the plugins in a future
PR.
* Cut test MySQL 8 with incompatible pymysql 0.7.11
It fails to connect with error about cryptography unsupported
* Fix dict key lookup
* Fix indentation
* Cut tests that was excluded in previous matrix
* Enable back sanity and unit tests
* Refactor get_driver_version to display name while passing sanity tests
* Fix variable name
* Fix missing cffi package to connect to MySQL 8 using Python 3.9
* Fix image not found
* Split Docker image workflow to rebuild only changed Dockerfile
My goal is not to save the planet but to make it work. Currently
docker/setup-buildx-action@v2 often fails. You have to rerun the
workflow multiple times until it succeed. When you do that with the
matrix with 15 containers, you never get to the point where they all
built successfully. Having separate workflows makes rerun the failing
build easier.
* Fix verify ansible 'devel' for which the version is unknown
Today 'devel' means 2.15, but in the future it will be something else.
* Fix ansible version extraction for "devel"
* Cut matrix from when build was done in a single workflow
* Document fix container name
* Add bold
* Add option to let containers alive at end of testing
* Enhance error handling and doc of get_driver_name and get_driver_version
* Migrate tests documentations in their own file
* Skip retry-on-error by default and add option to activate it on demand
* Rename folder to better purpose
* Enable back push and schedule workflow
* Rename registry from fork to upstream
* Cut Docker Image workflow's filter for branch from my fork
* Add changelog fragment
* Update supported versions
* Rename file for clarity
* Cut mariadb non long term releases
* Add '-client' to the block title to better explain what it is
* Update readme for tested versions of long term release of MariaDB
* Attempt to add the workflow to the Action tab
* Second attempt to add the workflow to the Action tab
* Cut folder re-created by merge from main
* Cut filter by branch
GHA will build the image using the branch name as tag. So we can safely
remove this filter.
* Cut changelog item done in #497
* Attempt to fix upload of image under c.mysql instead of my fork
* Add debug to buildkitd
* Bump setup-buildy-action to latest
* Cut dot in image name in attempt to fix buildx bad request 400 error
* Sanitize the repository name using metadata-action
https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#repository-name-must-be-lowercase
* Document why we use optional checkout action
* Cut debugging from setup-buildx-action
* Fix workflow to work both on fork and c.mysql repository
* Use apt-get instead of apt that not have a stable CLI interface
* Use apt-get instead of apt
* update docker image path to my personal repo
I'm unable to publish under community.mysql. Either it's the dot in
the name or I do something wrong with the GITHUB_TOKEN, but we need to
test my PR, so I'll use docker images from my fork for now.
* Fix test after merge of PR497
* Enhance testing documentation header
* Fix installation of ansible venv
ansible-test is included in ansible package. Also, on Fedora 37 with
python 3.11, pip is missing. By using ensurepip we solve that issue.
* Document usage of continue_on_errors
* Fix versions used in examples
* Add support for systems with unsupported python set as default
* Fix cleanup task
* Fix variable assignation to the include task
* Add forgotten variable to handle unsupported python version
* Fix user site-packages not visible in virtualenv
* Fix test connection to the database and tasks names
* Add create podman network for system missing it.
We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't
find in which release the default network changed and maybe it's
defined in the Linux distribution. So in doubt I always attempt to
create the network.
* Add full path to image to prevent podman asking which registry to use
* Add options to enforce recreate containers even if already exists
* Reformat command multiline to oneline
* Add deletion of anonymous volumes associated with the container
* Comment unused variable
* Change shebang from python to python3 to avoid confusion with python2
This script is a python3 script.
* Add disk and RAM requirements
* Cut the 3 from python command to follow shebang recommendations
https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html
* Reformat spelling
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Reformat file path
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Fix link URI
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Fix link URI
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Lint
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Lint
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Add better task name
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* Move utility task files in their own folder
* Refactor using reusable GHA workflows
* Fix path to called workflow file
* Fix path to use local workflow
* Fix cannot specify version when calling local workflows
* Attempt to use a fixed repo name in the image name
My last attempts produced duplicates images under my name + repo name:
laurent-indermuehle/community.mysql. Previously I had only my name. And
none of the above are what we want. We want only community.mysql in the
image name...
* Add called workflow file in the GHA hooks
Without this, the containers are not rebuilt when you modify the file
built-docker-image.yml.
* Rollback to github.repository in container image name
This time I think I understood. We publish in the
github.repository_owner's namespace. In my case it's laurent-indermuehle
and in case of upstream it's ansible-collection. A proof of that:
https://github.com/orgs/ansible-collections/packages <- here there is
one attempt I did in february to push my branch to the upstream.
So, our tests containers will be visible to the whole community, not
just community.mysql.
---------
Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com>
* change deprecated parameter pw to password
* change deprecated parameter db to database
* add changelog fragment
* Old plugin versions are no longer supported
* Use packaging version checking.
* Use stdlib version comparison
* Use parse_version from setuptools
* Revert to tuple/reduce version check
---------
Co-authored-by: Jorge-Rodriguez <jorge.rodriguez@futurice.com>
* Allow uppercase in variable names for Galera wsrep variables
* Changelog fragment for regex change
* Corrected for excessive line lengths
* Update changelogs/fragments/mysql_variables_allow_uppercase_identifiers.yml
---------
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Add methods to retrieve connector name and version
* Document that mysqlclient is also named MySQLdb
* Document version_added
* Add connector name and version in the returned block
* Cut condition to display any name that is return
In case of MySQLdb is renamed in mysqlclient. In that case, the
integration tests will catch this the day we update the connector
version.
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Fix mariadb test setup
* Update mysql src URL
* Add changelog fragment
* Update 491_fix_download_url.yaml
Sanity test failed because minor_changes in not an element of a list.
* Fix casing
Co-authored-by: Laurent Indermühle <laurent.indermuehle@pm.me>
* mysql_user: enabled autocommit to support MySQL 8
* Add changelog fragment
* Link to issue instead of pull request in changelog fragment
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* add service name to plugin pam/auth_pam usage
* typo fixed
* MySLQ is using identified with auth_pam by ... instead of identified with pam using ... like mariadb does
* a : in description lines breaks yaml syntax
* clearify documentation and add changelog fragment
* Update changelogs/fragments/445_add_service_name_to_plugin_pam_auth_pam_usage.yml
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Update plugins/module_utils/user.py
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Sync GHA workflow w/ the collection template
* Drop the trailing pre-cmd semicolon
* Recover missing `-e` flag of `sed`
* Use relative paths for version configs
* Unquote `env.connector_version_file`
* Use string formatting to fix the substitution problem
* Fix: devel requires python 3.9
Package 'ansible-core' requires a different Python: 3.8.13 not in '>=3.9'
* Exclude older version of Ansible when testing Python 3.9
* Compare privileges from before and after manipulation
* Add unit tests
* Fix FIXME integration tests related to this issue
* Fix sanity check
* Fix assertion when appending privs in mysql_role_initial integration tests
* Fix pylint
* [ci-skip] Add changelog fragment
* Fix: missing fragment file extension
* Replace privileges_equal() by a comparison
* Fix: sanity pylint
* Fix: forgot to remove privileges_equal import from unit tests
* Fix: exclude mysql 8 from test_mysql_user's 'Assert that priv did not change' test
* Add tests to verify that GRANT permission is present after user modification
* Fix: do not revoke GRANT permission when it's already allowed and present in priv parameter
* Deduplicate tests name
Easier to debug this way
* Fix assertions named 'GRANT permission is present'
* Only revoke grant option if it exists and absence is requested
* Fix assertion comments
* Fix: Only revoke grant option if it exists and absence is requested
* Avoid pointless revocations when ALL are granted
* Assert that priv did not change on mariadb also
* Fix: sanity and unity tests
* Format long lines
* Add changelog fragment
Co-authored-by: Laurent Indermühle <laurent.indermuehle@pm.me>
* Add SOURCE_SSL_VERIFY_SERVER_CERT parameter
* Rewiev fixs and add changelog fragment
* fix version
* Update changelogs/fragments/435-mysql_replication_verify_server_cert.yml
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Refactor tests to use "is" and "is not" changed
* Refactor tests to use is succeeded or is failed
* Reformat indentation
* Add filter "bool" to prevent issues
* Add changes from stable-1 integrations tests (PR 418)
* Refactor to use connectors' info declared in setup_mysql
* Fix 2nd replication stop marked changed by mysqlclient
* Add matrix for python and ansible-core versions for sanity tests
* Add python 3.9 to integrations tests
* Add python 3.9 to unit tests
* Reformat sort by python version first
* Add schema and tables for the tests
* Add tests for full dump with and without compression
* Add test for distinct dump with and without compression
* Fix sh not seeing errors for command before the pipe
sh is missing the pipefail flag. We must use bash for this.
* Add cleanup to prevent the following tests from failing
* Fix fqcn in module_defaults
* Add changelog fragment
* Add check to the error message to ensure we captured the right one
* Add option to activate the fix on systems with bash
* Fix errors when data schema is already absent
* Update changelogs/fragments/fix-256-mysql_dump-errors.yml
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Add markup for commands in the documentation string
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Add markup and next release version in the documentation string
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Fix missing dependency for MySQL 8
* Add pipefail to tests of uncompressed dumps to enure it still works
* Fix "bash command not found" if pipefail is used for uncompressed dump
* Fix sanity pep8
* Document example of dump with pipefail
* Add dedpulication to command construct
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
Using += on a list cause some problems druing creation of mysql command:
/usr/bin/mysql - - u s e r = r o o t - - p a s s w o r d = ' ' --socket=/run/mysqld/mysqld.sock
