filesystem: revamp module (#2472)

* revamp filesystem module to prepare next steps

* pass all commands to module.run_command() as lists
* refactor grow() and grow_cmd() to not need to override them so much
* refactor all existing get_fs_size() overrides to raise a ValueError if
  not able to parse command output and return an integer.
* override MKFS_FORCE_FLAGS the same way for all fstypes that require it
* improve documentation of limitations of the module regarding FreeBSD
* fix indentation in DOCUMENTATION
* add/update function/method docstrings
* fix pylint hints

filesystem: refactor integration tests

* Include *reiserfs* and *swap* in tests.
* Fix reiserfs related code and tests accordingly.
* Replace "other fs" (unhandled by this module), from *swap* to *minix*
  (both mkswap and mkfs.minix being provided by util-linux).
* Replace *dd* commands by *filesize* dedicated module.
* Use FQCNs and name the tasks.
* Update main tests conditionals.

* add a changelog fragment

* Apply suggestions from code review

Co-authored-by: Felix Fontein <felix@fontein.de>

* declare variables as lists when lists are needed

* fix construction without useless conversion

Co-authored-by: Felix Fontein <felix@fontein.de>
This commit is contained in:
quidame 2021-05-18 06:46:45 +02:00 committed by GitHub
parent d24fc92466
commit f6db0745fc
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 434 additions and 288 deletions

View file

@ -7,10 +7,11 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
DOCUMENTATION = '''
---
author:
- Alexander Bulimov (@abulimov)
- Alexander Bulimov (@abulimov)
module: filesystem
short_description: Makes a filesystem
description:
@ -18,13 +19,12 @@ description:
options:
state:
description:
- If C(state=present), the filesystem is created if it doesn't already
exist, that is the default behaviour if I(state) is omitted.
- If C(state=absent), filesystem signatures on I(dev) are wiped if it
contains a filesystem (as known by C(blkid)).
- When C(state=absent), all other options but I(dev) are ignored, and the
module doesn't fail if the device I(dev) doesn't actually exist.
- C(state=absent) is not supported and will fail on FreeBSD systems.
- If C(state=present), the filesystem is created if it doesn't already
exist, that is the default behaviour if I(state) is omitted.
- If C(state=absent), filesystem signatures on I(dev) are wiped if it
contains a filesystem (as known by C(blkid)).
- When C(state=absent), all other options but I(dev) are ignored, and the
module doesn't fail if the device I(dev) doesn't actually exist.
type: str
choices: [ present, absent ]
default: present
@ -32,48 +32,56 @@ options:
fstype:
choices: [ btrfs, ext2, ext3, ext4, ext4dev, f2fs, lvm, ocfs2, reiserfs, xfs, vfat, swap ]
description:
- Filesystem type to be created. This option is required with
C(state=present) (or if I(state) is omitted).
- reiserfs support was added in 2.2.
- lvm support was added in 2.5.
- since 2.5, I(dev) can be an image file.
- vfat support was added in 2.5
- ocfs2 support was added in 2.6
- f2fs support was added in 2.7
- swap support was added in 2.8
- Filesystem type to be created. This option is required with
C(state=present) (or if I(state) is omitted).
- reiserfs support was added in 2.2.
- lvm support was added in 2.5.
- since 2.5, I(dev) can be an image file.
- vfat support was added in 2.5
- ocfs2 support was added in 2.6
- f2fs support was added in 2.7
- swap support was added in 2.8
type: str
aliases: [type]
dev:
description:
- Target path to device or image file.
- Target path to block device or regular file.
- On systems not using block devices but character devices instead (as
FreeBSD), this module only works when applying to regular files, aka
disk images.
type: path
required: yes
aliases: [device]
force:
description:
- If C(yes), allows to create new filesystem on devices that already has filesystem.
- If C(yes), allows to create new filesystem on devices that already has filesystem.
type: bool
default: 'no'
resizefs:
description:
- If C(yes), if the block device and filesystem size differ, grow the filesystem into the space.
- Supported for C(ext2), C(ext3), C(ext4), C(ext4dev), C(f2fs), C(lvm), C(xfs) and C(vfat) filesystems.
Attempts to resize other filesystem types will fail.
- XFS Will only grow if mounted. Currently, the module is based on commands
from C(util-linux) package to perform operations, so resizing of XFS is
not supported on FreeBSD systems.
- vFAT will likely fail if fatresize < 1.04.
- If C(yes), if the block device and filesystem size differ, grow the filesystem into the space.
- Supported for C(ext2), C(ext3), C(ext4), C(ext4dev), C(f2fs), C(lvm), C(xfs) and C(vfat) filesystems.
Attempts to resize other filesystem types will fail.
- XFS Will only grow if mounted. Currently, the module is based on commands
from C(util-linux) package to perform operations, so resizing of XFS is
not supported on FreeBSD systems.
- vFAT will likely fail if fatresize < 1.04.
type: bool
default: 'no'
opts:
description:
- List of options to be passed to mkfs command.
- List of options to be passed to mkfs command.
type: str
requirements:
- Uses tools related to the I(fstype) (C(mkfs)) and C(blkid) command. When I(resizefs) is enabled, C(blockdev) command is required too.
- Uses tools related to the I(fstype) (C(mkfs)) and the C(blkid) command.
- When I(resizefs) is enabled, C(blockdev) command is required too.
notes:
- Potential filesystem on I(dev) are checked using C(blkid), in case C(blkid) isn't able to detect an existing filesystem,
this filesystem is overwritten even if I(force) is C(no).
- Potential filesystem on I(dev) are checked using C(blkid). In case C(blkid)
isn't able to detect an existing filesystem, this filesystem is overwritten
even if I(force) is C(no).
- On FreeBSD systems, either C(e2fsprogs) or C(util-linux) packages provide
a C(blkid) command that is compatible with this module, when applied to
regular files.
- This module supports I(check_mode).
'''
@ -102,6 +110,7 @@ import re
import stat
from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils._text import to_native
class Device(object):
@ -114,13 +123,15 @@ class Device(object):
statinfo = os.stat(self.path)
if stat.S_ISBLK(statinfo.st_mode):
blockdev_cmd = self.module.get_bin_path("blockdev", required=True)
dummy, devsize_in_bytes, dummy = self.module.run_command([blockdev_cmd, "--getsize64", self.path], check_rc=True)
return int(devsize_in_bytes)
dummy, out, dummy = self.module.run_command([blockdev_cmd, "--getsize64", self.path], check_rc=True)
devsize_in_bytes = int(out)
elif os.path.isfile(self.path):
return os.path.getsize(self.path)
devsize_in_bytes = os.path.getsize(self.path)
else:
self.module.fail_json(changed=False, msg="Target device not supported: %s" % self)
return devsize_in_bytes
def get_mountpoint(self):
"""Return (first) mountpoint of device. Returns None when not mounted."""
cmd_findmnt = self.module.get_bin_path("findmnt", required=True)
@ -141,9 +152,12 @@ class Device(object):
class Filesystem(object):
GROW = None
MKFS = None
MKFS_FORCE_FLAGS = ''
MKFS_FORCE_FLAGS = []
INFO = None
GROW = None
GROW_MAX_SPACE_FLAGS = []
GROW_MOUNTPOINT_ONLY = False
LANG_ENV = {'LANG': 'C', 'LC_ALL': 'C', 'LC_MESSAGES': 'C'}
@ -155,7 +169,11 @@ class Filesystem(object):
return type(self).__name__
def get_fs_size(self, dev):
""" Return size in bytes of filesystem on device. Returns int """
"""Return size in bytes of filesystem on device (integer).
Should query the info with a per-fstype command that can access the
device whenever it is mounted or not, and parse the command output.
Parser must ensure to return an integer, or raise a ValueError.
"""
raise NotImplementedError()
def create(self, opts, dev):
@ -163,31 +181,27 @@ class Filesystem(object):
return
mkfs = self.module.get_bin_path(self.MKFS, required=True)
if opts is None:
cmd = "%s %s '%s'" % (mkfs, self.MKFS_FORCE_FLAGS, dev)
else:
cmd = "%s %s %s '%s'" % (mkfs, self.MKFS_FORCE_FLAGS, opts, dev)
cmd = [mkfs] + self.MKFS_FORCE_FLAGS + opts + [str(dev)]
self.module.run_command(cmd, check_rc=True)
def wipefs(self, dev):
if platform.system() == 'FreeBSD':
msg = "module param state=absent is currently not supported on this OS (FreeBSD)."
self.module.fail_json(msg=msg)
if self.module.check_mode:
return
# wipefs comes with util-linux package (as 'blockdev' & 'findmnt' above)
# so it is not supported on FreeBSD. Even the use of dd as a fallback is
# that is ported to FreeBSD. The use of dd as a portable fallback is
# not doable here if it needs get_mountpoint() (to prevent corruption of
# a mounted filesystem), since 'findmnt' is not available on FreeBSD.
# a mounted filesystem), since 'findmnt' is not available on FreeBSD,
# even in util-linux port for this OS.
wipefs = self.module.get_bin_path('wipefs', required=True)
cmd = [wipefs, "--all", dev.__str__()]
cmd = [wipefs, "--all", str(dev)]
self.module.run_command(cmd, check_rc=True)
def grow_cmd(self, dev):
cmd = self.module.get_bin_path(self.GROW, required=True)
return [cmd, str(dev)]
def grow_cmd(self, target):
"""Build and return the resizefs commandline as list."""
cmdline = [self.module.get_bin_path(self.GROW, required=True)]
cmdline += self.GROW_MAX_SPACE_FLAGS + [target]
return cmdline
def grow(self, dev):
"""Get dev and fs size and compare. Returns stdout of used command."""
@ -196,31 +210,50 @@ class Filesystem(object):
try:
fssize_in_bytes = self.get_fs_size(dev)
except NotImplementedError:
self.module.fail_json(changed=False, msg="module does not support resizing %s filesystem yet." % self.fstype)
self.module.fail_json(msg="module does not support resizing %s filesystem yet" % self.fstype)
except ValueError as err:
self.module.warn("unable to process %s output '%s'" % (self.INFO, to_native(err)))
self.module.fail_json(msg="unable to process %s output for %s" % (self.INFO, dev))
if not fssize_in_bytes < devsize_in_bytes:
self.module.exit_json(changed=False, msg="%s filesystem is using the whole device %s" % (self.fstype, dev))
elif self.module.check_mode:
self.module.exit_json(changed=True, msg="Resizing filesystem %s on device %s" % (self.fstype, dev))
self.module.exit_json(changed=True, msg="resizing filesystem %s on device %s" % (self.fstype, dev))
if self.GROW_MOUNTPOINT_ONLY:
mountpoint = dev.get_mountpoint()
if not mountpoint:
self.module.fail_json(msg="%s needs to be mounted for %s operations" % (dev, self.fstype))
grow_target = mountpoint
else:
dummy, out, dummy = self.module.run_command(self.grow_cmd(dev), check_rc=True)
return out
grow_target = str(dev)
dummy, out, dummy = self.module.run_command(self.grow_cmd(grow_target), check_rc=True)
return out
class Ext(Filesystem):
MKFS_FORCE_FLAGS = '-F'
MKFS_FORCE_FLAGS = ['-F']
INFO = 'tune2fs'
GROW = 'resize2fs'
def get_fs_size(self, dev):
cmd = self.module.get_bin_path('tune2fs', required=True)
# Get Block count and Block size
dummy, size, dummy = self.module.run_command([cmd, '-l', str(dev)], check_rc=True, environ_update=self.LANG_ENV)
for line in size.splitlines():
"""Get Block count and Block size and return their product."""
cmd = self.module.get_bin_path(self.INFO, required=True)
dummy, out, dummy = self.module.run_command([cmd, '-l', str(dev)], check_rc=True, environ_update=self.LANG_ENV)
block_count = block_size = None
for line in out.splitlines():
if 'Block count:' in line:
block_count = int(line.split(':')[1].strip())
elif 'Block size:' in line:
block_size = int(line.split(':')[1].strip())
return block_size * block_count
if None not in (block_size, block_count):
break
else:
raise ValueError(out)
return block_size * block_count
class Ext2(Ext):
@ -237,52 +270,46 @@ class Ext4(Ext):
class XFS(Filesystem):
MKFS = 'mkfs.xfs'
MKFS_FORCE_FLAGS = '-f'
MKFS_FORCE_FLAGS = ['-f']
INFO = 'xfs_info'
GROW = 'xfs_growfs'
GROW_MOUNTPOINT_ONLY = True
def get_fs_size(self, dev):
cmd = self.module.get_bin_path('xfs_info', required=True)
"""Get bsize and blocks and return their product."""
cmdline = [self.module.get_bin_path(self.INFO, required=True)]
# Depending on the versions, xfs_info is able to get info from the
# device, whenever it is mounted or not, or only if unmounted, or
# only if mounted, or not at all. For any version until now, it is
# able to query info from the mountpoint. So try it first, and use
# device as the last resort: it may or may not work.
mountpoint = dev.get_mountpoint()
if mountpoint:
rc, out, err = self.module.run_command([cmd, str(mountpoint)], environ_update=self.LANG_ENV)
cmdline += [mountpoint]
else:
# Recent GNU/Linux distros support access to unmounted XFS filesystems
rc, out, err = self.module.run_command([cmd, str(dev)], environ_update=self.LANG_ENV)
if rc != 0:
self.module.fail_json(msg="Error while attempting to query size of XFS filesystem: %s" % err)
cmdline += [str(dev)]
dummy, out, dummy = self.module.run_command(cmdline, check_rc=True, environ_update=self.LANG_ENV)
block_size = block_count = None
for line in out.splitlines():
col = line.split('=')
if col[0].strip() == 'data':
if col[1].strip() != 'bsize':
self.module.fail_json(msg='Unexpected output format from xfs_info (could not locate "bsize")')
if col[2].split()[1] != 'blocks':
self.module.fail_json(msg='Unexpected output format from xfs_info (could not locate "blocks")')
block_size = int(col[2].split()[0])
block_count = int(col[3].split(',')[0])
return block_size * block_count
if col[1].strip() == 'bsize':
block_size = int(col[2].split()[0])
if col[2].split()[1] == 'blocks':
block_count = int(col[3].split(',')[0])
if None not in (block_size, block_count):
break
else:
raise ValueError(out)
def grow_cmd(self, dev):
# Check first if growing is needed, and then if it is doable or not.
devsize_in_bytes = dev.size()
fssize_in_bytes = self.get_fs_size(dev)
if not fssize_in_bytes < devsize_in_bytes:
self.module.exit_json(changed=False, msg="%s filesystem is using the whole device %s" % (self.fstype, dev))
mountpoint = dev.get_mountpoint()
if not mountpoint:
# xfs filesystem needs to be mounted
self.module.fail_json(msg="%s needs to be mounted for xfs operations" % dev)
cmd = self.module.get_bin_path(self.GROW, required=True)
return [cmd, str(mountpoint)]
return block_size * block_count
class Reiserfs(Filesystem):
MKFS = 'mkfs.reiserfs'
MKFS_FORCE_FLAGS = '-f'
MKFS_FORCE_FLAGS = ['-q']
class Btrfs(Filesystem):
@ -290,7 +317,8 @@ class Btrfs(Filesystem):
def __init__(self, module):
super(Btrfs, self).__init__(module)
dummy, stdout, stderr = self.module.run_command('%s --version' % self.MKFS, check_rc=True)
mkfs = self.module.get_bin_path(self.MKFS, required=True)
dummy, stdout, stderr = self.module.run_command([mkfs, '--version'], check_rc=True)
match = re.search(r" v([0-9.]+)", stdout)
if not match:
# v0.20-rc1 use stderr
@ -298,29 +326,27 @@ class Btrfs(Filesystem):
if match:
# v0.20-rc1 doesn't have --force parameter added in following version v3.12
if LooseVersion(match.group(1)) >= LooseVersion('3.12'):
self.MKFS_FORCE_FLAGS = '-f'
else:
self.MKFS_FORCE_FLAGS = ''
self.MKFS_FORCE_FLAGS = ['-f']
else:
# assume version is greater or equal to 3.12
self.MKFS_FORCE_FLAGS = '-f'
self.MKFS_FORCE_FLAGS = ['-f']
self.module.warn('Unable to identify mkfs.btrfs version (%r, %r)' % (stdout, stderr))
class Ocfs2(Filesystem):
MKFS = 'mkfs.ocfs2'
MKFS_FORCE_FLAGS = '-Fx'
MKFS_FORCE_FLAGS = ['-Fx']
class F2fs(Filesystem):
MKFS = 'mkfs.f2fs'
INFO = 'dump.f2fs'
GROW = 'resize.f2fs'
@property
def MKFS_FORCE_FLAGS(self):
def __init__(self, module):
super(F2fs, self).__init__(module)
mkfs = self.module.get_bin_path(self.MKFS, required=True)
cmd = "%s %s" % (mkfs, os.devnull)
dummy, out, dummy = self.module.run_command(cmd, check_rc=False, environ_update=self.LANG_ENV)
dummy, out, dummy = self.module.run_command([mkfs, os.devnull], check_rc=False, environ_update=self.LANG_ENV)
# Looking for " F2FS-tools: mkfs.f2fs Ver: 1.10.0 (2018-01-30)"
# mkfs.f2fs displays version since v1.2.0
match = re.search(r"F2FS-tools: mkfs.f2fs Ver: ([0-9.]+) \(", out)
@ -328,69 +354,73 @@ class F2fs(Filesystem):
# Since 1.9.0, mkfs.f2fs check overwrite before make filesystem
# before that version -f switch wasn't used
if LooseVersion(match.group(1)) >= LooseVersion('1.9.0'):
return '-f'
return ''
self.MKFS_FORCE_FLAGS = ['-f']
def get_fs_size(self, dev):
cmd = self.module.get_bin_path('dump.f2fs', required=True)
# Get sector count and sector size
dummy, dump, dummy = self.module.run_command([cmd, str(dev)], check_rc=True, environ_update=self.LANG_ENV)
sector_size = None
sector_count = None
for line in dump.splitlines():
"""Get sector size and total FS sectors and return their product."""
cmd = self.module.get_bin_path(self.INFO, required=True)
dummy, out, dummy = self.module.run_command([cmd, str(dev)], check_rc=True, environ_update=self.LANG_ENV)
sector_size = sector_count = None
for line in out.splitlines():
if 'Info: sector size = ' in line:
# expected: 'Info: sector size = 512'
sector_size = int(line.split()[4])
elif 'Info: total FS sectors = ' in line:
# expected: 'Info: total FS sectors = 102400 (50 MB)'
sector_count = int(line.split()[5])
if None not in (sector_size, sector_count):
break
else:
self.module.warn("Unable to process dump.f2fs output '%s'", '\n'.join(dump))
self.module.fail_json(msg="Unable to process dump.f2fs output for %s" % dev)
raise ValueError(out)
return sector_size * sector_count
class VFAT(Filesystem):
if platform.system() == 'FreeBSD':
MKFS = "newfs_msdos"
else:
MKFS = 'mkfs.vfat'
INFO = 'fatresize'
GROW = 'fatresize'
GROW_MAX_SPACE_FLAGS = ['-s', 'max']
def __init__(self, module):
super(VFAT, self).__init__(module)
if platform.system() == 'FreeBSD':
self.MKFS = 'newfs_msdos'
else:
self.MKFS = 'mkfs.vfat'
def get_fs_size(self, dev):
cmd = self.module.get_bin_path(self.GROW, required=True)
dummy, output, dummy = self.module.run_command([cmd, '--info', str(dev)], check_rc=True, environ_update=self.LANG_ENV)
for line in output.splitlines()[1:]:
"""Get and return size of filesystem, in bytes."""
cmd = self.module.get_bin_path(self.INFO, required=True)
dummy, out, dummy = self.module.run_command([cmd, '--info', str(dev)], check_rc=True, environ_update=self.LANG_ENV)
fssize = None
for line in out.splitlines()[1:]:
param, value = line.split(':', 1)
if param.strip() == 'Size':
return int(value.strip())
self.module.fail_json(msg="fatresize failed to provide filesystem size for %s" % dev)
fssize = int(value.strip())
break
else:
raise ValueError(out)
def grow_cmd(self, dev):
cmd = self.module.get_bin_path(self.GROW)
return [cmd, "-s", str(dev.size()), str(dev.path)]
return fssize
class LVM(Filesystem):
MKFS = 'pvcreate'
MKFS_FORCE_FLAGS = '-f'
MKFS_FORCE_FLAGS = ['-f']
INFO = 'pvs'
GROW = 'pvresize'
def get_fs_size(self, dev):
cmd = self.module.get_bin_path('pvs', required=True)
"""Get and return PV size, in bytes."""
cmd = self.module.get_bin_path(self.INFO, required=True)
dummy, size, dummy = self.module.run_command([cmd, '--noheadings', '-o', 'pv_size', '--units', 'b', '--nosuffix', str(dev)], check_rc=True)
block_count = int(size)
return block_count
pv_size = int(size)
return pv_size
class Swap(Filesystem):
MKFS = 'mkswap'
MKFS_FORCE_FLAGS = '-f'
MKFS_FORCE_FLAGS = ['-f']
FILESYSTEMS = {
@ -439,6 +469,10 @@ def main():
force = module.params['force']
resizefs = module.params['resizefs']
mkfs_opts = []
if opts is not None:
mkfs_opts = opts.split()
changed = False
if not os.path.exists(dev):
@ -451,7 +485,7 @@ def main():
dev = Device(module, dev)
cmd = module.get_bin_path('blkid', required=True)
rc, raw_fs, err = module.run_command("%s -c /dev/null -o value -s TYPE %s" % (cmd, dev))
rc, raw_fs, err = module.run_command([cmd, '-c', os.devnull, '-o', 'value', '-s', 'TYPE', str(dev)])
# In case blkid isn't able to identify an existing filesystem, device is considered as empty,
# then this existing filesystem would be overwritten even if force isn't enabled.
fs = raw_fs.strip()
@ -481,7 +515,7 @@ def main():
module.fail_json(msg="'%s' is already used as %s, use force=yes to overwrite" % (dev, fs), rc=rc, err=err)
# create fs
filesystem.create(opts, dev)
filesystem.create(mkfs_opts, dev)
changed = True
elif fs: