Break the ioctl calls from blkdiscard into max 1GB worth of blocks per call. This ensures that the amount of pending discards (both for the virtual lvm thin volume as well as any lower level device recipients) do not dominate the IO streams to the devices. Before this change, IO blocking was leading to both UI and processing freezes in Qubes while very large LVM volumes were being removed. The change will allow other device IOs to be interleaved between the internal ioctl calls from blkdiscard.
umask() modifies the whole process, which could lead to odd permission
problems in concurrent code. Use explicit fchmod() calls instead.
Preserve the legacy storage/file behavior of creating files 0o664 in
FileVolume.create() and 0o644 (from the normal qubesd umask) otherwise.
Get rid of an otherwise unused module-level variable (HOST_MACHINE), and
show the relevant file (reflink.py) in the error message:
Traceback (most recent call last):
File "/usr/bin/qubesd", line 2, in <module>
from qubes.tools.qubesd import main
File "/usr/lib/python3.8/site-packages/qubes/__init__.py", line 801, in <module>
from qubes.app import Qubes
File "/usr/lib/python3.8/site-packages/qubes/app.py", line 64, in <module>
import qubes.storage.reflink
File "/usr/lib/python3.8/site-packages/qubes/storage/reflink.py", line 46, in <module>
FICLONE = {
KeyError: 'armv7l'
qubesd.service: Main process exited, code=exited, status=1/FAILURE
* origin/pr/389:
app: save qubes.xml with utils.replace_file()
app: use suppress() in simple cases
firewall: save firewall.xml with utils.replace_file()
utils: take tweaked helper functions from storage/reflink
storage/reflink: quote logged filenames
'with (yield from alock):' is incompatible with Python 3.9+.
Change it to 'async with alock:', and then change the affected functions
to 'async def'.
This makes the test suite pass again in a Fedora 33 VM.
QubesOS/qubes-issues#2738
is_dirty() returned a false positive if the volume was merely the source
of a currently running volume. For example, if fedora-33:root was the
source volume for myappvm:root and myappvm was running - then is_dirty()
returned True for fedora-33:root, because fedora-33/root-cow.img
contains some allocated blocks (one 256 KiB chunk containing only the
header) in this scenario, even though fedora-33 is shut down.
FixesQubesOS/qubes-issues#6371
replace_file(), rename_file(), and remove_file() now have optional
'logger' and 'log_level' (defaulting to DEBUG) arguments.
replace_file() now has a required 'permissions' and an optional
'close_on_success' (defaulting to True) argument. Also, it doesn't
create any directories; and in case of an exception, the tempfile is
removed even when closing it raises another exception.
remove_file() now returns a value: True if the file was removed, or
False if it already didn't exist.
(fsync_path() is unchanged.)
!!! After cherry-picking for release4.0, consider a fixup !!!
!!! adding 'import qubes.utils' to storage/reflink there !!!
* origin/pr/354: (35 commits)
tests/lvm: re-introduce POOL_CONF
tests/lvm & callback: remove explicit class references
storage/callback: remove the "word of caution"
storage/callback: comment fixes
storage/callback: add the config ID as callback argument
storage/callback: some callbacks added & removed
tests/lvm & callback: Refactoring
Revert "storage/callback: do not run sync code async"
tests/callback: ensure missing conf causes errors
storage/callback: do not run sync code async
tests/callback: added callback-specific tests
storage/callback: async Volume.export() & added Volume.export_end()
storage/lvm: make the "hack" work with CallbackPool instances
storage/callback: add the backend_class property
tests/callback: add them to the rpm build
storage/callback: various fixes
tests/callback: add rudimentary tests for the callback driver
tests/lvm: make the tests re-usable for other drivers
storage/callback: fix issues detected by pylint
storage/callback: volume callbacks now also rceive the source volume as argument (if there's any)
...
As discussed in the PR, sync code will not be interrupted when run from
async code as long as Qubes OS doesn't run dedicated threads for async
& sync code. So there's simply no issue to be expected and thus no special
caution required.
Added:
post_volume_create & post_volume_import as requested by Marek
Removed:
post_ctor as this wasn't really useful anyway, but required a lot of
sync code. Without it, some refactoring & potential async improvements
became possible.
This reverts commit 287a4a0429.
As Marek correctly pointed out, sync functions cannot be run async against one another even if run inside an async function
(the python interpreter will remain active until the next yield and that's at the end of the sync func / inside the async function).
--> So there's no need for a lock.
I still cannot protect against assumptions made by sync code authors about blocking the Qubes OS main loop. Those will be broken.
Moreover the code of this commit was botched anyway.
- Removed all own class attributes to avoid name clashes with delegated
class attributes.
- Implemented the previously missing Pool.usage_details property.
- Shadowed all class attributes as instance properties. This is required
as the parent classes enforce the class attributes upon the
CallbackPool & CallbackVolume classes, but they need to be delegated to
the class of the _cb_impl object. We also cannot implement them as class
attributes in CallbackVolume & CallbackPool as they need to work for
arbitrary backend drivers and two backend drivers must not interfere with
each other. Possible alternative: One could dynamically create classes.
Unfortunately this appears to be necessary due to
various Qubes OS `assert` checks and to get `__str__()` et al
from the super class. It also means that we have to implement
all methods of the super class (in the future as well).
