* origin/pr/389:
app: save qubes.xml with utils.replace_file()
app: use suppress() in simple cases
firewall: save firewall.xml with utils.replace_file()
utils: take tweaked helper functions from storage/reflink
storage/reflink: quote logged filenames
'with (yield from alock):' is incompatible with Python 3.9+.
Change it to 'async with alock:', and then change the affected functions
to 'async def'.
This makes the test suite pass again in a Fedora 33 VM.
QubesOS/qubes-issues#2738
is_dirty() returned a false positive if the volume was merely the source
of a currently running volume. For example, if fedora-33:root was the
source volume for myappvm:root and myappvm was running - then is_dirty()
returned True for fedora-33:root, because fedora-33/root-cow.img
contains some allocated blocks (one 256 KiB chunk containing only the
header) in this scenario, even though fedora-33 is shut down.
FixesQubesOS/qubes-issues#6371
replace_file(), rename_file(), and remove_file() now have optional
'logger' and 'log_level' (defaulting to DEBUG) arguments.
replace_file() now has a required 'permissions' and an optional
'close_on_success' (defaulting to True) argument. Also, it doesn't
create any directories; and in case of an exception, the tempfile is
removed even when closing it raises another exception.
remove_file() now returns a value: True if the file was removed, or
False if it already didn't exist.
(fsync_path() is unchanged.)
!!! After cherry-picking for release4.0, consider a fixup !!!
!!! adding 'import qubes.utils' to storage/reflink there !!!
* origin/pr/354: (35 commits)
tests/lvm: re-introduce POOL_CONF
tests/lvm & callback: remove explicit class references
storage/callback: remove the "word of caution"
storage/callback: comment fixes
storage/callback: add the config ID as callback argument
storage/callback: some callbacks added & removed
tests/lvm & callback: Refactoring
Revert "storage/callback: do not run sync code async"
tests/callback: ensure missing conf causes errors
storage/callback: do not run sync code async
tests/callback: added callback-specific tests
storage/callback: async Volume.export() & added Volume.export_end()
storage/lvm: make the "hack" work with CallbackPool instances
storage/callback: add the backend_class property
tests/callback: add them to the rpm build
storage/callback: various fixes
tests/callback: add rudimentary tests for the callback driver
tests/lvm: make the tests re-usable for other drivers
storage/callback: fix issues detected by pylint
storage/callback: volume callbacks now also rceive the source volume as argument (if there's any)
...
As discussed in the PR, sync code will not be interrupted when run from
async code as long as Qubes OS doesn't run dedicated threads for async
& sync code. So there's simply no issue to be expected and thus no special
caution required.
Added:
post_volume_create & post_volume_import as requested by Marek
Removed:
post_ctor as this wasn't really useful anyway, but required a lot of
sync code. Without it, some refactoring & potential async improvements
became possible.
This reverts commit 287a4a0429.
As Marek correctly pointed out, sync functions cannot be run async against one another even if run inside an async function
(the python interpreter will remain active until the next yield and that's at the end of the sync func / inside the async function).
--> So there's no need for a lock.
I still cannot protect against assumptions made by sync code authors about blocking the Qubes OS main loop. Those will be broken.
Moreover the code of this commit was botched anyway.
- Removed all own class attributes to avoid name clashes with delegated
class attributes.
- Implemented the previously missing Pool.usage_details property.
- Shadowed all class attributes as instance properties. This is required
as the parent classes enforce the class attributes upon the
CallbackPool & CallbackVolume classes, but they need to be delegated to
the class of the _cb_impl object. We also cannot implement them as class
attributes in CallbackVolume & CallbackPool as they need to work for
arbitrary backend drivers and two backend drivers must not interfere with
each other. Possible alternative: One could dynamically create classes.
Unfortunately this appears to be necessary due to
various Qubes OS `assert` checks and to get `__str__()` et al
from the super class. It also means that we have to implement
all methods of the super class (in the future as well).
Now Volume.export() may be a coroutine and also may be accompanied by
Volume.export_end() cleaning up after it.
See previous commits for building blocks for this.
This commit adjusts usage of Volume.export() and adds matching
Volume.export_end() throughout the code base.
FixesQubesOS/qubes-issues#5935
This is a counterpart to Volume.export(). Up until now, no driver needed
any cleanup after exporting data, but it doesn't mean there won't be
any. This is especially relevant because Volume.export() is supposed to
return a path of a snapshot from before VM start - which may be a
different one than currently active one.
QubesOS/qubes-issues#5935
Avoid local modification in a pool's init_volume influence
vm.volume_config. Currently every pool driver replaces
volume_config['pool'] with a pool object (instead of name) and it leads
to confusing cases where depending on start stage, it is sometimes an
object and sometimes a string.
Additionally, some pool drivers may modify volume_config in unexpected
way - for example test pool driver removes 'pool' entry entirely. Avoid
this fragile interface by giving pool driver a copy of volume_config,
instead of vm.volume_config directly.
Note one side effect is that 'vid' (and other pool-specific parameters)
is not set into vm.volume_config directly after creating a VM, but
possibly only after loading from XML. This should not be an issue in
theory (no core code should expect it), but if some place use
volume_config instead of Volume instance for getting pool-specific
options, it should be fixed.
Don't update _size in the getter, so it can be unlocked (which is
helpful for QubesOS/qubes-issues#5935).
!!! If cherry-picking for release4.0, also adjust import_data() to !!!
!!! use self.size (no underscore) instead of self._get_size() !!!
