'with (yield from alock):' is incompatible with Python 3.9+.
Change it to 'async with alock:', and then change the affected functions
to 'async def'.
This makes the test suite pass again in a Fedora 33 VM.
QubesOS/qubes-issues#2738
is_dirty() returned a false positive if the volume was merely the source
of a currently running volume. For example, if fedora-33:root was the
source volume for myappvm:root and myappvm was running - then is_dirty()
returned True for fedora-33:root, because fedora-33/root-cow.img
contains some allocated blocks (one 256 KiB chunk containing only the
header) in this scenario, even though fedora-33 is shut down.
FixesQubesOS/qubes-issues#6371
replace_file(), rename_file(), and remove_file() now have optional
'logger' and 'log_level' (defaulting to DEBUG) arguments.
replace_file() now has a required 'permissions' and an optional
'close_on_success' (defaulting to True) argument. Also, it doesn't
create any directories; and in case of an exception, the tempfile is
removed even when closing it raises another exception.
remove_file() now returns a value: True if the file was removed, or
False if it already didn't exist.
(fsync_path() is unchanged.)
!!! After cherry-picking for release4.0, consider a fixup !!!
!!! adding 'import qubes.utils' to storage/reflink there !!!
Add support for blocking shutdown call. This adds a symmetry to
admin.vm.Start call which is blocking.
Since the admin.vm.Shutdown has established semantic already, add a
'wait' parameter. It can be combined with 'force' as 'force+wait' (or
the other way around).
Fedora compresses the kernel with zstd now, which isn't suppported in
grub (as of 2.04). Since we don't care about PV that much anymore,
simply skip the test there.
* origin/pr/354: (35 commits)
tests/lvm: re-introduce POOL_CONF
tests/lvm & callback: remove explicit class references
storage/callback: remove the "word of caution"
storage/callback: comment fixes
storage/callback: add the config ID as callback argument
storage/callback: some callbacks added & removed
tests/lvm & callback: Refactoring
Revert "storage/callback: do not run sync code async"
tests/callback: ensure missing conf causes errors
storage/callback: do not run sync code async
tests/callback: added callback-specific tests
storage/callback: async Volume.export() & added Volume.export_end()
storage/lvm: make the "hack" work with CallbackPool instances
storage/callback: add the backend_class property
tests/callback: add them to the rpm build
storage/callback: various fixes
tests/callback: add rudimentary tests for the callback driver
tests/lvm: make the tests re-usable for other drivers
storage/callback: fix issues detected by pylint
storage/callback: volume callbacks now also rceive the source volume as argument (if there's any)
...
* origin/pr/375:
Fix line lengths
Return better error messages from file pool
Fix bugs found by Rusty Bird
Fix export locking
Re-add dirty check in case qubesd is restarted
File volumes are started NAND exported
file pool: snapshotting dirty volume not supported
Always snapshot in the FILE pool
* tests20201030:
tests: fix test_550_ipv6_spoof_ip
tests: wait for full VM startup before testing network and few other
tests: fix logging network info on failed test
tests: use dup-ed stdin in wait_on_fail
tests: add PVH grub2 tests
tests: adjust for applications list stored in features dict
tests: remove test_030_clone
tests: make sure dnsmasq is stopped before starting it again
tests: fail the test early if VM fails to start
Use 'ip route replace' instead of 'ip route add' to not fail on already
existing route entry. This worked before only because of a race
condition with a network setup.
Network may be configured (by qubes-misc-post service in some cases)
after qrexec is started. Delay any test after VM is fully started (as
the user would do).
Adjust wait_for_session function, to re-use VM startup timeout, which
make it adjustable for slower machines (like nested virt on openQA).
The Python version (and pylint) are new enough finally.
Adjust QubesVM.run* functions for now.
Python 3.8 brings also AsyncMock() which makes tests slightly less ugly.
It's still needed by the lvm integration tests. If possible, it should
be avoided inside lvm unit tests though as using it will break
compatibility with callback unit tests.
transport.close() (necessary to detach reader from the loop) will close
the FD. In case of stdin, it will prevent another call to this waiting
function. Use dup(2) to register cloned FD, which is safe to close
without side effects.
Since qubesd-side clone_vm function is gone, it doesn't make sense to
test it. For some time already this tested only if manual step-by-step
clone implemented _in the test itself_ was done correctly.
Actual cloning is part of qubes-core-admin-client and is tested there.
- nopat - added when PAT in Linux+Xen was buggy, no longer the case for a
long time.
- iommu=soft swiotlb=8192 - necessary to limit required memory on PV with
PCI devices; since we use HVM now (with proper IOMMU translation), this
is no longer needed. Furthermore, it will free some memory in sys-* vms.
Make an exception in vm.start() actually interrupt the test. The
asyncio.wait() returns list of completed tasks, where exception may be
stored - but is not raised directly. Change to asyncio.gather() that will
propagate the exception by default.
As a side effect, avoid deprecated direct coroutine passing to
asyncio.wait(). This functionality in asyncio.gather() is not
deprecated.
Add an API for VMs to announce support for non-service features. This is
very similar to supported-service.* features, but applies to non-service
features. This may be also used for announcing support for features that
do not use qvm-features framework itself - for example some VM kernel
features, installed drivers, packages etc.
QubesOS/qubes-issues#6030
Instead of unintuitive Value Error now we have dedicated QubesLabelNotFoundError.
Goal: to make qvm-prefs be less strange when one mixes up gray and grey again.
Deduplicate entries when iterating over TestVMsCollection values. Some
tests add given VM multiple times, to have it available under different
kind of keys (name, uuid etc) - similar to the real VMsCollection.
* origin/pr/254:
vm: allow StandaloneVM to be a DVM template
vm: do not allow setting template_for_dispvms=False if there are any DispVMs
vm: move DVM template specific code into separate mixin
* test-fixes20200806:
tests/extra: add vm.run(..., gui=) argument
tests: collect detailed diagnostics on failure
tests: workaround a race in qrexec test
tests: fix audio recording test
tests: make qvm-sync-clock test more reliable
Help debugging test failures by collecting detailed information on
failure. It will be logger to the standard logger, which will end up
either on stderr or in journalctl.
qrexec-client-vm may return earlier than it's child process (it exits
right away, without waiting for its child). Add a small wait before
reading exit code from a file.
* paranoid-restore:
tests: paranoid backup restore
Add policy for paranoid mode backup restore
Add an extension preventing starting a VM while it's being restored
Add support for 'tag-created-vm-with' feature
To calculate frequency it needs to use samples per second (44100), not
samples pre recording lenght. This caused shorter recordings to not fit
into the margin.
Compare the time with the "current" time retrieved from ClockVM just
before comparing, not with the test start time. This should work even if
the test machine is quite slow (test taking more than 30s).
Do not allow starting a VM while the restoring management VM has still
control over it. Specifically, that restoring VM will not be able to
start just restored VM.
QubesOS/qubes-issues#5310
When a VM with 'tag-created-vm-with' feature set creates a VM (using
Admin API), that VM will get all the tags listed in the feature.
Multiple tags can be separated with spaces.
This will be useful to tag VMs created during paranoid mode backup
restore.
QubesOS/qubes-issues#5310
As discussed in the PR, sync code will not be interrupted when run from
async code as long as Qubes OS doesn't run dedicated threads for async
& sync code. So there's simply no issue to be expected and thus no special
caution required.
Added:
post_volume_create & post_volume_import as requested by Marek
Removed:
post_ctor as this wasn't really useful anyway, but required a lot of
sync code. Without it, some refactoring & potential async improvements
became possible.
This reverts commit 287a4a0429.
As Marek correctly pointed out, sync functions cannot be run async against one another even if run inside an async function
(the python interpreter will remain active until the next yield and that's at the end of the sync func / inside the async function).
--> So there's no need for a lock.
I still cannot protect against assumptions made by sync code authors about blocking the Qubes OS main loop. Those will be broken.
Moreover the code of this commit was botched anyway.
- Removed all own class attributes to avoid name clashes with delegated
class attributes.
- Implemented the previously missing Pool.usage_details property.
- Shadowed all class attributes as instance properties. This is required
as the parent classes enforce the class attributes upon the
CallbackPool & CallbackVolume classes, but they need to be delegated to
the class of the _cb_impl object. We also cannot implement them as class
attributes in CallbackVolume & CallbackPool as they need to work for
arbitrary backend drivers and two backend drivers must not interfere with
each other. Possible alternative: One could dynamically create classes.
