The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.
Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
Catch exception there and log it. Otherwise asyncio complains about not
retrieved exception. There is no one else to handle this exception,
because shutdown event is triggered from libvirt, not any Admin API.
Allow to get domain class as a property, not using admin.vm.List call.
This makes it unnecessary to call admin.vm.List on the client side to
construct wrapper object.
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.
FixesQubesOS/qubes-issues#3115
If VM startup failed before starting anything (even in paused state),
there will be no further event, not even domain-shutdown. This makes it
hard for event-listening applications (like domains tray) to account
domain state. Fix this by emiting domain-start-failed event in every
case of failed startup after emiting domain-pre-start.
Related QubesOS/qubes-issues#3100
* qubesos/pr/150:
qubes/tests: moar fixes
test-packages: add missing libvirt classes
qubes/tests: do not deadlock on .drain()
qubes/vm: put name= first in __repr__
tests: fix some memory leaks
tests: complain about memory leaks
tests: use one event loop and one libvirtaio impl
Recently libvirt removed support for changing event implementation.
Therefore we have to use a single, global one and we check if it is
empty between tests.
On certain locales (e.g. danish) `usage_percent` will output a comma-separated number, which will make `attr` point the last two decimal points, s.t. `return vol_info['attr'][4] == 'a'` (in the `verify` func) will fail and `qubesd` wont run.
* cdrom-boot:
devices: fix error reporting
api/admin: implement admin.vm.device....Set.persistent
devices: implement DeviceCollection.update_persistent()
devices: move DeviceInfo definition earlier
api: do not fail events when listener is cancelled in the meantime
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.
FixesQubesOS/qubes-issues#3047
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).
FixesQubesOS/qubes-issues#3057
qvm-sync-clock in dom0 now synchronize only dom0 time. For VM time,
qvm-sync-clock needs to be called in VM. Also, both will communicate
with qubesd, so must be called asynchronously from tests.
Allow attached device to be converted from persistent to non-persistent
and the other way around.
This is to allow starting a VM with some device attached temporarily.
When VM is not running, it is possible to attach device only
persistently, so this change will allow to do that, then, after starting
the VM, change it to non-persistent - so it will not be attached again
at further startups.
QubesOS/qubes-issues#3055
This is because .tearDown() is not executed if the exception occurs in
setUp() [for example self.skipTest() raises an exception]. The lower
levels of .tearDown() being executed are critical to not leaking file
descriptors.
First, cache objects created with init_volume - this is the only place
where we have full volume configuration (including snap_on_start and
save_on_stop properties).
But also implement get_volume method, to get a volume instance for given
volume id. Such volume instance may be incomplete (other attributes are
available only in owning domain configuration), but it will be enough
for basic operations - like cheching and changing its size, cloning
etc.
Listing volumes still use list of physically present volumes.
This makes it possible to start qubesd service, without physical
presence of some storage devices. Starting VMs using such storage would
still fail, of course.
FixesQubesOS/qubes-issues#2960
This is useful to select default DispVM template for VMs started
directly by the user. This makes sense as long as AdminVM == GUIVM.
QubesOS/qubes-issues#2974
Add public Admin API call to create Disposable VM that would be
automatically destroyed after shutdown. Do not keep this functionality
for qrexec-policy tool only.
Also, use admin.vm.Start there, instead of internal.vm.Start and
admin.vm.Kill instead of internal.vm.CleanupDispVM (this is enough,
because DispVM now have auto_cleanup property).
QubesOS/qubes-issues#2974
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.
New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).
Tests included.
QubesOS/qubes-issues#2974
* services:
tests: check clockvm-related handlers
doc: include list of extensions
qubesvm: fix docstring
ext/services: move exporting 'service.*' features to extensions
app: update handling features/service os ClockVM
* tests-storage:
tests: register libvirt events
tests: even more agressive cleanup in tearDown
app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
api: keep track of established connections
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
storage: fix Storage.clone and Storage.clone_volume
tests: more tests fixes
firewall: raise ValueError on invalid hostname in dsthost=
qmemman: don't load qubes.xml
tests: fix AdminVM test
tests: create temporary files in /tmp
tests: remove renaming test - it isn't supported anymore
tests: various fixes for storage tests
tests: fix removing LVM volumes
tests: fix asyncio usage in some tests
tests: minor fixes to api/admin tests
storage/file: create -cow.img only when needed
storage: move volume_config['source'] filling to one place
app: do not create 'default' storage pool
app: add missing setters for default_pool* global properties
* qdb-watch:
tests: add qdb_watch test
ext/block: make use of QubesDB watch
vm: add API for watching changes in QubesDB
vm: optimize imports
api/admin: don't send internal events in admin.Events
Add explanation why admin.vm.volume.Import is a custom script
Follow change of qubesdb path return type
Rename vm.qdb to vm.untrusted_qdb
Threis no more ntpd service used - new approach do not conflict with
ntpd. Because of this, new feature is named 'service.clocksync', and
should be _enabled_ in ClockVM ('ntpd' was disabled there).
QubesOS/qubes-issues#1230
When an API call is interrupted, the relevant coroutine is cancelled -
which means it may throw CancelledError. At the same time, cancelled
call have related socket already closed (and transport set to None). But
QubesDaemonProtocol.respond try to close the transport again, which
fails. Fix handling this case.
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.
QubesOS/qubes-issues#853
Remove some more references to objects holding (possibly indirectly)
reference to libvirt connection:
- local variables in tearDown function
- running Admin API calls (especially admin.Events)
- vmm._libvirt_conn directly, in case some reference to Qubes()
is still there
- any instance attribute that is an object from 'qubes' python package
(instead of just those descending from BaseVM)
- do not create new Qubes() instance for removing VMs - if we already
have one in self.app
Then trigger garbage collector to really cleanup those objects (and
close relevant file descriptors). It's important do do this before
closing event loop, because some of descructors may try to use it (for
example remove registered handlers).
When tearDownClass is executed, event loop is already closed. Since no
test really need it right now, drop support for test class-wide VMs and
convert those methods back to instance methods.
Also put coroutines (vm.remove_from_disk, vm.kill) onto event loop.
Only qubesd should load qubes.xml directly. Put a TODO comments for now
in place of slow VM reporting, invent some better mechanism later.
This loading of qubes.xml caused deadlocks, because qmemnan kept open
file descriptor (in locked state).
Since it is no longer child of QubesVM, constructor do not take 'qid'
and 'name' arguments.
Also:
- remove other dropped properties tests (netvm, storage related)
- make the test working in non-dom0
- improve TestPool mock - init_volume now return appropriate mock type,
instead of TestPool
- improve patching base directory (/var/lib/qubes) - it is stored in
more than one place...
- fix inheritance in TC_01_ThinPool class
- fix expected LVM volume names ('vm-' prefix)
- fix cleanup after FilePool tests - remove temporary qubes.xml
- asyncio usage
- better reporting in integ.storage - include error message in the
report, not only as a comment in code
