Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)
As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.
FixesQubesOS/qubes-issues#3247
* qubesos/pr/166:
create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
change default pool code to be fast
cache PropertyHolder.property_list and use O(1) property name lookups
remove unused netid code
cache isinstance(default, collections.Callable)
don't access netvm if it's None in visible_gateway/netmask
There were many cases were the check was missing:
- changing default_netvm
- resetting netvm to default value
- loading already broken qubes.xml
Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...
Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
* 20171107-storage:
api/admin: add API for changing revisions_to_keep dynamically
storage/file: move revisions_to_keep restrictions to property setter
api/admin: hide dd statistics in admin.vm.volume.Import call
storage/lvm: fix importing different-sized volume from another pool
storage/file: fix preserving spareness on volume clone
api/admin: add pool size and usage to admin.pool.Info response
storage: add size and usage properties to pool object
* 20171107-tests-backup-api-misc:
test: make race condition on xterm close less likely
tests/backupcompatibility: fix handling 'internal' property
backup: fix handling target write error (like no disk space)
tests/backupcompatibility: drop R1 format tests
backup: use offline_mode for backup collection
qubespolicy: fix handling '$adminvm' target with ask action
app: drop reference to libvirt object after undefining it
vm: always log startup fail
api: do not log handled errors sent to a client
tests/backups: convert to new restore handling - using qubesadmin module
app: clarify error message on failed domain remove (used somewhere)
Fix qubes-core.service ordering
xterm is very fast on closing when application inside terminates. It is
so fast with closing on keydown event that xdotool do not manage to send
keyup event, resulting in xdotool crash. Add a little more time for
that.
Besides converting itself, change how the test verify restore
correctness: first collect VM metadata (and hashes of data) into plain
dict, then compare against it. This allow to destroy old VMs objects
before restoring the backup, so avoid having duplicate objects of the
same VM - which results in weird effects like trying to undefine libvirt
object twice.
This one pool/volume property makes sense to change dynamically. There
may be more such properties, but lets be on the safe side and take
whitelist approach - allow only selected (just one for now), instead of
blacklisting any harmful ones.
QubesOS/qubes-issues#3256
Do not check for accepted value only in constructor, do that in property
setter. This will allow enforcing the limit regardless of how the value
was set.
This is preparation for dynamic revisions_to_keep change.
QubesOS/qubes-issues#3256
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.
This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.
Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.
Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
If domain got removed during the tests (for example DispVM), vm.close()
wouldn't be called in cleanup and some file descriptors will be
leaked. Add event handler for cleaning this up. Do not use close()
method here, because it is destructive, but the object may still be used
by the test.
Allow to get default value even it isn't set currently. This will allow
(G)UI to present better view, without duplicating logic for default
value.
FixesQubesOS/qubes-issues#3197
Check resize on each template separately, because it involves VM's
scripts (either qubes.ResizeDisk service, or some startup script).
QubesOS/qubes-issues#3173
- clone all features, not just qrexec (especially include 'gui')
- do not leak VM reference on failed test
- add test for online root volume resize
QubesOS/qubes-issues#3173
Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).
Related QubesOS/qubes-issues#3020
* bug3164:
tests: add regression test for #3164
storage/lvm: make sure volume cache is refreshed after changes
storage/lvm: fix Volume.verify()
storage/lvm: remove old volume only after successfully cloning new one
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.
QubesOS/qubes-issues#3164
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
* fixes-20170929:
vm: do not start QubesDB watch instance multiple times
vm: report storage.stop() errors to log
vm: move comment
storage: fix method name in LinuxModules volume
Prevent removing domain that is referenced from anywhere
vm: add vm.klass property
Move QubesVM.{name,qid,uuid,label} to BaseVM
vm: do not allow deleting template property from AppVM and DispVM
vm/qubesvm: emit event on failed startup
vm/qubesvm: remove duplicated qmemman_client.close()
vm/dispvm: cleanup DispVM also on failed startup
vm/dispvm: fix error message
ext/block: properly list devtype=cdrom option
block: fix handling non-existing devices
block: improve handling device name and description
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
- Prefer instance attributes over local variables - the former ones do
not leak into traceback object and are cleaned up by tests framework.
- Use 'with' syntax for handling files.
- Use subprocess.DEVNULL instead of open('/dev/null') where applicable
- Delete local variables when not needed anymore.
- Fix str/bytes
- Call skipTest as early as possible - before doing any setup
- Fix networking tests - configuration commands needs to be called as
root (missing user= argument).
- Fix setting firewall - policy is no longer changeable
- Add missing loop.run_until_complete() calls.
- Convert subprocess.Popen to asyncio.create_subprocess_exec where
needed (when called process needs to communicate with qubesd).
- Cleanup processes (call .wait()).
There is no more significant difference between PV and HVM. VMs are HVM
by default anyway. More important for this test is difference between
Linux (with Qubes packages installed) and other OS-es. Rename tests
accordingly. The later one is still incomplete.
The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.
Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.
FixesQubesOS/qubes-issues#3115
* qubesos/pr/150:
qubes/tests: moar fixes
test-packages: add missing libvirt classes
qubes/tests: do not deadlock on .drain()
qubes/vm: put name= first in __repr__
tests: fix some memory leaks
tests: complain about memory leaks
tests: use one event loop and one libvirtaio impl
Recently libvirt removed support for changing event implementation.
Therefore we have to use a single, global one and we check if it is
empty between tests.
* cdrom-boot:
devices: fix error reporting
api/admin: implement admin.vm.device....Set.persistent
devices: implement DeviceCollection.update_persistent()
devices: move DeviceInfo definition earlier
api: do not fail events when listener is cancelled in the meantime
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.
FixesQubesOS/qubes-issues#3047
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).
FixesQubesOS/qubes-issues#3057
qvm-sync-clock in dom0 now synchronize only dom0 time. For VM time,
qvm-sync-clock needs to be called in VM. Also, both will communicate
with qubesd, so must be called asynchronously from tests.
Allow attached device to be converted from persistent to non-persistent
and the other way around.
This is to allow starting a VM with some device attached temporarily.
When VM is not running, it is possible to attach device only
persistently, so this change will allow to do that, then, after starting
the VM, change it to non-persistent - so it will not be attached again
at further startups.
QubesOS/qubes-issues#3055
This is because .tearDown() is not executed if the exception occurs in
setUp() [for example self.skipTest() raises an exception]. The lower
levels of .tearDown() being executed are critical to not leaking file
descriptors.
Add public Admin API call to create Disposable VM that would be
automatically destroyed after shutdown. Do not keep this functionality
for qrexec-policy tool only.
Also, use admin.vm.Start there, instead of internal.vm.Start and
admin.vm.Kill instead of internal.vm.CleanupDispVM (this is enough,
because DispVM now have auto_cleanup property).
QubesOS/qubes-issues#2974
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.
New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).
Tests included.
QubesOS/qubes-issues#2974
* services:
tests: check clockvm-related handlers
doc: include list of extensions
qubesvm: fix docstring
ext/services: move exporting 'service.*' features to extensions
app: update handling features/service os ClockVM
* tests-storage:
tests: register libvirt events
tests: even more agressive cleanup in tearDown
app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
api: keep track of established connections
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
storage: fix Storage.clone and Storage.clone_volume
tests: more tests fixes
firewall: raise ValueError on invalid hostname in dsthost=
qmemman: don't load qubes.xml
tests: fix AdminVM test
tests: create temporary files in /tmp
tests: remove renaming test - it isn't supported anymore
tests: various fixes for storage tests
tests: fix removing LVM volumes
tests: fix asyncio usage in some tests
tests: minor fixes to api/admin tests
storage/file: create -cow.img only when needed
storage: move volume_config['source'] filling to one place
app: do not create 'default' storage pool
app: add missing setters for default_pool* global properties
* qdb-watch:
tests: add qdb_watch test
ext/block: make use of QubesDB watch
vm: add API for watching changes in QubesDB
vm: optimize imports
api/admin: don't send internal events in admin.Events
Add explanation why admin.vm.volume.Import is a custom script
Follow change of qubesdb path return type
Rename vm.qdb to vm.untrusted_qdb
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.
QubesOS/qubes-issues#853
