Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).
Related QubesOS/qubes-issues#3020
* bug3164:
tests: add regression test for #3164
storage/lvm: make sure volume cache is refreshed after changes
storage/lvm: fix Volume.verify()
storage/lvm: remove old volume only after successfully cloning new one
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.
QubesOS/qubes-issues#3164
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
* fixes-20170929:
vm: do not start QubesDB watch instance multiple times
vm: report storage.stop() errors to log
vm: move comment
storage: fix method name in LinuxModules volume
Prevent removing domain that is referenced from anywhere
vm: add vm.klass property
Move QubesVM.{name,qid,uuid,label} to BaseVM
vm: do not allow deleting template property from AppVM and DispVM
vm/qubesvm: emit event on failed startup
vm/qubesvm: remove duplicated qmemman_client.close()
vm/dispvm: cleanup DispVM also on failed startup
vm/dispvm: fix error message
ext/block: properly list devtype=cdrom option
block: fix handling non-existing devices
block: improve handling device name and description
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
- Prefer instance attributes over local variables - the former ones do
not leak into traceback object and are cleaned up by tests framework.
- Use 'with' syntax for handling files.
- Use subprocess.DEVNULL instead of open('/dev/null') where applicable
- Delete local variables when not needed anymore.
- Fix str/bytes
- Call skipTest as early as possible - before doing any setup
- Fix networking tests - configuration commands needs to be called as
root (missing user= argument).
- Fix setting firewall - policy is no longer changeable
- Add missing loop.run_until_complete() calls.
- Convert subprocess.Popen to asyncio.create_subprocess_exec where
needed (when called process needs to communicate with qubesd).
- Cleanup processes (call .wait()).
There is no more significant difference between PV and HVM. VMs are HVM
by default anyway. More important for this test is difference between
Linux (with Qubes packages installed) and other OS-es. Rename tests
accordingly. The later one is still incomplete.
The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.
Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.
FixesQubesOS/qubes-issues#3115
* qubesos/pr/150:
qubes/tests: moar fixes
test-packages: add missing libvirt classes
qubes/tests: do not deadlock on .drain()
qubes/vm: put name= first in __repr__
tests: fix some memory leaks
tests: complain about memory leaks
tests: use one event loop and one libvirtaio impl
Recently libvirt removed support for changing event implementation.
Therefore we have to use a single, global one and we check if it is
empty between tests.
* cdrom-boot:
devices: fix error reporting
api/admin: implement admin.vm.device....Set.persistent
devices: implement DeviceCollection.update_persistent()
devices: move DeviceInfo definition earlier
api: do not fail events when listener is cancelled in the meantime
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.
FixesQubesOS/qubes-issues#3047
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).
FixesQubesOS/qubes-issues#3057
qvm-sync-clock in dom0 now synchronize only dom0 time. For VM time,
qvm-sync-clock needs to be called in VM. Also, both will communicate
with qubesd, so must be called asynchronously from tests.
Allow attached device to be converted from persistent to non-persistent
and the other way around.
This is to allow starting a VM with some device attached temporarily.
When VM is not running, it is possible to attach device only
persistently, so this change will allow to do that, then, after starting
the VM, change it to non-persistent - so it will not be attached again
at further startups.
QubesOS/qubes-issues#3055
This is because .tearDown() is not executed if the exception occurs in
setUp() [for example self.skipTest() raises an exception]. The lower
levels of .tearDown() being executed are critical to not leaking file
descriptors.
Add public Admin API call to create Disposable VM that would be
automatically destroyed after shutdown. Do not keep this functionality
for qrexec-policy tool only.
Also, use admin.vm.Start there, instead of internal.vm.Start and
admin.vm.Kill instead of internal.vm.CleanupDispVM (this is enough,
because DispVM now have auto_cleanup property).
QubesOS/qubes-issues#2974
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.
New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).
Tests included.
QubesOS/qubes-issues#2974
* services:
tests: check clockvm-related handlers
doc: include list of extensions
qubesvm: fix docstring
ext/services: move exporting 'service.*' features to extensions
app: update handling features/service os ClockVM
* tests-storage:
tests: register libvirt events
tests: even more agressive cleanup in tearDown
app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
api: keep track of established connections
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
storage: fix Storage.clone and Storage.clone_volume
tests: more tests fixes
firewall: raise ValueError on invalid hostname in dsthost=
qmemman: don't load qubes.xml
tests: fix AdminVM test
tests: create temporary files in /tmp
tests: remove renaming test - it isn't supported anymore
tests: various fixes for storage tests
tests: fix removing LVM volumes
tests: fix asyncio usage in some tests
tests: minor fixes to api/admin tests
storage/file: create -cow.img only when needed
storage: move volume_config['source'] filling to one place
app: do not create 'default' storage pool
app: add missing setters for default_pool* global properties
* qdb-watch:
tests: add qdb_watch test
ext/block: make use of QubesDB watch
vm: add API for watching changes in QubesDB
vm: optimize imports
api/admin: don't send internal events in admin.Events
Add explanation why admin.vm.volume.Import is a custom script
Follow change of qubesdb path return type
Rename vm.qdb to vm.untrusted_qdb
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.
QubesOS/qubes-issues#853
Remove some more references to objects holding (possibly indirectly)
reference to libvirt connection:
- local variables in tearDown function
- running Admin API calls (especially admin.Events)
- vmm._libvirt_conn directly, in case some reference to Qubes()
is still there
- any instance attribute that is an object from 'qubes' python package
(instead of just those descending from BaseVM)
- do not create new Qubes() instance for removing VMs - if we already
have one in self.app
Then trigger garbage collector to really cleanup those objects (and
close relevant file descriptors). It's important do do this before
closing event loop, because some of descructors may try to use it (for
example remove registered handlers).
When tearDownClass is executed, event loop is already closed. Since no
test really need it right now, drop support for test class-wide VMs and
convert those methods back to instance methods.
Also put coroutines (vm.remove_from_disk, vm.kill) onto event loop.
Since it is no longer child of QubesVM, constructor do not take 'qid'
and 'name' arguments.
Also:
- remove other dropped properties tests (netvm, storage related)
- make the test working in non-dom0
- improve TestPool mock - init_volume now return appropriate mock type,
instead of TestPool
- improve patching base directory (/var/lib/qubes) - it is stored in
more than one place...
- fix inheritance in TC_01_ThinPool class
- fix expected LVM volume names ('vm-' prefix)
- fix cleanup after FilePool tests - remove temporary qubes.xml
- asyncio usage
- better reporting in integ.storage - include error message in the
report, not only as a comment in code
Don't set 'source' volume in various places (each VM class constructor
etc), do it as part of volume initialization. And when it needs to be
re-calculated, call storage.init_volume again.
This code was duplicated, and as usual in such a case, those copies
were different - one have set 'size', the other one not.
QubesOS/qubes-issues#2256
Since we have app.default_pool* properties, create appropriately named
pool and let those properties choose the right pool. This also means we
don't need to specify pool name in default volume config anymore
QubesOS/qubes-issues#2256
The old format have many issues and is discouraged by tar developers. In
this case the most important one is header with possible non-ASCII
characters, which will result in UnicodeDecodeError (tarfile module
require header parts in utf-8).
PAX format is much cleaner, as it use standard mechanism for extended
headers.
Changed the inheritance hierarchy:
1. Renamed `SystemTestsMixin` to `SystemTestCase`
2. `SystemTestCase` is a child of `QubesTestCase`
3. All classes extending the prior `SystemTestsMixin` now just extend `object`
* tests-fixes-1:
api: extract function to make pylint happy
tests/vm: simplify AppVM storage test
storage: do not use deepcopy on volume configs
api: cleanup already started servers when some later failed
tests: fix block devices tests when running on real system
tests: fix some FD leaks
Use minimal TestPool(), instead of Mock().
This allow effectively compare Volume instances (init_volume with the
same parameters return equal Volume instance).
This driver isn't used in default Qubes 4.0 installation, but if we do
have it, let it follow defined API and its own documentation. And also
explicitly reject not supported operations:
- support only revisions_to_keep<=1, but do not support revert() anyway
(implemented version were wrong on so many levels...)
- use 'save_on_stop'/'snap_on_start' properties directly instead of
obsolete volume types
- don't call sudo - qubesd is running as root
- consistently use path, path_cow, path_source, path_source_cow
Also, add tests for BlockDevice instance returned by
FileVolume.block_device().
QubesOS/qubes-issues#2256
This will allow starting processes and calling RPC services in those
events. This if required for usb devices, which are attached using RPC
services.
Intentionally keep device listing events synchronous only - to
discourage putting long-running actions there.
This change also require some not-async attach method version for
loading devices from qubes.xml - have `load_persistent` for this.
The first operation returns a token, which can be passed to the second
one to actually perform clone operation. This way the caller needs have
power over both source and destination VMs (or at least appropriate
volumes), so it's easier to enforce appropriate qrexec policy.
The pending tokens are stored on Qubes() instance (as QubesAdminAPI is
not persistent). It is design choice to keep them in RAM only - those
are one time use and this way restarting qubesd is a simple way to
invalidate all of them. Otherwise we'd need some additional calls like
CloneCancel or such.
QubesOS/qubes-issues#2622
In the end firewall is implemented as .Get and .Set rules, with policy
statically set to 'drop'. This way allow atomic firewall updates.
Since we already have appropriate firewall format handling in
qubes.firewall module - reuse it from there, but adjust the code to be
prepared for potentially malicious input. And also mark such variables
with untrusted_ prefix.
There is also third method: .Reload - which cause firewall reload
without making any change.
QubesOS/qubes-issues#2622FixesQubesOS/qubes-issues#2869
There is a problem with having separate default action ("policy") and
rules because it isn't possible to set both of them atomically at the
same time.
To solve this problem, always have policy 'drop' (as a safe default),
but by default have a single rule with action 'accept'
FixesQubesOS/qubes-issues#2869
Check if exit code retrieved from dom0 is really the one expected.
Fix typo in test_065_qrexec_exit_code_vm (testvm1/testvm2), adjust for
reporing remote exit code and remove expectedFailure.
QubesOS/qubes-issues#2861
Since tests expose qubesd socket, qvm-start-gui should handle starting
GUI daemons (so, GUI session inside VM). Add synchronization with it
using qubes.WaitForSession service.
When test expect to wait for remote process, use vm.run_for_stdio.
Additionally, when the call fail, (stdout, stderr) is not assigned - use
the one attached to exception object instead.
Since run_for_stdio raise an exception for non-zero exit code, it isn't
ignored anymore. So, check if qrexec-client-vm return expected value,
instead of keep ignoring it.
QubesOS/qubes-issues#2861
Test suite creates some VMs and needs to pass the knowledge about them
to qrexec policy checker. This is done using Admin API, so we need to
substitute qubesd with our own API server.
* qubesos/pr/111:
vm: drop 'internal' property
qmemman: make sure to release lock
qmemman: fix meminfo parsing for python 3
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
* qubesos/pr/110:
storage: use direct object references, not only identifiers
vm: fix volume_config
storage/lvm: prefix VM LVM volumes with 'vm-'
storage: fix VM rename
Make qubes.NotifyTools reuse logic of qubes.FeaturesRequest, then move
actual request processing to 'features-request' event handler. At the
same time implement handling 'qrexec' and 'gui' features request -
allowing to set template features when wasn't already there.
Behavior change: template is no longer allowed to change feature value
(regardless of being True or False). This means the user will always be
able to override what template have set.
Drop DeviceInfo.data - device extension should provide a subclass with
proper individual fields.
Drop DeviceAssignment.frontend_domain - this information is redundant -
frontend domain is defined by where DeviceAssignment is attached.
Rename DeviceCollection.devclass to bus - devclass if confusing here,
because this term is also used for DeviceInfo subclass.
