Remove intermediate qubesd-query-fast proxy process.
This requires changing socket protocol to match what qrexec is sending
in the header.
FixesQubesOS/qubes-issues#3293
* origin/pr/326:
ext/admin: workaround for extension's __init__() called multiple times
tests: teardown fixes
travis: include core-qrexec in tests
api/admin: (ext/admin) limit listing VMs based on qrexec policy
api/internal: extract get_system_info() function
- wait for the client be listed in dom0
- report parecord stderr
- allow up to 20ms to be missing, to account for potentially suspended
device initially
Make the check if remote file wasn't removed meaningful. Previously the
user didn't have permission to remote the source file, so even if the
tool would try, it would fail.
Various qrexec tests create auxiliary process (service_proc) as a local
variable. In case of test failure, process cleanup isn't called and may
lead to FD leaks and breaking subsequent tests.
Fix this by always saving such process instance in self.service_proc and
cleaning it up in self.tearDown() (this code is already there).
Add also waiting (and in case of timeout - killing) of a service call
process too.
First the main bug: when meminfo xenstore watch fires, in some cases
(just after starting some domain) XS_Watcher refreshes internal list of
domains before processing the event. This is done specifically to
include new domain in there. But the opposite could happen too - the
domain could be destroyed. In this case refres_meminfo() function raises
an exception, which isn't handled and interrupts the whole xenstore
watch loop. This issue is likely to be triggered by killing the domain,
as this way it could disappear shortly after writing updated meminfo
entry. In case of proper shutdown, meminfo-writer is stopped earlier and
do not write updates just before domain destroy.
Fix this by checking if the requested domain is still there just after
refreshing the list.
Then, catch exceptions in xenstore watch handling functions, to not
interrupt xenstore watch loop. If it gets interrupted, qmemman basically
stops memory balancing.
And finally, clear force_refresh_domain_list flag after refreshing the
domain list. That missing line caused domain refresh at every meminfo
change, making it use some more CPU time.
While at it, change "EOF" log message to something a bit more
meaningful.
Thanks @conorsch for capturing valuable logs.
FixesQubesOS/qubes-issues#4890
... during tests.
qubes.ext.Extension class is a weird thing that tries to make each extension
a singleton. But this unfortunately have a side effect that __init__()
is called separately for each "instance" (created in Qubes()'s
__init__()), even though this is really the same object. During normal
execution this isn't an issue, because there is just one Qubes() object
instance. But during tests, multiple objects are created.
In this particular case, it caused PolicyCache() to be created twice and
the second one overriden the first one - without properly cleaning it
up. This leaks a file descriptor (inotify one). The fact that cleanup()
was called twice too didn't helped, because it was really called on
the same object, the one requiring cleanup was already gone.
Workaround this by checking if policy_cache field is initialize and
avoid re-initialize it. Also, on Qubes() object cleanup remove that
field, so it can be properly initialized on the next test iteration.
Add few missing app.close() calls on test teardown.
Fix socket cleanup in TC_00_QubesDaemonProtocol() - not only close the
FD, but also unregister it from asyncio event loop.
Various Admin API calls, when directed at dom0, retrieve global system
view instead of a specific VM. This applies to admin.vm.List (called at
dom0 retrieve full VM list) and admin.Events (called at dom0 listen for
events of all the VMs). This makes it tricky to configure a management
VM with access to limited set of VMs only, because many tools require
ability to list VMs, and that would return full list.
Fix this issue by adding a filter to admin.vm.List and admin.Events
calls (using event handlers in AdminExtension) that filters the output
using qrexec policy. This version evaluates policy for each VM or event
(but loads only once). If the performance will be an issue, it can be
optimized later.
FixesQubesOS/qubes-issues#5509
* origin/pr/330:
gui: fixes from Marek's comments
gui: improvements of feature keyboard layout checks
tests: adapt tests for keyboard-layout
gui: drop legacy qubes-keyboard support
and not mark it as expected failure anymore. Note the removal of the
expected failure isn't just about the changes here, but also about the
actual fix on the qrexec side (ffafd01 "Fix not closed file descriptors in
qubes-rpc-multiplexer" commit in core-qrexec repository).
QubesVM.run_for_stdio() by default captures stderr. In case of call fail
(non-zero return code), captured stderr is included in the exception
object, but isn't printed by default CalledProcessError message.
Make it visible by:
- handling CalledProcessError and including in the test failure message
(when exception is captured already)
- not capturing stderr (if no exception handling is present in the
test)
The main use case for this function is to create qrexec services in VMs.
Since qrexec now require service scripts to be executable, make
create_remote_file() adjust permissions.
If any test-* VMs remains from previous test run, there are removed
before test. self.app doesn't exist at this point, so don't require it
in self.remove_vms().
This commit adds a test case for the QubesVM class's is_fully_usable
method. The verified scenarios are as follows:
* The VM has qrexec enabled, and the qrexec service has been
successfully started.
(The VM becomes "fully usable" in this case.)
* The VM has qrexec enabled, and the qrexec service has failed to start
(Error handling case; the VM is *not* fully usable.)
* The VM does *not* have qrexec enabled.
(The VM becomes "fully usable" in this case.)
Prior to this commit, a properly configured Linux HVM would not
transition from the 'Transient' state to the 'Running' state according
to qvm-ls output, even if the HVM in question had the 'qrexec' feature
disabled.
This issue is caused by an unconditional qrexec check in the
'on_domain_is_fully_usable' method, and is resolved by adding
a check that short-circuits the qrexec check if the aforementioned
feature is not enabled for the VM in question.
* origin/pr/295:
tests: fix tag name in audiovm test
tests: ensure notin while setting Audio/Gui VM
gui: add checks for changing/removing guivm
audio: add checks for changing/removing audiovm
audio/gui: use simply vm.tags instead of list()
tests: fix tests for gui/audio vm
Make pylint happy
gui/audio: fixes from Marek's comments
Allow AudioVM to be ran after any attached qubes
Allow GuiVM to be ran after any attached qubes
xid: ensure vm is not running
tests: fix missing default audiovm and guivm tags
gui, audio: better handling of start/stop guivm/audiovm
gui, audio: ensure guivm and audiovm tag are set
Support for AudioVM
Wait specifically for pulseaudio to start in the VM, instead of just
waiting few seconds.
Also, improve failure message to distinguish total lack of audio from
just missing few samples.
Volume with save_on_stop=False and snap_on_start=False exists only when
VM is running and this is already handled under is_dirty() condition.
For non-running VMs, skip LVM resize step and only update internal size
attribute.
FixesQubesOS/qubes-issues#5645
When volume is resized, underlying LVM is adjusted if exists. For
persistent volumes it is enough, as generally the driver prefers
underlying LVM size than internal attribute. But volatile volume is
removed at vm shutdown, so that value is lost.
Fix it by updating Volume.size attribute too, to be updated in qubes.xml
later too.
FixesQubesOS/qubes-issues#5645
* origin/pr/316:
storage/reflink: trivial style tweaks
storage/reflink: comment on _get_size() use in start()
storage/reflink: don't bother using _get_size() in create()
storage/reflink: delete all images at beginning of create()
storage/reflink: get VM dir from less arbitrary-looking path
storage/reflink: factor out _remove_all_images()
storage/reflink: _remove_incomplete_{files -> images}()
storage/reflink: omit redundant comment
Ensure that there are no leftover image files for the volume, e.g. from
an unsuccessful removal of a previous incarnation of this vid, or from
an messily restored pool filesystem backup. We don't want to preserve
any stale data (revisions) or metadata (size) in the new incarnation.
The size config parameter might be a string coming from XML.
The Volume base class handles the conversion to integer already.
FixesQubesOS/qubes-issues#5219.
This should allow importing a volume and changing the size at the
same time, without performing the resize operation on original
volume first.
The internal API has been renamed to internal.vm.volume.ImportBegin
to avoid confusion, and for symmetry with ImportEnd.
See QubesOS/qubes-issues#5239.
* origin/pr/310:
storage/reflink: fix comment
storage/reflink: bail out early on most FICLONE errnos
storage/reflink: pool.setup_check -> pool._setup_check
Don't fall back on 'cp' if the FICLONE ioctl gives an errno that's not
plausibly reflink specific, because in such a case any fallback could
theoretically mask real but intermittent system/storage errors.
Looking through ioctl_ficlone(2) and the kernel source, it should be
sufficient to do the fallback only on EBADF/EINVAL/EOPNOTSUPP/EXDEV.
(EISDIR/ETXTBSY don't apply to this storage driver, which will never
legitimately attempt to reflink a directory or an active - in the
storage domain - swap file.)
* origin/pr/303:
Update tests after adding /connected-ips
Also reload /connected-ips on firewall change / domain spawn
Also store /connected-ips6 for machines that have IPv6 addresses
Don't try to write to qubesdb of an offline VM
Maintain a list of connected machine IPs in qubesdb
Extension objects are singletons and normally do not require any special
cleanup. But in case of tests, we try to remove all the qubes objects
between tests and the cache in usb extension makes it hard.
Add a 'qubes-close' event that extensions can handle to remove extra
references stored in extension objects themselves.
Due to strangeness of KeyError (it overrrides __str__ method) in some
cases exceptions received superflous quotation marks when inheriting
from it.
fixesQubesOS/qubes-issues#5106
* origin/pr/298:
tests/network: let xl devd bring the interfaces up
tests/network: improve error reporting
api/admin: implement *.property.GetAll methods
Avoid resetting clocksync service of just enabled clockvm
doc/tests: extend qubes-specific quirks in tests
tests: add include and exclude lists for extra tests loader
xl devd doesn't use startup notify, so when the service is started
(according to systemd) it may still be initializing interfaces. Add a
little sleep for that.
Allow getting all the VM properties with one call. This greatly improve
performance of an applications retrieving many/all of them (qvm-ls,
qubes manager etc)
QubesOS/qubes-issues#5415FixesQubesOS/qubes-issues#3293
One alternative would look like
import ctypes
sizeof_int = ctypes.sizeof(ctypes.c_int)
FICLONE = (1073741824 % 256**sizeof_int) | 37897 | (sizeof_int << 16)
but, even if the above really(?) is a 100% correct Python port of
$ echo FICLONE | cpp -include linux/fs.h | tail -n 1
it still seems more likely that the ctypes package is somehow buggy
somewhere than for Qubes storage to run on an exotic architecture with
non 32 bit ints (in the foreseeable future).
So just document the baked in assumption.
The default (= text) mode for a loop device which contains a VM image
looked weird, even though it didn't make a difference here because the
dev_io object was never actually read from.
When setting global clockvm property, 'clocksync' service is
automatically added to the new value and then removed from the old one.
But if the new value is the same as the old one, the service gets
removed from the just set new value.
Check for this case explicitly.
FixesQubesOS/qubes-issues#4939
'extra' tests run is getting ridiculously long. Allow splitting it into
several jobs. Since this appears as just one class from the test loader
perspective, implement it as environment variables:
- QUBES_TEST_EXTRA_INCLUDE - load just selected tests
- QUBES_TEST_EXTRA_EXCLUDE - skip selected tests (to select "the rest"
tests)
* origin/pr/297:
doc: remove useless _static generated by sphinx
cleanup-dispvms: fix python shebang
spec: fix missing dependency
Fix Sphinx 2 new API for Fedora 31+
doc: Make PEP8 happier
qmemmand: separate SystemState init xc and xs to a 'init' method
doc: drop moved elsewhere components
Only first 4 disks can be emulated as IDE disks by QEMU. Specifically,
CDROM must be one of those first 4 disks, otherwise it will be
ignored. This is especially important if one wants to boot the VM from
that CDROM.
Since xvdd normally is a kernel-related volume (boot image, modules) it
makes perfect sense to re-use it for CDROM. It is either set for kernel
volume (in which case, VM should boot from it and not the CDROM), or
(possibly bootable) CDROM.
This needs to be done in two places:
- BlockExtension for dynamic attach
- libvirt xen.xml - for before-boot attach
In theory the latter would be enough, but it would be quite confusing
that device will get different options depending on when it's attached
(in addition to whether the kernel is set - introduced here).
This all also means, xvdd not always is a "system disk". Adjust listing
connected disks accordingly.
- allow TestQubesDB to be populated with initial data
- support list() method
- allow to register pre-created VM instance (useful for AdminVM, which
don't accept setting qid)
Pool.volumes property is implemented in a base class, individual drivers
should provide list_volumes() method as a backend for that property.
Fix this in a LinuxKernel pool.
Similar to admin.vm.volume.List. There are still other
admin.pool.volume.* methods missing, but lets start with just this one.
Those with both pool name and volume id arguments may need some more
thoughts.
Don't realy on a volume configuration only, it's easy to miss updating
it. Specifically, import_volume() function didn't updated the size based
on the source volume.
The size that the actual VM sees is based on the
file size, and so is the filesystem inside. Outdated size property can
lead to a data loss if the user perform an action based on a incorrect
assumption - like extending size, which actually will shrink the volume.
FixesQubesOS/qubes-issues#4821
Handle this case specifically, as way too many users ignore the message
during installation and complain it doesn't work later.
Name the problem explicitly, instead of pointing at libvirt error log.
FixesQubesOS/qubes-issues#4689
admin.pool.UsageDetails reports the usage data, unlike
admin.pool.Info, which should report the config/unchangeable data.
At the moment admin.Pool.Info still reports usage, to maintain
compatibility, but once all relevant tools are updated,
it should just return configuration data.
Added usage_details method to Pool class
(returns a dictionary with detailed information
on pool usage) and LVM implementation that returns
metadata info.
Needed for QubesOS/qubes-issues#5053
* origin/pr/287:
app: fix docstrings PEP8 refactor
tests: remove iptables_header content in test_622_qdb_keyboard_layout
tests: add test for guivm and keyboard_layout
gui: simplify setting guivm xid and keyboard layout
Make pylint happier
gui: set keyboard layout from feature
Handle GuiVM properties
Make PEP8 happier
Similar to domain-start-failed, add an event fired when
domain-pre-shutdown was fired but the actual operation failed.
Note it might not catch all the cases, as shutdown() may be called with
wait=False, which means it won't wait fot the actual shutdown. In that
case, timeout won't result in domain-shutdown-failed event.
QubesOS/qubes-issues#5380
Move this functionality from our custom runner (qubes.tests.run),
into base test class. This is very useful for correlating logs, so lets
have it with nose2 runner too.
Prevent starting a VM while it's being removed. Something could try to
start a VM just after it's being killer but before removing it (Whonix
example from previous commit is real-life case). The window specifically
is between kill() call and removing it from collection
(`del app.domains[vm.qid]`). Grab a startup_lock for the whole operation
to prevent it.
Check early (but after grabbing a startup_lock) if VM isn't just
removed. This could happen if someone grabs its reference from other
places (netvm of something else?) or just before removing it.
This commit makes the simple removal from the collection (done as the
first step in admin.vm.Remove implementation) efficient way to block
further VM startups, without introducing extra properties.
For this to be effective, removing from the collection, needs to happen
with the startup_lock held. Modify admin.vm.Remove accordingly.
There are cases when destination domain doesn't exist when the call gets
to qubesd. Namely:
1. The call comes from dom0, which bypasses qrexec policy
2. Domain was removed between checking the policy and here
Handle the the same way as if the domain wouldn't exist at policy
evaluation stage either - i.e. refuse the call.
On the client side it doesn't change much, but on the server call it
avoids ugly, useless tracebacks in system journal.
FixesQubesOS/qubes-issues#5105
Allow to manual inspect test environment after test fails. This is
similar to --do-not-clean option we had in R3.2.
The decorator should be used only while debugging and should never be
applied to the code committed into repository.
Domain shutdown handling may take extended amount of time, especially on
slow machine (all the LVM teardown etc). Take care of it by
synchronizing using vm.startup_lock, instead of increasing constant
delay. This way, the shutdown event handler needs to be started within
3s, not finish in this time.
When libvirt daemon is restarted, qubesd attempt to re-connect to the
new instance transparently (through virConnect object wrapper). But the
code lacked re-registering event handlers.
Fix this by adding reconnect callback argument to virConnectWrappper, to
be called after new connection is established. This callback will
additionally get old connection as an argument, if any cleanup is
needed. The old connection is closed just after callback returns.
Use this to re-register event handler, but also unregister old handler
first. While full unregister wont work on since old libvirt daemon
instance is dead already, it will still cleanup client structures.
Since the old libvirt connection is closed now, adjust also domain
reconnection logic, to handle stale connection object. In that case
isAlive() call throws an exception.
FixesQubesOS/qubes-issues#5303
Qubesd wrongly required default_template global property to be not None.
Furthermore, even without hard failure set, require_property method
raised an exception in case of a property having incorrect None value.
It now logs an error message instead, as designed.
fixesQubesOS/qubes-issues#5326
This fixes an invalid response generated by get_timezone when the time zones are composed by 3 parts, for example:
America/Argentina/Buenos_Aires
America/Indiana/Indianapolis
Update utils.py
Give raw cpu_time value, instead of normalized one (to number of vcpus),
as documented.
Move the normalization to cpu_usage calculation. At the same time, add
cpu_usage_raw without it, if anyone needs it.
QubesOS/qubes-issues#4531
* origin/pr/273:
tests: check importing empty data into ReflinkVolume
tests: check importing empty data into ThinVolume
tests: check importing empty data into FileVolume
tests: improve cleanup after LVM tests
During regular VM shutdown, the VM should sync() anyway. (And
admin.vm.volume.Import does fdatasync(), which is also fine.) But let's
be extra careful.
This is needed as a consequence of d8b6d3ef ("Make add_pool/remove_pool
coroutines, allow Pool.{setup,destroy} as coroutines"), but there hasn't
been any problem so far because no storage driver implemented pool
setup() as a coroutine.
Revert to use umount -l in storage tests cleanup. With fixed permissions
in 4234fe51 "tests: fix cleanup after reflink tests", it shouldn't cause
issues anymore, but apparently on some systems test cleanup fails
otherwise.
Reported by @rustybird
This reverts commit b6f77ebfa1.
There were (at least) five ways for the volume's nominal size and the
volume image file's actual size to desynchronize:
- loading a stale qubes.xml if a crash happened right after resizing the
image but before saving the updated qubes.xml (-> previously fixed)
- restarting a snap_on_start volume after resizing the volume or its
source volume (-> previously fixed)
- reverting to a differently sized revision
- importing a volume
- user tinkering with image files
Rather than trying to fix these one by one and hoping that there aren't
any others, override the volume size getter itself to always update from
the image file size. (If the getter is called though the storage API, it
takes the volume lock to avoid clobbering the nominal size when resize()
is running concurrently.)
And change the volume lock from an asyncio.Lock to a threading.Lock -
locking is now handled before coroutinization.
This will allow the coroutinized resize() and a new *not* coroutinized
size() getter from one of the next commits ("storage/reflink: preferably
get volume size from image size") to both run under the volume lock.
Successfully resize volumes without any currently existing image file,
e.g. cleanly stopped volatile volumes: Just update the nominal size in
this case.
