* devel-20180115:
tests: update PCI devices tests for core3 API
tests: make tests.extra.VMWrapper hashable
qmemman: do not close stdout/stderr in daemon mode
qmemman: fix early crash
qubesvm: do not try to define libvirt object in offline mode
tests: one more missing virt_mode=hvm
qubesvm: PVH minor improvements
tests: minor fixes
tests: add run_service and qrexec_policy wrappers to ExtraTestCase
tests: clear PCIDevice cache after each test
importing pkg_resources and looking up entry points wastes 100ms+
of time, which is totally unnecessary
This is based on
QubesOS/qubes-core-admin-client@b731ef3885
by @qubesuser
clear_outdated_error_markers crashes if memory stats are not retrieved
yet. In practice it crashes at the very first call during daemon
startup, making the whole qmemman unusable.
This fixes bf4306b815
qmemman: clear "not responding" flags when VM require more memory
QubesOS/qubes-issues#3265
- use capital letters in acronyms in documentation to match upstream
documentation.
- refuse to start a PVH with without kernel set - provide meaningful
error message
Forward-ported from qubes-core-agent-linux:
commit aad6fa6d190d24393e326a4c2ff7ebc3b5921641
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date: Sat Sep 30 04:56:02 2017 +0200
Hint shellcheck where to look for sourced files, if in repository
This will ease running shellcheck from the repository.
* qubesos/pr/180:
vm/qubesvm: default to PVH unless PCI devices are assigned
vm/qubesvm: expose 'start_time' property over Admin API
vm/qubesvm: revert backup_timestamp to '%s' format
doc: link qvm-device man page for qvm-block, qvm-pci, qvm-usb
* qubesos/pr/179:
qmemman: request VMs balloon down with 16MB safety margin
qmemman: clear "not responding" flags when VM require more memory
qmemman: slightly improve logging
qmemman: reformat code, especially comments
Human readable format `str(datetime.datetime)` is a nightmare for Admin
API level communication. Especially setting the property in a format
that it was read was not supported, and handling such format in
untrusted input handling code is a bad idea. Revert to a simple intiger
format.
It looks like Linux balloon driver do not always precisely respect
requested target memory, but perform some rounding. Also, in some cases
(HVM domains), VM do not see all the memory that Xen have assigned to it
- there are some additional Xen pools for internal usage.
Include 16MB safety margin in memory requests to account for those two
things. This will avoid setting "no_response" flag for most of VMs.
QubesOS/qubes-issues#3265
Clear slow_memset_react/no_progress flags when VM request more memory
than it have assigned. If there is some available, it may be given to
such VM, solving the original problem (not reacting to balloon down
request). In any case, qmemman algorithm should not try to take away
memory from under-provisioned VM.
FixesQubesOS/qubes-issues#3265
Add logging more info about each domain state:
- last requested target
- no_progress and slow_memset_react flags
This makes it unnecessary to log separately when those flags are cleared.
Rename events:
- domain-feature-set -> domain-feature-set:feature
- domain-feature-delete -> domain-feature-delete:feature
- domain-tag-add -> domain-tag-add:tag
- domain-tag-delete -> domain-tag-delete:tag
Make it consistent with property-* events. It makes more sense to
include tag/feature name in event name, so handler can watch a single
tag/feature - which is the most common case. Otherwise, most handlers
would begin with `if feature == '...'` anyway, wasting time on most
events.
In cases where multiple features/tags should be handled by a single
handler, it is now possible to register a handler with wildcard, for
example `domain-feature-set:*`.
Support registering handlers for more flexible wildcard events: not only
'*', but also 'something*'. This allows to register handlers for
'property-set:*' and such.
When creating a new VM of type DispVM without specifying any template
(e.g. "qvm-create --class DispVM --label red foo"), use default_dispvm.
Otherwise it would fail saying "Got empty response from qubesd."
Load integration tests from outside of core-admin repository, through
entry points.
Create wrapper for VM object to keep very basic compatibility with tests
written for core2. This means if test use only basic functionality
(vm.start(), vm.run()), the same test will work for both core2 and
core3. This is especially important for app-* repositories, where the
same version serves multiple Qubes branches.
This also hides asyncio usage from tests writer.
See QubesOS/qubes-issues#1800 for details on original feature.
Test base functions of dom0 module (creating VM, setting property) and
configuring system inside of VM (through DispVM). The later is done for
each available template (the process use salt installed in that
template, not copied from dom0).
QubesOS/qubes-issues#3316
