New qvm-backup tool can either use pre-existing backup profile
(--profile), or - when running in dom0 - can create new one based on
used options (--save-profile).
This commit add a tool itself, update its man page, and add tests for
it.
FixesQubesOS/qubes-issues#2931
Do not check for qubesd socket (at module import time), because if not
running at this precise time, it will lead to wrong choice. And a weird
error message in consequence (looking for qrexec-client-vm in dom0).
FixesQubesOS/qubes-issues#2917
* devel-2-qvm-run-1:
Make pylint happy
tools/qvm-run: fix handling EOF
tests: mark qvm-run tests with "expected failure"
tools/qvm-run: fix handling copying stdin to the process
since qvm-run use multiprocessing.Process now, stdin sent to it is
processed in separate process and doesn't come back to
TestApp.actual_calls (self.app). Annotate tests for now, to be fixed
later.
The most important part is fixing resize handling - call size_func
before data_func, but after tar gets initial data (and output file
size).
But other than that, it makes the process a little faster.
QubesOS/qubes-issues#1214
Opt for a simple one-liner error messages, instead of meaningless stack
trace (it's most of the time about qubesd responding with error, so the
stack trace of actual problem is elsewhere).
This will be less realistic (private.img of 2MB?!), but makes tests much
quicker. And since tar is used to make files sparse, we don't really
test multi-part archives anyway.
Based on "backup compatibility" tests, which manually assemble the
backup. This is because we don't have access to actual backup creation
code here.
QubesOS/qubes-issues#1214
Launch stdin copy loop in a separate process (multiprocessing.Process)
and terminate it when target process is terminated.
Another idea here was threads, but there is no API to kill a thread
waiting on read().
VMs can have runtime dependencies - for example it isn't possible to
shutdown netvm used by some other running VM(s). Since client-side tools
may not have full knowledge about rules enforcing those dependencies
(for example may not have access to 'netvm' property), implement
best-effort approach:
1. Try to shutdown all requested VMs
2. For those where shutdown request succeed, wait for actual shutdown
3. For others - go back to step 1
And loop until all VMs are shutdown, or all shutdown requests fails.
Allow programming different responses for the same request when called
multiple times. This is useful for example for shutdown tests - first
domain is running, but after issuing shutdown request is is not.
Do not close event loop in utility function - handle it only in main().
For this reason, change appropriate functions to coroutines.
FixesQubesOS/qubes-issues#2865
Don't fail the whole process when "just" appmenus import fails.
But if data import fails, remove the VM
Also update for vm.run_service_for_stdio raising CalledProcessError.
Don't start GUI daemon for given VM twice when qvm-start-gui was started
during VM startup (while waiting for qrexec startup). This is especially
common while running tests.
Report failed qubes.SetMonitorLayout as warning (instead of unhandled
exception).
Clear VM cache on qubesd reconnect.
Fix logging.
qubesd may be restarted during different stages of connection - either
while attempting to connect, or while already listening on events.
Adjust exception list accordingly.
This is especially important for qvm-start-gui - otherwise it crashes on
qubesd restart.
This way we don't need separate admin.vm.Clone call, which is tricky to
handler properly with policy.
A VM may not have access to all the properties and other metadata, so
add ignore_errors argument, for best-effort approach (copy what is
possible). In any case, failure of cloning VM data fails the whole
operation.
When operation fails, VM is removed.
While at it, allow to specify alternative VM class - this allows
morphing one VM into another (for example AppVM -> StandaloneVM).
Adjust qvm-clone tool and tests accordingly.
QubesOS/qubes-issues#2622
Enable 'qrexec' VM feature to wait for qrexec initialization - it is
required to call qubes.PostInstall service. If VM start fails, assume
there is no qrexec and drop that feature.
Since Admin API, qvm-ls takes a long time to complete. Therefore,
Corporate Headquarters commanded that a Enterprise Spinner is to be
implemented and mandated it's use unto us.
We take amusement from its endless gyrations.
Empty parameter value is encoded as b'parameter\0\0', so we can't simply
read the data until b'\0\0', because it isn't necessary event end.
Instead, read event parts separately, according to specification.
Use newly introduced payload_stream= argument to qubesd_call to pass
data directly from some file-like object - without loading it all into
memory.
QubesOS/qubes-issues#853
Add qubesd_call(..., payload_stream=...) argument to allow streaming
payload directly from some file/process stdout. This is mainly (only?)
useful for admin.vm.volume.Import, where disk volume raw data is passed
to the service.
Since migration to Admin API, qvm-ls no longer have list of all VM
properties in advance, so can't really validate fields list. Simply
assume that unknown columns are properties.
Don't terminate qvm-run on any SIGCHLD, check if the process we're
waiting for have finished.
Currently the only situation when it's broken is a test (which starts
additional process, whose SIGCHLD may be caught here), but lets do not
assume that much (starting only one process) about environment.
Make it easy to retrieve DeviceInfo object out of DeviceAssignment
object. The only missing piece of information for that is device class,
so add it. Make it optional, as it can be filled on demand when passing
the object through DeviceCollection (either by listing devices, or
attaching/detaching).
This also makes DeviceCollection._device method not needed anymore.
It is supported only from dom0, but it's still useful to have, to save
on simultaneous vchan connections (only waiting for MSG_DATA_EXIT_CODE).
This is especially important for Windows VMs, as qrexec-agent there have
pretty low limit on simultaneous connections (about 20).
Make qvm-run use it.
Since we use qubes.VMShell service now and send requested command on its
stdin, we need to terminate that shell after requested command -
otherwise the service will not terminate automatically waiting for
further input (next commands).
qubesd_call expect "bytes" type. Additionally serialize false value as
empty string (which is treated by python as false value), because
otherwise would be serialized to (non-empty) string, which is true value
in python.
Do not color qvm-run diagnostic messages, but also avoid ANSI control
sequences in logs. While at it, do not print 'Running ...' message when
--pass-io is used.
