Commit Graph

335 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
ce2215c603
backup: improve error logging
Include VM and volume name in data-related error mesages.

QubesOS/qubes-issues#1214
2017-07-17 23:32:31 +02:00
Marek Marczykowski-Górecki
f2fa613dce
backup: use 'cat' instead of read-write loop in python
The most important part is fixing resize handling - call size_func
before data_func, but after tar gets initial data (and output file
size).
But other than that, it makes the process a little faster.

QubesOS/qubes-issues#1214
2017-07-17 23:30:37 +02:00
Marek Marczykowski-Górecki
96d4a2f066
backup: change 'hvm' property to 'virt_mode'
QubesOS/qubes-issues#2912
2017-07-17 20:40:08 +02:00
Marek Marczykowski-Górecki
ced735b476
backup: do not show full stacktrace to the user
Opt for a simple one-liner error messages, instead of meaningless stack
trace (it's most of the time about qubesd responding with error, so the
stack trace of actual problem is elsewhere).
2017-07-17 20:28:23 +02:00
Marek Marczykowski-Górecki
ca399c1a5a
app: call admin.vm.volume.Import as root
This is needed to write LVM data.
2017-07-17 20:28:23 +02:00
Marek Marczykowski-Górecki
f0151d73b3
tools: add qvm-backup-restore
Frontend tool for backup restore code.

Fixes QubesOS/qubes-issues#1214
2017-07-17 20:28:22 +02:00
Marek Marczykowski-Górecki
137e1ed877
tests: backup: add firewall.xml and appmenus checking 2017-07-17 20:28:22 +02:00
Marek Marczykowski-Górecki
51f77d5834
firewall: fix handling DstHost.prefixlen=0
Do not silently convert it into 32 or 128 netmask. And also do not
include it actual rule (it's no-op check).
2017-07-17 20:28:22 +02:00
Marek Marczykowski-Górecki
f1036c27a7
backup: add firewall and appmenus list handling 2017-07-17 20:28:21 +02:00
Marek Marczykowski-Górecki
04ad224a9d
tests: add v4 (Qubes 4.0) backup format tests, move qubes.xml
Move qubes.xml to be tested into separate files. The backup tests script
is long enouch already.
2017-07-17 20:28:21 +02:00
Marek Marczykowski-Górecki
525f8dc7f3
tests/backup: use smaller images
This will be less realistic (private.img of 2MB?!), but makes tests much
quicker. And since tar is used to make files sparse, we don't really
test multi-part archives anyway.
2017-07-17 20:28:20 +02:00
Marek Marczykowski-Górecki
a91372a919
devices,features: fix bool values handling
API define False value serialized as '' and True as 'True'. Do not
serialize 0 as '' (features) or True as 'yes' (devices).
2017-07-17 20:27:35 +02:00
Marek Marczykowski-Górecki
2ab31e63dc
Exclude 'qubesadmin.backup' module from python2 install 2017-07-17 20:27:35 +02:00
Marek Marczykowski-Górecki
e6d3425047
tests/backup: tests for backup-restore code
Based on "backup compatibility" tests, which manually assemble the
backup. This is because we don't have access to actual backup creation
code here.

QubesOS/qubes-issues#1214
2017-07-17 20:27:32 +02:00
Marek Marczykowski-Górecki
268a3453a9
backup: initial support for backup restore over Admin API
The code is mostly copied from core-admin.

QubesOS/qubes-issues#1214
2017-07-17 20:24:34 +02:00
Marek Marczykowski-Górecki
66f2e9c889
Merge remote-tracking branch 'qubesos/pr/11'
* qubesos/pr/11:
  added helper function updates_vms_status
2017-07-15 22:01:04 +02:00
Marek Marczykowski-Górecki
68ed06a200
Don't try to set 'created-by-' tag when cloning VM
This tag can't be set from outside of qubesd.
2017-07-14 04:14:46 +02:00
Marek Marczykowski-Górecki
941b553b81
tools/qvm-run: fix handling copying stdin to the process
Launch stdin copy loop in a separate process (multiprocessing.Process)
and terminate it when target process is terminated.
Another idea here was threads, but there is no API to kill a thread
waiting on read().
2017-07-08 00:11:15 +02:00
Marek Marczykowski-Górecki
a3e3dac6dc
tests: improve test failure message
When multiple Admin API calls are expected, but not all were made, show
which were missing.
2017-07-08 00:08:20 +02:00
Marek Marczykowski-Górecki
b35303ff62
events: fix cleanup function in qrexec case
qrexec-client-vm process may be already dead, don't treat this as an
error.
2017-07-08 00:08:20 +02:00
Marek Marczykowski-Górecki
e6149b09ce
Fix VM creation with default template
Fixes QubesOS/qubes-issues#2866
2017-07-08 00:08:19 +02:00
Marek Marczykowski-Górecki
9036103102
tests: qvm-shutdown --wait tests 2017-07-08 00:08:19 +02:00
Marek Marczykowski-Górecki
6b1c6141f6
tests/tools: add MockEventsReader
Make it easy to test things listening for events
2017-07-08 00:08:19 +02:00
Marek Marczykowski-Górecki
389252f386
tools: handle shutdown order in qvm-shutdown
VMs can have runtime dependencies - for example it isn't possible to
shutdown netvm used by some other running VM(s). Since client-side tools
may not have full knowledge about rules enforcing those dependencies
(for example may not have access to 'netvm' property), implement
best-effort approach:
1. Try to shutdown all requested VMs
2. For those where shutdown request succeed, wait for actual shutdown
3. For others - go back to step 1

And loop until all VMs are shutdown, or all shutdown requests fails.
2017-07-08 00:08:19 +02:00
Marek Marczykowski-Górecki
2052b32202
events: simplify wait_for_domain_shutdown coroutine
1. Handle timeout externally - using asyncio.wait_for.
2. Add support for waiting for multiple VMs.
2017-07-06 22:01:17 +02:00
Marek Marczykowski-Górecki
43ef244eaa
vm: make QubesVM objects hashable 2017-07-06 22:01:16 +02:00
Marek Marczykowski-Górecki
8f7b0009db
version 4.0.1 2017-07-05 20:14:12 +02:00
Marek Marczykowski-Górecki
a2d9303ea9
app: fix policy deny reporting when running in VM
qrexec-client-vm non-zero exit code means policy have denied the call.
Treat this exactly the same as empty response (in dom0 case).
2017-07-05 14:16:31 +02:00
Marek Marczykowski-Górecki
d8f018eb95
tools: clarify help in qvm-template-postprocess 2017-07-05 14:16:31 +02:00
Marek Marczykowski-Górecki
7c9699cd87
tools: split calling qubes.PostInstall service to a separate function
The code is long enough to warrant separate function (suggested by
pylint).
2017-07-05 14:16:31 +02:00
Marek Marczykowski-Górecki
627aebf2cd
tools: remove policy handling from qvm-firewall tool
Follow the API removal

QubesOS/qubes-issues#2869
2017-07-05 14:16:30 +02:00
Marek Marczykowski-Górecki
942e122d27
firewall: drop GetPolicy/SetPolicy calls
Firewall policy is now hardcoded to 'drop'. Keep the property, so anyone
trying to assign it will get an exception

QubesOS/qubes-issues#2869
2017-07-05 14:16:30 +02:00
Marek Marczykowski-Górecki
ade5083e5e
app: do not clone 'uuid' property
Cloned VM have new UUID
2017-07-05 14:16:30 +02:00
Marek Marczykowski-Górecki
c6eb4c49a3
storage: rework clone as two-stage operation
Split clone to two Admin API calls - one to the source volume, then
other to destination.
2017-07-05 14:16:30 +02:00
Marek Marczykowski-Górecki
88de4f77a5
tests: handle returning different response for the same request
Allow programming different responses for the same request when called
multiple times. This is useful for example for shutdown tests - first
domain is running, but after issuing shutdown request is is not.
2017-07-05 14:16:29 +02:00
Marta Marczykowska-Górecka
9174d8c378
added helper function updates_vms_status 2017-06-25 22:09:34 +02:00
Marek Marczykowski-Górecki
5430e04e1c
tools: move event loop creation/closing to main function
Do not close event loop in utility function - handle it only in main().
For this reason, change appropriate functions to coroutines.

Fixes QubesOS/qubes-issues#2865
2017-06-25 20:09:10 +02:00
Marek Marczykowski-Górecki
0012eb3ac6
tools/qvm-template-postprocess: improve error handling
Don't fail the whole process when "just" appmenus import fails.
But if data import fails, remove the VM

Also update for vm.run_service_for_stdio raising CalledProcessError.
2017-06-25 18:22:06 +02:00
Marek Marczykowski-Górecki
ce7d4865b6
events: fix calling mgmt.Events from VM
asyncio.create_subprocess_exec expects program and arguments directly,
not as a list.
2017-06-25 14:01:14 +02:00
Marek Marczykowski-Górecki
c545c95660
tests: fix qvm-run test on travis
stdout there is not a tty, which change default value of filter_esc.
2017-06-25 13:44:17 +02:00
Marek Marczykowski-Górecki
3cf5840d7a
Merge branch 'devel-4'
* devel-4:
  tools/qvm-start-gui: multiple fixes
  vm: raise CalledProcessError instead of QubesVMError on failed service call
  events: improve handling qubesd restart
2017-06-25 13:16:50 +02:00
Marek Marczykowski-Górecki
cef80a76e4
Merge branch 'devel-3'
* devel-3:
  Implement VM clone as create + copy data+metadata
  storage: make Volumes sortable
  base: add PropertyHolder.clone_properties
  doc: minor fixes to man pages
  storage: add volume clone method
  doc: fix skel-manpage tool
  tools: add qvm-tags tool
  tags support
2017-06-25 13:16:22 +02:00
Marek Marczykowski-Górecki
64377207a8
tools/qvm-start-gui: multiple fixes
Don't start GUI daemon for given VM twice when qvm-start-gui was started
during VM startup (while waiting for qrexec startup). This is especially
common while running tests.

Report failed qubes.SetMonitorLayout as warning (instead of unhandled
exception).

Clear VM cache on qubesd reconnect.

Fix logging.
2017-06-21 06:01:53 +02:00
Marek Marczykowski-Górecki
5ac7632dd0
vm: raise CalledProcessError instead of QubesVMError on failed service call
follow core-admin change.
2017-06-21 06:01:53 +02:00
Marek Marczykowski-Górecki
ba2057a2c6
events: improve handling qubesd restart
qubesd may be restarted during different stages of connection - either
while attempting to connect, or while already listening on events.
Adjust exception list accordingly.

This is especially important for qvm-start-gui - otherwise it crashes on
qubesd restart.
2017-06-21 05:03:04 +02:00
Marek Marczykowski-Górecki
bcd026d141
Implement VM clone as create + copy data+metadata
This way we don't need separate admin.vm.Clone call, which is tricky to
handler properly with policy.
A VM may not have access to all the properties and other metadata, so
add ignore_errors argument, for best-effort approach (copy what is
possible). In any case, failure of cloning VM data fails the whole
operation.
When operation fails, VM is removed.

While at it, allow to specify alternative VM class - this allows
morphing one VM into another (for example AppVM -> StandaloneVM).

Adjust qvm-clone tool and tests accordingly.

QubesOS/qubes-issues#2622
2017-06-20 01:34:18 +02:00
Marek Marczykowski-Górecki
a5a459840a
storage: make Volumes sortable
But do not fetch any additional info just for this purpose.
2017-06-20 01:34:18 +02:00
Marek Marczykowski-Górecki
e94bdca206
base: add PropertyHolder.clone_properties
Be compatible with core-admin
2017-06-20 01:34:17 +02:00
Marek Marczykowski-Górecki
12fd4eb035
doc: minor fixes to man pages 2017-06-20 01:34:17 +02:00
Marek Marczykowski-Górecki
998a42703f
storage: add volume clone method
Clone volume without retrieving all the data.

QubesOS/qubes-issues#2622
2017-06-20 01:34:17 +02:00