Commit Graph

2007 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
bfb09f567f
Fix saving 'updates-available' flag reported by TemplateBasedVM
Look for the first updateable template up in the template chain, instead
of going just one level up. Especially this applies to
DispVM -> AppVM -> TemplateVM case.
If DispVM reports available updates, 'updates-available'
flag should be set on relevant TemplateVM, not AppVM (*-dvm).

Include test for the new case.

Fixes QubesOS/qubes-issues#3736
2019-01-19 03:25:21 +01:00
Marek Marczykowski-Górecki
18b0fa9db7
tests: improve mock setup in api_misc
Make self.src.template separate mock instance for better control over
results.
2019-01-19 03:25:20 +01:00
Marek Marczykowski-Górecki
a9ec2bb2c3
vm/qubesvm: fix race condition in failed startup handling
Instead of checking if domain is still running/paused, try to kill it
anyway and ignore appropriate exception. Otherwise domain could die
before the check and killing.
2019-01-19 03:25:20 +01:00
Marek Marczykowski-Górecki
4e5a14774a
storage/lvm: fix listing volume revisions when VM name include volume name
some-vm-root is a valid VM name, and in that case it's volume can be
named some-vm-root-private. Do not let it confuse revision listing,
check for unexpected '-' in volume revision number.

The proper solution would be to use different separator, that is not
allowed in VM names. But that would require migration code that is
undesired in the middle of stable release life cycle.

Fixes QubesOS/qubes-issues#4680
2019-01-19 03:25:19 +01:00
Marek Marczykowski-Górecki
dbd85c75e2
Merge branch 'tests-20181223'
* tests-20181223:
  tests: drop expectedFailure from qubes_desktop_run test
  tests: grub in HVM qubes
  tests: update dom0_update for new updates available flag
  tests: regression test LVM listing code
  tests/extra: wrap ProcessWrapper.wait() to be asyncio-aware
  tests: adjust backupcompat for new maxmem handling
2019-01-19 03:24:29 +01:00
Marek Marczykowski-Górecki
ed6acf4a33
Merge remote-tracking branch 'origin/pr/249'
* origin/pr/249:
  qubes/utils: parse_size: Fix string comparison
2019-01-17 22:28:36 +01:00
M. Vefa Bicakci
d934e8114a
qubes/utils: parse_size: Fix string comparison
This commit resolves a bug which causes strings such as "350MiB" to be
rejected by parse_size, due to the fact that parse_size changes the case
of letters in the input string ("350MiB") to uppercase ("350MIB"), but
fails to do the same for the elements of the units conversion table.

The correction is simple: Apply the same case change to the units
table elements before comparison.
2019-01-16 21:18:50 -05:00
Marek Marczykowski-Górecki
186702646e
tests: drop expectedFailure from qubes_desktop_run test 2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
08621e7225
tests: grub in HVM qubes
Test grub in both PV and HVM. This change will also ease adding grub
test for PVH, when supported.
2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
adc3c4daa4
tests: update dom0_update for new updates available flag
Pending updates for dom0 are now signaled the same way as for templates
- with 'updates-available' feature, not a flag file.
2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
4c4e008368
tests: regression test LVM listing code
The revisions listing code mistakes VM name suffix with volume name.

QubesOS/qubes-issues#4680
2019-01-06 04:33:16 +01:00
Marek Marczykowski-Górecki
b08804e7c8
tests/extra: wrap ProcessWrapper.wait() to be asyncio-aware
The user of ExtraTestCase don't need to know anything about asyncio.
vm.run().wait() normally is a coroutine, but provide a wrapper that
handle asyncio.

This fixes FD leak in input proxy tests.
2019-01-06 04:33:16 +01:00
Marek Marczykowski-Górecki
0099aa1037
tests: adjust backupcompat for new maxmem handling
Since 4dc86310 "Use maxmem=0 to disable qmemman, add more automation to
it" meminfo-writer service is not accessible directly. maxmem property
is used to encode memory management instead.
2019-01-06 04:33:16 +01:00
Wojtek Porczyk
17704c8d0a
qubes/ext/gui: set videoram qdb entries
This is for adjusting videoram size. See
QubesOS/qubes-gui-agent-linux@d222e16003
2018-12-20 18:28:40 +01:00
Wojtek Porczyk
ff612a870b qubes/features: check_with_(template_and_)adminvm
- Two new methods: .features.check_with_adminvm() and
  .check_with_template_and_adminvm(). Common code refactored.

- Two new AdminAPI calls to take advantage of the methods:
  - admin.vm.feature.CheckWithAdminVM
  - admin.vm.feature.CheckWithTemplateAndAdminVM

- Features manager moved to separate module in anticipation of features
  on app object in R5.0. The attribute Features.vm renamed to
  Features.subject.

- Documentation, tests.
2018-12-20 18:28:33 +01:00
Marek Marczykowski-Górecki
3728230e3c
Merge branch 'maxmem' 2018-12-09 18:38:21 +01:00
Marek Marczykowski-Górecki
3606010591
Merge branch 'devel-20181205'
* devel-20181205:
  vm/dispvm: fix /qubes-vm-presistence qubesdb entry
  vm/mix/net: prevent setting provides_network=false if qube is still used
  tests: updates-available notification
  tests/network: reduce code duplication
  tests: listen on 'misc' socket too
2018-12-09 18:35:35 +01:00
Marek Marczykowski-Górecki
f66aae220f
vm/dispvm: fix /qubes-vm-presistence qubesdb entry
It should be 'none' for DispVM.

Fixes QubesOS/qubes-issues#4007
2018-12-08 23:13:30 +01:00
AJ Jordan
d4e567cb10
Fix typo 2018-12-06 20:43:39 -05:00
Marek Marczykowski-Górecki
7a6e4b284f
vm/mix/net: prevent setting provides_network=false if qube is still used
Add symmetric check to the one in 'netvm' property.

Fixes QubesOS/qubes-issues#4552
2018-12-06 20:37:50 +01:00
Marek Marczykowski-Górecki
9f6c97596d
tests: updates-available notification
First install test-pkg-1.0, then add test-pkg-1.1 to repo and check if
updates-available flag is set. Then install updates and check if the
flag is cleared.

QubesOS/qubes-issues#2009
2018-12-05 06:19:34 +01:00
Marek Marczykowski-Górecki
cea9f66bc3
tests/network: reduce code duplication 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
2edafacbe4
tests: listen on 'misc' socket too 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
ccd7e285a1
Remove unused core2migration.py
It is part of backup-restore code, but it was moved to
core-admin-client repo.
2018-12-04 16:12:33 +01:00
Marek Marczykowski-Górecki
162196a1f8
tests: fix lookup domain by name 2018-12-04 16:12:09 +01:00
Marek Marczykowski-Górecki
91c9e49612
Drop unnecessary 'pass' to make pylint happy 2018-12-04 16:06:33 +01:00
Marek Marczykowski-Górecki
eee9964b11
Merge remote-tracking branch 'origin/release4.0' 2018-12-04 16:03:33 +01:00
Marek Marczykowski-Górecki
7d1bcaf64c Introduce management_dispvm property
The new property is meant for management stack (Salt) to set which DVM
template should be used to maintain given VM. Since the DispVM based on
it will be given ultimate control over target VM (qubes.VMShell
service), it should be trusted. The one pointed to by default_dispvm
not necessary is one.

The property defaults to the value from the template (if any), and then
to a global management_dispvm property. By default it is set to None.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-12-03 19:18:26 +01:00
Marek Marczykowski-Górecki
d1e20e5178
Merge remote-tracking branch 'origin/pr/242'
* origin/pr/242:
  Drop async future check with python3 version
  tracis-ci: add support for fc29 dom0
2018-11-24 14:43:43 +01:00
fepitre
313a22d4d6 Drop async future check with python3 version
QubesOS/qubes-issues#4225
2018-11-22 19:34:58 +01:00
Marek Marczykowski-Górecki
087a02c7f4
ext/services: add automatic migration meminfo-writer=False -> maxmem=0
Migrate meminfo-writer=False service setting to maxmem=0 as a method to
disable dynamic memory management. Remove the service from vm.features
dict in the process.

Additionally, translate any attempt to set the service.meminfo-writer
feature to either setting maxmem=0 or resetting it to the default (which
is memory balancing enabled if supported by given domain). This is to at
least partially not break existing tools using service.meminfo-writer as
a way to control dynamic memory management. This code does _not_ support
reading service.meminfo-writer feature state to get the current state of
dynamic memory management, as it would require synchronizing with all
the factors affecting its value. One of main reasons for migrating to
maxmem=0 approach is to avoid the need of such synchronization.

QubesOS/qubes-issues#4480
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
62bc462a23
tests: default maxmem 2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
b8052f864a
tests: more cases for libvirt xml generation
Related to automatic mem balance enabling/disabling. Check how it behave
in presence of PCI devices, or explicit disabling it.
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
4dc8631010
Use maxmem=0 to disable qmemman, add more automation to it
Use maxmem=0 for disabling dynamic memory balance, instead of cryptic
service.meminfo-writer feature. Under the hood, meminfo-writer service
is also set based on maxmem property (directly in qubesdb, not
vm.features dict).
Having this as a property (not "feature"), allow to have sensible
handling of default value. Specifically, disable it automatically if
otherwise it would crash a VM. This is the case for:
 - domain with PCI devices (PoD is not supported by Xen then)
 - domain without balloon driver and/or meminfo-writer service

The check for the latter is heuristic (assume presence of 'qrexec' also
can indicate balloon driver support), but it is true for currently
supported systems.

This also allows more reliable control of libvirt config: do not set
memory != maxmem, unless qmemman is enabled.

memory != maxmem only makes sense if qmemman for given domain is
enabled.  Besides wasting some domain resources for extra page tables
etc, for HVM domains this is harmful, because maxmem-memory difference
is made of Popupate-on-Demand pool, which - when depleted - will kill
the domain. This means domain without balloon driver will die as soon
as will try to use more than initial memory - but without balloon driver
it sees maxmem memory and doesn't know about the lower limit.

Fixes QubesOS/qubes-issues#4135
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
35a53840f1
vm: send domain-start-failed event also if some device is missing
Checking device presence wasn't covered with try/except that send the
event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
0eab082d85
ext/core-features: make 'template-postinstall' event async
It makes a lot of sense to call long-running operations in that event
handler, including calling back into the VM. Allow that by using
fire_event_async, not just fire_event.

Also, document the event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
d2585aa871
tests/lvm: fix checking lvm pool existence cont.
Commit 15cf593bc5 "tests/lvm: fix checking
lvm pool existence" attempted to fix handling '-' in pool name by using
/dev/VG/LV symlink. But those are not created for thin pools. Change
back to /dev/mapper, but include '-' mangling.

Related QubesOS/qubes-issues#4332
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
f023b3dd6e
backup: fix naming qubes.xml.000 in the archive
Restore old code for calculating subdir within the archive. The new one
had two problems:
 - set '/' for empty input subdir - which caused qubes.xml.000 to be
 named '/qubes.xml.000' (and then converted to '../../qubes.xml.000');
 among other things, this results in the wrong path used for encryption
 passphrase
 - resolved symlinks, which breaks calculating path for any symlinks
 within VM's directory (symlinks there should be treated as normal files
 to be sure that actual content is included in the backup)

This partially reverts 4e49b951ce.

Fixes QubesOS/qubes-issues#4493
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
328697730b
vm: fix deadlock on qrexec timeout handling
vm.kill() will try to get vm.startup_lock, so it can't be called while
holding it already.
Fix this by extracting vm._kill_locked(), which expect the lock to be
already taken by the caller.
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
68dffb6895
api/admin: fix error message when refusing to create template on template
Fixes QubesOS/qubes-issues#4463
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
64f290c9ba
ext/pci: fix error message about missing device
Print human readable device name, instead of "<PCIDevice at ...".

QubesOS/qubes-issues#4461
2018-11-01 22:28:50 +01:00
Marek Marczykowski-Górecki
00ca0459d9
ext/pci: use correct backend domain for getting PCIDevice instance
In practice backend_domain is ignored (all PCI devices belongs to dom0),
but lets fix this anyway.
2018-11-01 22:21:50 +01:00
Marek Marczykowski-Górecki
15cf593bc5
tests/lvm: fix checking lvm pool existence
If pool or group name have '-', it will be mangled as '--' in
/dev/mapper. Use /dev/VG_NAME/LV_NAME symlink instead.

Related QubesOS/qubes-issues#4332
2018-10-30 01:17:00 +01:00
Marek Marczykowski-Górecki
1ae6abdff5
exc: fix QubesMemoryError constructor
QubesVMError require 'vm' argument.
Fixes 2f3a9847 "exc: Make QubesMemoryError inherit from QubesVMError"
2018-10-30 01:14:58 +01:00
Marek Marczykowski-Górecki
b9a18a819c
Merge remote-tracking branch 'origin/pr/239'
* origin/pr/239:
  storage: fix NotImplementedError message for import_data()
  storage/reflink: make resize()/import_volume() more readable
  storage/reflink: unblock import_data() and import_data_end()
2018-10-29 23:00:04 +01:00
Marek Marczykowski-Górecki
fa2429aae4
Merge remote-tracking branch 'origin/pr/237'
* origin/pr/237:
  progress thresold removed as Marek suggested
  Avoid progress events flooding

Fixes QubesOS/qubes-issues#4406
Fixes QubesOS/qubes-issues#3035
2018-10-29 22:54:05 +01:00
Rusty Bird
7e4812a525
storage: fix NotImplementedError message for import_data() 2018-10-29 20:21:42 +00:00
Rusty Bird
73db2751b8
storage/reflink: make resize()/import_volume() more readable 2018-10-29 20:21:41 +00:00
Rusty Bird
425d993769
storage/reflink: unblock import_data() and import_data_end() 2018-10-29 20:21:39 +00:00
Marek Marczykowski-Górecki
f621e8792c
Merge branch 'master' into devel-no-assert 2018-10-29 20:29:53 +01:00
Marek Marczykowski-Górecki
3740e2d48b
api: make enforce() static
It doesn't use 'self'. And pylint complains.
2018-10-29 20:22:35 +01:00
Marek Marczykowski-Górecki
db6094f397
tests/api: adjust for proper exceptions instead of AssertionError 2018-10-29 20:22:10 +01:00
Marek Marczykowski-Górecki
2b5fc6299e
tests/api: do not test non-existing methods
Remove methods not included in specification (or with different
constraints). Keep commented out methods included in spec but not
implemented.
2018-10-29 20:21:36 +01:00
Marek Marczykowski-Górecki
c1c6dc2acd
Use ValueError in PropertyHolder.property_require()
Specifically do not use AssertionError, but also be consistent with
other value verification.
2018-10-29 20:16:41 +01:00
Marek Marczykowski-Górecki
26a553737f
storage/lvm: minor fix for lvs command building
Do not prepend 'sudo' each time - do a copy of array if that's
necessary.
2018-10-29 05:16:23 +01:00
Marek Marczykowski-Górecki
42061cb194
tests: try to collect qvm-open-in-dvm output if no editor window is shown
Try to collect more details about why the test failed. This will help
only if qvm-open-in-dvm exist early. On the other hand, if it hang, or
remote side fails to find the right editor (which results in GUI error
message), this change will not provide any more details.
2018-10-29 01:20:57 +01:00
Marek Marczykowski-Górecki
84c321b923
tests: increase session startup timeout for whonix-ws based VMs
First boot of whonix-ws based VM take extended period of time, because
a lot of files needs to be copied to private volume. This takes even
more time, when verbose logging through console is enabled. Extend the
timeout for that.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
84d3547f09
tests: adjust extra tests loader to work with nose2
Nose loader do not provide loader.loadTestsFromTestCase(), use
loader.loadTestsFromNames() instead.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
fb14f589cb
tests: wait for full user session before doing rest of the test
Clean VM shutdown may timeout if its initiated before full startup, so
make sure the full startup is completed first.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
4742a630f2
tests: use iptables --wait
QubesOS/qubes-issues#3665 affects also tests...
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
2f3a984742
exc: Make QubesMemoryError inherit from QubesVMError
Same as other vm-related errors.
This helps QubesTestCase.cleanup_traceback() cleanup VM reference.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
8be70c9e4d
ext/services: allow for os=Linux feature request from VM
It's weird to set it for Windows, but not Linux.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
e244c192ae
tests: use /bin/uname instead of /bin/hostname as dummy output generator
Use something included in coreutils installed everywhere.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
f13029219b
vm: disable/enable qubes-vm@ service when domain is removed/created
If domain is set to autostart, qubes-vm@ systemd service is used to
start it at boot. Cleanup the service when domain is removed, and
similarly enable the service when domain is created and already have
autostart=True.

Fixes QubesOS/qubes-issues#4014
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
0b7aa546c6
tests: remove VM reference from QubesVMError
Yet another place wheren object references are leaked.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
a972c61914
tests: use socat instead of nc
socat have only one variant, so one command line syntax to handle. It's
also installed by default in Qubes VMs.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
08ddeee9fb
tests: improve VMs cleanup wrt custom templates
Cleanup VMs in template reverse topological order, not network one.
Network can be set to None to break dependency, but template can't. For
netvm to be changed, kill VMs first (kill doesn't check network
dependency), so netvm change will not trigger side effects (runtime
change, which could fail).

This fixes cleanup for tests creating custom templates - previously
order was undefined and if template was tried removed before its child
VMs, it fails. All the relevant files were removed later anyway, but it
lead to python objects leaks.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
4e762788a9
tests: check if qubes-vm@ service is disabled on domain removal
Test for QubesOS/qubes-issues#4014
2018-10-27 01:43:00 +02:00
Marek Marczykowski-Górecki
cf8b6219a9
tests: make use of vm.shutdown(wait=True) 2018-10-27 01:43:00 +02:00
Marek Marczykowski-Górecki
2c1629da04
vm: call after-shutdown cleanup also from vm.kill and vm.shutdown
Cleaning up after domain shutdown (domain-stopped and domain-shutdown
events) relies on libvirt events which may be unreliable in some cases
(events may be processed with some delay, of if libvirt was restarted in
the meantime, may not happen at all). So, instead of ensuring only
proper ordering between shutdown cleanup and next startup, also trigger
the cleanup when we know for sure domain isn't running:
 - at vm.kill() - after libvirt confirms domain was destroyed
 - at vm.shutdown(wait=True) - after successful shutdown
 - at vm.remove_from_disk() - after ensuring it isn't running but just
 before actually removing it

This fixes various race conditions:
 - qvm-kill && qvm-remove: remove could happen before shutdown cleanup
 was done and storage driver would be confused about that
 - qvm-shutdown --wait && qvm-clone: clone could happen before new content was
 commited to the original volume, making the copy of previous VM state
(and probably more)

Previously it wasn't such a big issue on default configuration, because
LVM driver was fully synchronous, effectively blocking the whole qubesd
for the time the cleanup happened.

To avoid code duplication, factor out _ensure_shutdown_handled function
calling actual cleanup (and possibly canceling one called with libvirt
event). Note that now, "Duplicated stopped event from libvirt received!"
warning may happen in normal circumstances, not only because of some
bug.

It is very important that post-shutdown cleanup happen when domain is
not running. To ensure that, take startup_lock and under it 1) ensure
its halted and only then 2) execute the cleanup. This isn't necessary
when removing it from disk, because its already removed from the
collection at that time, which also avoids other calls to it (see also
"vm/dispvm: fix DispVM cleanup" commit).
Actually, taking the startup_lock in remove_from_disk function would
cause a deadlock in DispVM auto cleanup code:
 - vm.kill (or other trigger for the cleanup)
   - vm.startup_lock acquire   <====
     - vm._ensure_shutdown_handled
       - domain-shutdown event
         - vm._auto_cleanup (in DispVM class)
           - vm.remove_from_disk
             - cannot take vm.startup_lock again
2018-10-26 23:54:08 +02:00
Marek Marczykowski-Górecki
5be003d539
vm/dispvm: fix DispVM cleanup
First unregister the domain from collection, and only then call
remove_from_disk(). Removing it from collection prevent further calls
being made to it. Or if anything else keep a reference to it (for
example as a netvm), then abort the operation.

Additionally this makes it unnecessary to take startup lock when
cleaning it up in tests.
2018-10-26 23:54:08 +02:00
Marek Marczykowski-Górecki
e1f65bdf7b
vm: add shutdown_timeout property, make vm.shutdown(wait=True) use it
vm.shutdown(wait=True) waited indefinitely for the shutdown, which makes
useless without some boilerplate handling the timeout. Since the timeout
may depend on the operating system inside, add a per-VM property for it,
with value inheritance from template and then from global
default_shutdown_timeout property.

When timeout is reached, the method raises exception - whether to kill
it or not is left to the caller.

Fixes QubesOS/qubes-issues#1696
2018-10-26 23:54:04 +02:00
Marek Marczykowski-Górecki
b65fdf9700
storage: convert lvm driver to async version
LVM operations can take significant amount of time. This is especially
visible when stopping a VM (`vm.storage.stop()`) - in that time the
whole qubesd freeze for about 2 seconds.

Fix this by making all the ThinVolume methods a coroutines (where
supported). Each public coroutine is also wrapped with locking on
volume._lock to avoid concurrency-related problems.
This all also require changing internal helper functions to
coroutines. There are two functions that still needs to be called from
non-coroutine call sites:
 - init_cache/reset_cache (initial cache fill, ThinPool.setup())
 - qubes_lvm (ThinVolume.export()

So, those two functions need to live in two variants. Extract its common
code to separate functions to reduce code duplications.

Fixes QubesOS/qubes-issues#4283
2018-10-23 16:53:35 +02:00
Marek Marczykowski-Górecki
299c514647
tests: fix asyncio usage in storage_lvm.TC_01_ThinPool
Both vm.create_on_disk() and vm.start() are coroutines. Tests in this
class didn't run them, so basically didn't test anything.

Wrap couroutine calls with self.loop.run_until_complete().

Additionally, don't fail if LVM pool is named differently.
In that case, the test is rather sily, as it probably use the same pool
for source and destination (operation already tested elsewhere). But it
isn't a reason for failing the test.
2018-10-23 16:53:35 +02:00
Marek Marczykowski-Górecki
6170edb291
storage: allow import_data and import_data_end be coroutines
On some storage pools this operation can also be time consuming - for
example require creating temporary volume, and volume.create() already
can be a coroutine.
This is also requirement for making common code used by start()/create()
etc be a coroutine, otherwise neither of them can be and will block
other operations.

Related to QubesOS/qubes-issues#4283
2018-10-23 16:53:35 +02:00
Marek Marczykowski-Górecki
d1f5cb5d15
ext/services: mechanism for advertising supported services
Support 'supported-service.*' features requests coming from VMs. Set
such features directly (allow only value '1') and remove any not
reported in given call. This way uninstalling package providing given
service will automatically remove related 'supported-service...'
feature.

Fixes QubesOS/qubes-issues#4402
2018-10-23 16:47:39 +02:00
donoban
a2db90f85f
progress thresold removed as Marek suggested 2018-10-22 22:12:46 +02:00
donoban
61c6288391
Avoid progress events flooding 2018-10-22 21:14:43 +02:00
Marek Marczykowski-Górecki
58bcec2a64
qubesvm: improve error message about same-pool requirement
Make it clear that volume creation fails because it needs to be in the
same pool as its parent. This message is shown in context of `qvm-create
-p root=MyPool` for example and the previous message didn't make sense
at all.

Fixes QubesOS/qubes-issues#3438
2018-10-18 00:03:05 +02:00
Marek Marczykowski-Górecki
ba210c41ee
qubesvm: don't crash VM creation if icon symlink already exists
It can be leftover from previous failed attempt. Don't crash on it, and
replace it instead.

QubesOS/qubes-issues#3438
2018-10-18 00:01:45 +02:00
Marek Marczykowski-Górecki
c01ae06fee
tests: add basic ServicesExtension tests 2018-10-17 17:37:02 +02:00
Marek Marczykowski-Górecki
133219f6d3
Do not generate R3 compat firewall rules if R4 format is supported
R3 format had limitation of ~40 rules per VM. Do not generate compat
rules (possibly hitting that limitation) if new format, free of that
limitation is supported.

Fixes QubesOS/qubes-issues#1570
Fixes QubesOS/qubes-issues#4228
2018-10-15 06:05:05 +02:00
Marek Marczykowski-Górecki
e8dc6cb916
tests: use smaller root.img in backupcompatibility tests
1GB image easily exceed available space on openQA instances. Use 100MB
instead.
2018-10-15 05:24:24 +02:00
Marek Marczykowski-Górecki
fd9f2e2a6c
tests: type commands into specific found window
Make sure events are sent to specific window found with xdotool search,
not the one having the focus. In case of Whonix, it can be first
connection wizard or whonixcheck report.
2018-10-15 05:18:42 +02:00
Marek Marczykowski-Górecki
9e81087b25
tests: use improved wait_for_window in various tests
Replace manual `xdotool search calls` with wait_for_window(), where
compatible.
2018-10-15 05:16:29 +02:00
Marek Marczykowski-Górecki
f1621c01e9
tests: add search based on window class to wait_for_window
Searching based on class is used in many tests, searching by class, not
only by name in wait_for_window will allow to reduce code duplication.
While at it, improve it additionally:
 - avoid active waiting for window and use `xdotool search --sync` instead
 - return found window id
 - add wait_for_window_coro() for use where coroutine is needed
 - when waiting for window to disappear, check window id once and wait
   for that particular window to disappear (avoid xdotool race
   conditions on window enumeration)

Besides reducing code duplication, this also move various xdotool
imperfections handling into one place.
2018-10-15 05:08:25 +02:00
Marek Marczykowski-Górecki
3f5618dbb0
tests/integ/network: make the tests independend of default netvm
Network tests create own temporary netvm. Make it disconnected from the
real netvm, to not interefere in tests.
2018-10-15 00:39:32 +02:00
Marek Marczykowski-Górecki
15140255d5
tests/integ/network: few more code style improvement
Remove unused imports and unused variables, add some more docstrings.
2018-10-15 00:38:38 +02:00
Marek Marczykowski-Górecki
375688837c
tests/integ/network: add type annotations
Make PyCharm understand what mixin those objects are for.
2018-10-15 00:30:17 +02:00
Marek Marczykowski-Górecki
3b6703f2bd
tests: fix race condition in gui_memory_pinning test
Don't rely on top update timing, pause it updates for taking
screenshots.
2018-10-14 05:48:25 +02:00
Marek Marczykowski-Górecki
9887b925b4
tests: increase timeout for vm shutdown
start_standalone_with_cdrom_vm test essence is somewhere else, let not
fail it for just slow shutdown (LVM cleanup etc).
2018-10-14 05:29:05 +02:00
Marek Marczykowski-Górecki
29a26e7d69
tests: make timeout it shutdown test even longer
Reduce false positives when testing on busy machine.
2018-10-14 03:32:17 +02:00
Marek Marczykowski-Górecki
3e28ccefde
tests: fix cleanup after backup compatibility tests
Allow removing VMs based on multiple prefixes at once. Removing them
separately doesn't handle all the dependencies (default_netvm, netvm)
correctly. This is needed for backup compatibility tests, where VMs are
created with `test-` prefix and `disp-tests-`. Additionally backup code
will create `disp-no-netvm`, which also may need to be removed.
2018-10-14 03:29:30 +02:00
Marek Marczykowski-Górecki
00c0b4c69f
tests: cleanup tracebacks also for expectedFailure exception
Continuation of 5bc0baea "tests: do not leak objects in object leaks
checking function".
2018-10-12 14:41:35 +02:00
Marek Marczykowski-Górecki
d23636fa02
tests: migrate qvm-block tests to core3 2018-10-10 00:44:15 +02:00
Marek Marczykowski-Górecki
1245606453
tests: fix cleanup of dom0_update tests
Reset updatevm to None before removing VMs, otherwise removing updatevm
will fail.
2018-10-10 00:08:57 +02:00
Marek Marczykowski-Górecki
8dab298b89
tests: create testcases on module import if environment variable is set
If QUBES_TEST_TEMPLATES or QUBES_TEST_LOAD_ALL is set, create testcases
on modules import, instead of waiting until `load_tests` is called.
The `QUBES_TEST_TEMPLATES` doesn't require `qubes.xml` access, so it
should be safe to do regardless of the environment. The
`QUBES_TEST_LOAD_ALL` force loading tests (and reading `qubes.xml`)
regardless.

This is useful for test runners not supporting load_tests protocol. Or
with limited support - for example both default `unittest` runner and
`nose2` can either use load_tests protocol _or_ select individual tests.
Setting any of those variable allow to run a single test with those
runners.

With this feature used together load_tests protocol, tests could be
registered twice. Avoid this by not listing already defined test classes
in create_testcases_for_templates (according to load_tests protocol,
those should already be registered).
2018-10-10 00:08:29 +02:00
Marek Marczykowski-Górecki
c8929cfee9
tests: improve handling backups in core3 2018-10-07 19:51:55 +02:00
Marek Marczykowski-Górecki
7a607e3731
tests: add QUBES_TEST_TEMPLATES env variable
Allow easily list templates to be tested, without enumerating all the
test classes. This is especially useful with nose2 runner which can't
use load tests protocol _and_ select subset of tests.
2018-10-07 19:51:55 +02:00
Marek Marczykowski-Górecki
35c66987ab
tests: improve clearing tracebacks from Qubes* objects
Clear also tracebacks of chained exceptions.
2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
7c91e82365
tests: handle KWrite editor in DispVM tests 2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
02f9661169
tests: migrate mime handlers test to core3 2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
5bc0baeafa
tests: do not leak objects in object leaks checking function
If any object is leaked, QubesTestCase.cleanup_gc() raises an exception,
which have leaked objects list referenced in its traceback. This happens
after cleanup_traceback(), so isn't cleaned, causing cleanup_gc() fail
for all the further tests in the same test run.

Avoid this, by dropping list just before checking if any object is
leaked.
2018-09-29 02:40:43 +02:00
Marek Marczykowski-Górecki
8b75b264ae
Merge remote-tracking branch 'origin/pr/235'
* origin/pr/235:
  Add default_qrexec_timeout to qubes-prefs
2018-09-19 06:03:10 +02:00
Marek Marczykowski-Górecki
b2387389d0
Update documentation for device-attach event 2018-09-19 05:44:02 +02:00
Rusty Bird
bee69a98b9
Add default_qrexec_timeout to qubes-prefs
When a VM (or its template) does not explicitly set a qrexec_timeout,
fall back to a global default_qrexec_timeout (with default value 60),
instead of hardcoding the fallback value to 60.

This makes it easy to set a higher timeout for the whole system, which
helps users who habitually launch applications from several (not yet
started) VMs at the same time. 60 seconds can be too short for that.
2018-09-16 18:42:48 +00:00
Marek Marczykowski-Górecki
e26655bc82
tests: fix time sync test
qvm-sync-clock no longer fetches time from the network, by design.
So, lets not break clockvm's time and check only if everything else
correctly synchronize with it.
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
c4a84b3298
tests: wait for DispVM's qubes.VMShell exit
It isn't enough to wait for window to disappear, the service may still
be running. And if it is, test cleanup logic will complain about FD
leak.
To avoid deadlock on some test failure, do it with a timeout.
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
240b1dd75e
tests: exclude whonixcheck and NetworkManager from editor window search
Those may pop up before actual editor is found, which fails the test as
it can't handle such "editor".
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
ac8b8a3ad4
tests: reenable some qrexec tests, convert them to py3k/asyncio 2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
576bcb158e
tests: skip tests not relevant on Whonix 2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
556a08cb78
tests: improve shutdown timeout handling
Instead of waiting 1sec, wait up to 5sec but skip when vm is shut off.
This fix tests on slow machines, including openQA nested virt.
2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
5a1bf11d0d
tests: drop qvm-prefs tests
Those are moved to qubes-core-admin-client repository.
2018-09-15 05:12:41 +02:00
Rusty Bird
5756e870bd
storage/reflink: use context managers in is_supported()
Don't rely on garbage collection to close and remove the tempfiles.
2018-09-13 19:46:48 +00:00
Rusty Bird
867baf47d1
api/admin: fix typo 2018-09-13 19:46:46 +00:00
Rusty Bird
b3983f5ef8
'except FileNotFoundError' instead of ENOENT check 2018-09-13 19:46:45 +00:00
AJ Jordan
5aa35a1208
Make log location more explicit in error message
See https://github.com/QubesOS/qubes-issues/issues/4224#issuecomment-414513721.
2018-09-12 22:40:21 -04:00
Rusty Bird
cf1ea5cee1
tests/app: test varlibqubes pool driver selection 2018-09-11 23:50:33 +00:00
Rusty Bird
797bbc43a0
tests/storage_reflink: test some file-reflink helpers
Tested:

    - _copy_file()
    - _create_sparse_file()
    - _resize_file()
    - _update_loopdev_sizes()

Smoke tested by calls from the functions above:

    - _replace_file()
    - _rename_file()
    - _make_dir()
    - _fsync_dir()
2018-09-11 23:50:32 +00:00
Rusty Bird
b82e739346
tests/integ/storage: add file-reflink integration tests 2018-09-11 23:50:31 +00:00
Rusty Bird
8c117549ad
tests/integ/basic: use export() in get_rootimg_checksum()
volume.path and volume.export() refer to the same thing in lvm_thin and
'file', but not in file-reflink (where volume.path is the -dirty.img,
which doesn't exist if the volume is not started).
2018-09-11 23:50:30 +00:00
Rusty Bird
49e7ce025f
tests/integ/backupcompatibility: Storage.verify() is a coro 2018-09-11 23:50:28 +00:00
Rusty Bird
8d1913a8cc
app: create /var/lib/qubes as file-reflink if supported
Use the file-reflink storage driver if /var/lib/qubes is on a filesystem
that supports reflinks, e.g. when the btrfs layout was selected in
Anaconda. If it doesn't support reflinks (or if detection fails, e.g. in
an unprivileged test environment), use 'file' as before.
2018-09-11 23:50:26 +00:00
Rusty Bird
53ef5ed431
app: uncouple pool setup from loading initial configuration
And ensure that setup is called on every type of these pools, not just
lvm_thin.
2018-09-11 23:50:25 +00:00
Rusty Bird
8eb9c64f20
tools/qubes-create: fix docstring 2018-09-11 23:50:24 +00:00
Rusty Bird
d181bf1aa4
storage: factor out _wait_and_reraise(); fix clone/create
_wait_and_reraise() is similar to asyncio.gather(), but it preserves the
current behavior of waiting for all futures and only _then_ reraising
the first exception (if there is any) in line.

Also switch Storage.create() and Storage.clone() to _wait_and_reraise().
Previously, they called asyncio.wait() and implicitly swallowed all
exceptions.
2018-09-11 23:50:22 +00:00
Rusty Bird
d33bd3f2b6
storage: fix search_pool_containing_dir()
Canonicalize both directories, resolving symlink components. Compare
with commonpath() instead of startswith(), because /foo doesn't contain
/foobar.
2018-09-11 23:50:21 +00:00
Rusty Bird
f0ee73e63f
storage: remove broken default parameter from isodate()
With that syntax, the default timestamp would have been from the time of
the function's definition (not invocation). But all callers are passing
an explicit timestamp anyway.
2018-09-11 23:50:20 +00:00
Rusty Bird
44ca78523f
storage: insert missing NotImplementedError in Volume.stop() 2018-09-11 23:50:19 +00:00
Rusty Bird
266d90c2f9
storage: fix docstrings 2018-09-11 23:50:18 +00:00
Rusty Bird
c75fe09814
storage/reflink: is_reflink_supported() -> is_supported() 2018-09-11 23:50:16 +00:00
Rusty Bird
1889c9b75f
storage/reflink: run synchronous volume methods in executor
Convert create(), verify(), remove(), start(), stop(), revert(),
resize(), and import_volume() into coroutine methods, via a decorator
that runs them in the event loop's thread-based default executor.

This reduces UI hangs by unblocking the event loop, and can e.g. speed
up VM starts by starting multiple volumes in parallel.
2018-09-11 23:50:15 +00:00
Rusty Bird
3d986be02a
storage/reflink: native FICLONE in _copy_file() happy path
Avoid a subprocess launch, and distinguish reflink vs. fallback copy in
the log.
2018-09-11 23:50:14 +00:00
Rusty Bird
edda3a1734
storage/reflink: factor out _ficlone() 2018-09-11 23:50:13 +00:00
Rusty Bird
69af0a48ec
storage/reflink: inline and simplify _cmd() 2018-09-11 23:50:12 +00:00
Rusty Bird
fb06a8089a
storage/reflink: _update_loopdev_sizes() without losetup
Factor out a function, and use the LOOP_SET_CAPACITY ioctl instead of
going through losetup.
2018-09-11 23:50:10 +00:00
Rusty Bird
385ba91772
storage/reflink: resize(): don't look for loopdevs if clean 2018-09-09 20:01:22 +00:00
Rusty Bird
e7b7c253ac
storage/reflink: inline _require_self_on_stop() 2018-09-09 20:01:20 +00:00
Rusty Bird
6e8d7d4201
storage/reflink: no-op import_volume() if not save_on_stop
Instead of raising a NotImplementedError, just return self like 'file'
and lvm_thin. This is needed when Storage.clone() is modified in another
commit* to no longer swallow exceptions.

* "storage: factor out _wait_and_reraise(); fix clone/create"
2018-09-09 20:01:19 +00:00
Rusty Bird
60bf68a748
storage/reflink: add _path_import (don't reuse _path_dirty)
Import volume data to a new _path_import (instead of _path_dirty) before
committing to _path_clean. In case the computer crashes while an import
operation is running, the partially written file should not be attached
to Xen on the next volume startup.

Use <name>-import.img as the filename like 'file' does, to be compatible
with qubes.tests.api_admin/TC_00_VMs/test_510_vm_volume_import.
2018-09-09 20:01:18 +00:00
Rusty Bird
d301aa2e50
storage/reflink: delete stale tempfiles on start and remove
When the AT_REPLACE flag for linkat() finally lands in the Linux kernel,
_replace_file() can be modified to use unnamed (O_TMPFILE) tempfiles.
Until then, make sure stale tempfiles from previous crashes can't hang
around for too long.
2018-09-09 20:01:17 +00:00
Rusty Bird
75a4a1340e
storage/reflink: don't recompute static properties per call 2018-09-09 20:01:15 +00:00
Rusty Bird
ef2698adb4
storage/reflink: make revisions() more readable, use iglob 2018-09-09 20:01:14 +00:00
Rusty Bird
18f9356c2c
storage/reflink: refuse to revert() dirty volume 2018-09-09 20:01:13 +00:00
Rusty Bird
677183d8a6
storage/reflink: add revision even if empty
It's sort of useful to be able to revert a volume that has only ever
been started once to its empty state. And the lvm_thin driver allows it
too, so why not.
2018-09-09 20:01:12 +00:00
Rusty Bird
850778b52a
storage/reflink: remove redundant format specifiers 2018-09-09 20:01:11 +00:00
Marek Marczykowski-Górecki
b2cc605f4b
tests: clean local variables from traceback objects
System tests are fragile for any object leaks, especially those holding
open files. Instead of wrapping all tests with try/finally removing
those local variables (as done in qubes.tests.integ.backup for example),
apply generic solution: clean all traceback objects from local
variables. Those aren't used to generate text report by either test
runner (qubes.tests.run and nose2). If one wants to break into debugger
and inspect tracebacks interactively, needs to comment out call to
cleanup_traceback.
2018-09-09 02:43:17 +02:00
Marek Marczykowski-Górecki
c102fa3d68
tests: add basic audio play/rec tests
QubesOS/qubes-issues#4204
2018-09-08 04:13:24 +02:00
Marek Marczykowski-Górecki
8ce3433406
tests: drop sudo in tests already running as root
Don't spam already trashed log.
2018-09-07 15:12:05 +02:00
Marek Marczykowski-Górecki
1b99bd9ab3
Merge branch 'tests-dispvm-fix'
* tests-dispvm-fix:
  tests: fix DispVM related tests
2018-09-07 01:57:19 +02:00
Marek Marczykowski-Górecki
7f1e2741ec
Merge remote-tracking branch 'qubesos/pr/228'
* qubesos/pr/228:
  storage/lvm: filter out warning about intended over-provisioning
  tests: fix getting kernel package version inside VM
  tests/extra: add start_guid option to VMWrapper
  vm/qubesvm: fire 'domain-start-failed' event even if fail was early
  vm/qubesvm: check if all required devices are available before start
  storage/lvm: fix reporting lvm command error
  storage/lvm: save pool's revision_to_keep property
2018-09-07 01:06:59 +02:00
Marek Marczykowski-Górecki
ee25f7c7bb
vm: fix error reporting on PVH without kernel set
Fixes QubesOS/qubes-issues#4254
2018-09-03 00:23:05 +02:00
Marek Marczykowski-Górecki
62ac42f06a
Merge remote-tracking branch 'qubesos/pr/220'
* qubesos/pr/220:
  code style fixes
  Add domain-paused/-unpaused events
2018-09-02 05:34:24 +02:00
Marek Marczykowski-Górecki
b72b81fb9e
Merge remote-tracking branch 'qubesos/pr/224'
* qubesos/pr/224:
  Fix error on non ASCII PCI IDs upon qvm-device list

Fixed code style.
2018-09-02 03:36:00 +02:00
Marek Marczykowski-Górecki
57c9b2edf7
code style fixes 2018-09-02 03:27:14 +02:00
Galland
6f04c8d65b
Fix error on non ASCII PCI IDs upon qvm-device list
solves https://github.com/QubesOS/qubes-issues/issues/4229
2018-08-21 03:08:17 +02:00
Jean-Philippe Ouellet
e95ef5f61d
Add domain-paused/-unpaused events
Needed for event-driven domains-tray UI updating and anti-GUI-DoS
usability improvements.

Catches errors from event handlers to protect libvirt, and logs to
main qubesd logger singleton (by default meaning systemd journal).
2018-08-01 05:41:50 -04:00
Marek Marczykowski-Górecki
0e089ca38d
ext/windows: copy private.img on windows TemplateBasedVM creation
This is a workaround for missing private.img initialization in Qubes
Windows Tools.

QubesOS/qubes-issues#3585
2018-07-17 05:25:31 +02:00
Marek Marczykowski-Górecki
e6edbabf94
tests: exclude windows templates from linux tests 2018-07-16 22:02:59 +02:00
Marek Marczykowski-Górecki
e51efcf980
vm: document domain-start-failed event 2018-07-16 22:02:59 +02:00
Marek Marczykowski-Górecki
af2435c0d4
Make some properties default to template's value (if any)
Multiple properties are related to system installed inside the VM, so it
makes sense to have them the same for all the VMs based on the same
template. Modify default value getter to first try get the value from a
template (if any) and only if it fails, fallback to original default
value.
This change is made to those properties:
 - default_user (it was already this way)
 - kernel
 - kernelopts
 - maxmem
 - memory
 - qrexec_timeout
 - vcpus
 - virt_mode

This is especially useful for manually installed templates (like
Windows).

Related to QubesOS/qubes-issues#3585
2018-07-16 22:02:58 +02:00
Marek Marczykowski-Górecki
af7d54d388
Update windows-related feature requests
Handle 'os' feature - if it's Windows, then set rpc-clipboard feature.
Handle 'gui-emulated' feature - request for specifically stubdomain GUI.
With 'gui' feature it is only possible to enable gui-agent based on, or
disable GUI completely.
Handle 'default-user' - verify it for weird characters and set
'default_user' property (if wasn't already set).

QubesOS/qubes-issues#3585
2018-07-16 22:02:58 +02:00
Marek Marczykowski-Górecki
ce87451c73
Merge branch 'lvm-snapshots'
* lvm-snapshots:
  tests: fix handling app.pools iteration
  storage/lvm: add repr(ThinPool) for more meaningful test reports
  tests: adjust for variable volume path
  api/admin: expose volume path in admin.vm.volume.Info
  tests: LVM: import, list_volumes, volatile volume, snapshot volume
  tests: collect all SIGCHLD before cleaning event loop
  storage/lvm: use temporary volume for data import
  tests: ThinVolume.revert()
  tests: LVM volume naming migration, and new naming in general
  storage/lvm: improve handling interrupted commit
2018-07-16 16:40:35 +02:00
Marek Marczykowski-Górecki
be2465c1f9
Fix issues found by pylint 2.0
Resolve:
 - no-else-return
 - useless-object-inheritance
 - useless-return
 - consider-using-set-comprehension
 - consider-using-in
 - logging-not-lazy

Ignore:
 - not-an-iterable - false possitives for asyncio coroutines

Ignore all the above in qubespolicy/__init__.py, as the file will be
moved to separate repository (core-qrexec) - it already has a copy
there, don't desynchronize them.
2018-07-15 23:51:15 +02:00
Marek Marczykowski-Górecki
c688641363
tests: fix DispVM related tests
- fix regex for editor window search - 'disp*' matches 'disk_space.py'
(a dom0 local widget...)
- increase a timeout for automatic DispVM cleanup
2018-07-15 22:29:22 +02:00
Marek Marczykowski-Górecki
f8d17012c3
tests: fix loading tests for templates
Fix #209
2018-07-15 21:57:34 +02:00
Marek Marczykowski-Górecki
69e3018b94
tests: fix handling app.pools iteration 2018-07-15 21:31:48 +02:00
Marek Marczykowski-Górecki
2af1815ab7
storage/lvm: add repr(ThinPool) for more meaningful test reports 2018-07-15 21:30:04 +02:00
Marek Marczykowski-Górecki
e644378f18
tests: adjust for variable volume path
LVM volumes now have variable volume path. Compare strip path before
comparing content's hash in tests.
2018-07-12 16:52:48 +02:00
Marek Marczykowski-Górecki
d211a2771a
api/admin: expose volume path in admin.vm.volume.Info
Since (for LVM at least) path is dynamic now, add information about it
to volume info. This is not very useful outside of dom0, but in dom0 it
can be very useful for various scripts.
This will disclose current volume revision id, but it is already
possible to deduce it from snapshots list.
2018-07-12 16:52:48 +02:00
Marek Marczykowski-Górecki
4282a41fcb
tests: LVM: import, list_volumes, volatile volume, snapshot volume 2018-07-12 16:52:47 +02:00
Marek Marczykowski-Górecki
76c872a43a
tests: collect all SIGCHLD before cleaning event loop
On python 3.6.4 apparently it requires two callbacks runs to cleanup
stale SIGCHLD handlers.
2018-07-12 16:52:47 +02:00
Marek Marczykowski-Górecki
2b80f0c044
storage/lvm: use temporary volume for data import
Do not write directly to main volume, instead create temporary volume
and only commit it to the main one when operation is finished. This
solve multiple problems:
 - import operation can be aborted, without data loss
 - importing new data over existing volume will not leave traces of
previous content - especially when importing smaller volume to bigger
one
 - import operation can be reverted - it create separate revision,
similar to start/stop
 - easier to prevent qube from starting during import operation
 - template still can be used when importing new version

QubesOS/qubes-issues#2256
2018-07-12 16:52:46 +02:00
Marek Marczykowski-Górecki
aea0de35ad
tests: ThinVolume.revert() 2018-07-12 16:52:46 +02:00
Marek Marczykowski-Górecki
8cf9264283
tests: LVM volume naming migration, and new naming in general 2018-07-12 16:52:45 +02:00
Marek Marczykowski-Górecki
11c7b4bb51
storage/lvm: improve handling interrupted commit
First rename volume to backup revision, regardless of revisions_to_keep,
then rename -snap to current volume. And only then remove backup
revision (if exceed revisions_to_keep). This way even if commit
operation is interrupted, there is still a volume with the data.
This requires also adjusting few functions to actually fallback to most
recent backup revision if the current volume isn't found - create
_vid_current property for this purpose.
Also, use -snap volume for clone operation and commit it normally later.
This makes it safer to interrupt or even revert.

QubesOS/qubes-issues#2256
2018-07-12 16:10:44 +02:00
Wojtek Porczyk
4e49b951ce Do not use assert statement in security logic
This is because assert statement gets optimised out when Python is run
with -O flag. This was pointed out to me by audience at PyWaw 76.
2018-06-11 12:32:05 +02:00
Marek Marczykowski-Górecki
39a9e4e422
Merge remote-tracking branch 'qubesos/pr/209'
* qubesos/pr/209:
  qubes/tests: change globals= to module= and fix syntax errors
  qubes/tests: use loadTestsFromNames for nose2 compat
2018-04-30 01:13:11 +02:00
Wojtek Porczyk
843bbdb2c5 qubes/tests: change globals= to module= and fix syntax errors 2018-04-24 18:30:55 +02:00
Marek Marczykowski-Górecki
f9e34eff51
api/admin: fix volume info formatting
Newline was put in a wrong place.

Fixes QubesOS/qubes-issues#3831
2018-04-16 22:17:48 +02:00
Marek Marczykowski-Górecki
bb40d61af9
storage/lvm: filter out warning about intended over-provisioning
Over-provisioning on LVM is intended. Since LVM do not have any option
to disable it (see [1] and discussion linked from there), filter the
warning in post-processing.

[1] https://bugzilla.redhat.com/1347008

Fixes QubesOS/qubes-issues#3744
2018-04-14 21:36:03 +02:00
Marek Marczykowski-Górecki
4794232745
tests: fix getting kernel package version inside VM
Use `sort -V` instead of `sort -n`.
2018-04-13 21:44:44 +02:00
Marek Marczykowski-Górecki
69f19bb7bb
tests/extra: add start_guid option to VMWrapper
Pass start_guid option to vm.start(), when using core2 compatibility
layer.
2018-04-13 21:43:15 +02:00
Marek Marczykowski-Górecki
6a191febc3
vm/qubesvm: fire 'domain-start-failed' event even if fail was early
Fire 'domain-start-failed' even even if failure occurred during
'domain-pre-start' event. This will make sure if _anyone_ have seen
'domain-pre-start' event, will also see 'domain-start-failed'. In some
cases it will look like spurious 'domain-start-failed', but it is safer
option than the alternative.
2018-04-13 16:07:32 +02:00
Marek Marczykowski-Górecki
ba82d9dc21
vm/qubesvm: check if all required devices are available before start
Fail the VM start early if some persistently-assigned device is missing.
This will both save time and provide clearer error message.

Fixes QubesOS/qubes-issues#3810
2018-04-13 16:03:42 +02:00
Marek Marczykowski-Górecki
2aa14623bf
storage/lvm: fix reporting lvm command error
Escape '%' in error message, as required by Admin API.

Fixes QubesOS/qubes-issues#3809
2018-04-13 15:57:09 +02:00
Marek Marczykowski-Górecki
94c8e25d3c
storage/lvm: save pool's revision_to_keep property
And also report it as part of admin.pool.Info Admin API.

QubesOS/qubes-issues#3256
2018-04-13 15:56:23 +02:00
Marek Marczykowski-Górecki
b168c4a6d7
api/admin: fix admin.vm.volume.Info when is_outdated() isn't supported
is_outdated() may be not supported by given volume pool driver. In that
case skip is_outdated information, instead of crashing the call.

Fixes QubesOS/qubes-issues#3767
2018-04-08 23:25:34 +02:00
Marek Marczykowski-Górecki
93b2424867
vm/qubesvm: fix missing icon handling in clone_disk_files()
Check for icon existence, not a directory for it.
2018-04-06 12:10:50 +02:00
Marek Marczykowski-Górecki
8df90f1a86
tests: yet another fix for integration tests cleanup code
Before waiting for remaining tasks on event loop (including libvirt
events), make sure all destroyed objects are really destroyed. This is
especially important for libvirt connections, which gets cleaned up only
when appropriate destructor (__del__) register a cleanup callback and it
gets called by the loop.
2018-04-05 01:28:55 +02:00
Marek Marczykowski-Górecki
3726692530
tests: skip tests using netcat when it isn't installed 2018-04-05 01:28:54 +02:00
Marek Marczykowski-Górecki
2dee554ab7
vm/mix/net: make vm.gateway6 consistent with vm.gateway
Use VM's actual IP address as a gateway for other VMs, instead of
hardcoded link-local address. This is important for sys-net generated
ICMP diagnostics packets - those must _not_ have link-local source
address, otherwise wouldn't be properly forwarded back to the right VM.
2018-04-03 00:20:06 +02:00
Marek Marczykowski-Górecki
f4be284331
vm/qubesvm: handle libvirt reporting domain already dead when killing
If domain die when trying to kill it, qubesd may loose a race and try to
kill it anyway. Handle libvirt exception in that case and conver it to
QubesVMNotStartedError - as it would be if qubesd would win the race.

Fixes QubesOS/qubes-issues#3755
2018-04-02 23:56:03 +02:00
Marek Marczykowski-Górecki
1e9bf18bcf
Typo fix 2018-04-02 23:24:30 +02:00
Wojtek Porczyk
dfe7688158 qubes/tests: use loadTestsFromNames for nose2 compat 2018-03-30 03:05:01 +02:00
Marek Marczykowski-Górecki
5e6c94bc35
Merge remote-tracking branch 'origin/tests-less-verbose'
* origin/tests-less-verbose:
  api/admin: do not log expected full tracebacks with ERROR level
2018-03-29 20:49:28 +02:00
Marek Marczykowski-Górecki
2c13ba9c48
storage/lvm: force default locale for lvm command cont.
Few more places where lvm commands are called.

Fixes QubesOS/qubes-issues#3753
2018-03-29 20:39:15 +02:00
Marek Marczykowski-Górecki
faca89875b
storage/lvm: force default locale for lvm command
Scripts do parse its output sometimes (especially `lvs`), so make sure
we always gets the same format, regardless of the environment. Including
decimal separator.

Fixes QubesOS/qubes-issues#3753
2018-03-29 00:53:17 +02:00
Marek Marczykowski-Górecki
bb7bb25d89
api/admin: do not log expected full tracebacks with ERROR level
Make tests output cleaner.
2018-03-23 02:26:05 +01:00
Marek Marczykowski-Górecki
d6b422cc36
Merge remote-tracking branch 'qubesos/pr/207'
* qubesos/pr/207:
  storage/reflink: strictly increasing revision ID
2018-03-22 01:54:38 +01:00
Marek Marczykowski-Górecki
74c1893ce3
tests: fix firewall tests after 6e8e48e3 "Avoid UTC datetime" 2018-03-22 01:47:41 +01:00
Rusty Bird
6a303760e9
storage/reflink: strictly increasing revision ID
Don't rely on timestamps to sort revisions - the clock can go backwards
due to time sync. Instead, use a monotonically increasing natural number
as the revision ID.

Old revision example: private.img@2018-01-02-03T04:05:06Z (ignored now)
New revision example: private.img.123@2018-01-02-03T04:05:06Z
2018-03-21 16:00:13 +00:00
Marek Marczykowski-Górecki
6469705196
Merge branch 'devel-storage-fixes'
* devel-storage-fixes:
  storage/file: use proper exception instead of assert
  storage/file: import data into temporary volume
  storage/lvm: check for LVM LV existence and type when creating ThinPool
  storage/lvm: fix size reporting just after creating LV
2018-03-21 02:06:25 +01:00
Marek Marczykowski-Górecki
5eceff84cb
storage/file: use proper exception instead of assert
Return readable message to the user.
2018-03-21 01:48:20 +01:00
Marek Marczykowski-Górecki
510fad9163
storage/file: import data into temporary volume
Similar to LVM changes, this fixes/improves multiple things:
 - no old data visible in the volume
 - failed import do not leave broken volume
 - parially imported data not visible to running VM

QubesOS/qubes-issues#3169
2018-03-21 01:48:07 +01:00
Marek Marczykowski-Górecki
e5413a3036
Merge branch 'storage-properties'
* storage-properties:
  storage: use None for size/usage properties if unknown
  tests: call search_pool_containing_dir with various dirs and pools
  storage: make DirectoryThinPool helper less verbose, add sudo
  api/admin: add 'included_in' to admin.pool.Info call
  storage: add Pool.included_in() method for checking nested pools
  storage: move and generalize RootThinPool helper class
  storage/kernels: refuse changes to 'rw' and 'revisions_to_keep'
  api/admin: implement admin.vm.volume.Set.rw method
  api/admin: include 'revisions_to_keep' and 'is_outdated' in volume info
2018-03-21 01:43:53 +01:00
Marek Marczykowski-Górecki
03dc3e315e
storage: use None for size/usage properties if unknown
Raising NotImplementedError in a _property_ is weird behaviour, better
suited for actions (methods). Use None instead.

QubesOS/qubes-issues#3241
2018-03-20 17:31:16 +01:00
Marek Marczykowski-Górecki
05c80c4531
tests: call search_pool_containing_dir with various dirs and pools
QubesOS/qubes-issues#3241
2018-03-20 16:53:41 +01:00
Marek Marczykowski-Górecki
825de49767
storage: make DirectoryThinPool helper less verbose, add sudo
Don't print scary messages when given pool cannot be found. Also, add
sudo to make it work from non-root user (tests)
2018-03-20 16:53:41 +01:00
Marek Marczykowski-Górecki
1bc640f3e0
api/admin: add 'included_in' to admin.pool.Info call
QubesOS/qubes-issues#3240
QubesOS/qubes-issues#3241
2018-03-20 16:53:40 +01:00
Marek Marczykowski-Górecki
d40fae9756
storage: add Pool.included_in() method for checking nested pools
It may happen that one pool is inside a volume of other pool. This is
the case for example for varlibqubes pool (file driver,
dir_path=/var/lib/qubes) and default lvm pool (lvm_thin driver). The
latter include whole root filesystem, so /var/lib/qubes too.
This is relevant for proper disk space calculation - to not count some
space twice.

QubesOS/qubes-issues#3240
QubesOS/qubes-issues#3241
2018-03-20 16:53:39 +01:00
Marek Marczykowski-Górecki
99f430511a
storage: move and generalize RootThinPool helper class
This is a class for finding thin pool containing root filesytem.
Generalize it to work for other filesystems too and rename to
DirectoryThinPool.
2018-03-20 16:52:48 +01:00
Marek Marczykowski-Górecki
a0723a9e32
storage/kernels: refuse changes to 'rw' and 'revisions_to_keep'
This pool driver support only rw=False and revisions_to_keep=0 volumes.
Since there is API for changing those properties dynamically, block it
at pool driver level, instead of silently ignoring them.
2018-03-20 16:52:46 +01:00
Marek Marczykowski-Górecki
376c8ec00d
api/admin: implement admin.vm.volume.Set.rw method
Allow setting 'rw' volume property.
2018-03-20 16:52:06 +01:00
Marek Marczykowski-Górecki
b66a2e9a51
Merge remote-tracking branch 'qubesos/pr/202'
* qubesos/pr/202:
  api/admin: fix backup exclude
2018-03-20 14:44:20 +01:00
Rusty Bird
a0c5014737
api/admin: fix backup exclude
Bugfix on 59abdeb
2018-03-20 02:41:37 +00:00
Marek Marczykowski-Górecki
93bccf583e
Merge remote-tracking branch 'qubesos/pr/200'
* qubesos/pr/200:
  Removed self.rules != old_rules
  Avoid UTC datetime
  Wrong init var to bool and missing call to total_seconds()

Fixes QubesOS/qubes-issues#3661
2018-03-20 01:19:58 +01:00
Marek Marczykowski-Górecki
a3a6a462f3
Merge remote-tracking branch 'qubesos/pr/201'
* qubesos/pr/201:
  storage/reflink: reorder start() to be more readable
  storage/reflink: simplify
  storage/reflink: let _remove_empty_dir() ignore ENOTEMPTY
  storage/reflink: show size in refused volume shrink message
  storage/reflink: fsync() after resizing existing file
2018-03-20 00:57:31 +01:00
Marek Marczykowski-Górecki
2a962c54db
api/admin: include 'revisions_to_keep' and 'is_outdated' in volume info
Since Volume.is_outdated() is a method, not a property, add a function
for handling serialization. And at the same time, fix None serialization
(applicable to 'source' property).

QubesOS/qubes-issues#3256
2018-03-19 04:00:21 +01:00
Rusty Bird
1743c76ca9
storage/reflink: reorder start() to be more readable
This also makes slightly more sense in the exotic (and currently unused)
case of restarting a crashed snap_on_start *and* save_on_stop volume.
2018-03-12 16:38:56 +00:00
Rusty Bird
31810db977
storage/reflink: simplify 2018-03-11 17:39:51 +00:00
Rusty Bird
c382eb3752
storage/reflink: let _remove_empty_dir() ignore ENOTEMPTY 2018-03-11 17:39:51 +00:00
Rusty Bird
023cb49293
storage/reflink: show size in refused volume shrink message
Like e6bb282 did for lvm.
2018-03-11 15:34:56 +00:00
Rusty Bird
c31d317c63
storage/reflink: fsync() after resizing existing file
Ensure that the updated metadata is written to disk.
2018-03-11 15:34:55 +00:00
donoban
39d904ea82
Removed self.rules != old_rules
After lot of testing it does not work properly. Could do something more
sophisticated but since calling save() is safe and probably lightweigth it is
not worth probably.
2018-03-08 05:25:42 -05:00
donoban
6e8e48e32d
Avoid UTC datetime
utcfromtimestamp() does not seems reliable and qubes-manager uses local time
2018-03-07 16:40:07 -05:00
donoban
b3b18f97f8
Wrong init var to bool and missing call to total_seconds()
fix https://github.com/QubesOS/qubes-issues/issues/3661
2018-03-07 16:37:44 -05:00
Marek Marczykowski-Górecki
e2b70306e5
Fix error message when using invalid VM as a template for DispVM
Don't crash (producing misleading error) when checking if
template_for_dispvms=True.

Fixes QubesOS/qubes-issues#3341
2018-03-05 23:47:33 +01:00
Marek Marczykowski-Górecki
7c4566ec14
vm/qubesvm: allow 'features-request' to have async handlers
Some handlers may want to call into other VMs (or even the one asking),
but vm.run() functions are coroutines, so needs to be called from
another coroutine. Allow for that.
Also fix typo in documentation.
2018-03-02 01:16:38 +01:00
Marek Marczykowski-Górecki
b1e83f899f
tests: convert pvgrub tests to core3 API
QubesOS/qubes-issues#3563
2018-02-28 04:41:30 +01:00
Marek Marczykowski-Górecki
1774be1c69
storage/kernel: skip modules.img block device if kernel dir doesn't have it
Some kernels (like pvgrub2) may not provide modules.img and it isn't an
error. Don't break VM startup in that case, skip that device instead.

Fixes QubesOS/qubes-issues#3563
2018-02-27 04:35:05 +01:00
Marek Marczykowski-Górecki
ba5d19e1b4
vm: provide better error message for VM startup timeout
"Cannot execute qrexec-daemon!" error is very misleading for a startup
timeout error, make it clearer. This rely on qrexec-daemon using
distinct exit code for timeout error, but even without that, include its
stderr in the error message.
2018-02-27 04:35:05 +01:00
Marek Marczykowski-Górecki
173e7e4250
vm: fix calling vm.detach_network() when really needed
*oldvalue* argument to property event handler is provided only when the
value was not default. Check vm.netvm directly to resolve also default.
2018-02-26 02:52:27 +01:00
Marek Marczykowski-Górecki
92a460526f
tests: fix extra test's VMWrapper.run(wait=False)
The call was ignored.
2018-02-26 02:45:26 +01:00
Marek Marczykowski-Górecki
81aea0c2c7
Merge remote-tracking branch 'qubesos/pr/198'
* qubesos/pr/198:
  backup.py: add vmN/empty file if no other files to backup
  Allow include=None to be passed to admin.backup.Info
  Add include_in_backups property for AdminVM
  Use !auto_cleanup as DispVM include_in_backups default
2018-02-25 00:09:12 +01:00
Rusty Bird
6886e70aa9
backup.py: add vmN/empty file if no other files to backup 2018-02-24 22:43:53 +00:00
Rusty Bird
59abdeb30e
Allow include=None to be passed to admin.backup.Info
This is forwarded to qubes.backup.Backup(), which uses None to choose
VMs according to their include_in_backups property.

('git show -w' recommended)
2018-02-23 21:29:15 +00:00
Rusty Bird
dbaf60ca24
Add include_in_backups property for AdminVM 2018-02-23 21:29:15 +00:00
Rusty Bird
84ca0c6df5
Use !auto_cleanup as DispVM include_in_backups default 2018-02-23 21:29:15 +00:00
Marek Marczykowski-Górecki
716114f676
Merge remote-tracking branch 'qubesos/pr/197'
* qubesos/pr/197:
  Don't fire domain-stopped/-shutdown while VM is still Dying
2018-02-22 21:14:55 +01:00
Rusty Bird
f96fd70f76
Don't fire domain-stopped/-shutdown while VM is still Dying
Lots of code expects the VM to be Halted after receiving one of these
events, but it could also be Dying or Crashed. Get rid of the Dying case
at least, by waiting until the VM has transitioned out of it.

Fixes e.g. the following DispVM cleanup bug:

    $ qvm-create -C DispVM --prop auto_cleanup=True -l red dispvm
    $ qvm-start dispvm
    $ qvm-shutdown --wait dispvm  # this won't remove dispvm
    $ qvm-start dispvm
    $ qvm-kill dispvm  # but this will
2018-02-22 19:53:29 +00:00
Marek Marczykowski-Górecki
1562defd42
Merge remote-tracking branch 'qubesos/pr/193'
* qubesos/pr/193:
  tests: add a test for removing expired firewall rules
  firewall: use asyncio's call_later instead of systemd to reload rules
2018-02-22 19:47:37 +01:00
Marek Marczykowski-Górecki
7731a56daa
Merge remote-tracking branch 'qubesos/pr/192'
* qubesos/pr/192:
  storage/lvm: add more details to refused volume shrink message
  tests: storage/lvm resize
  storage/lvm: fix resizing not persistent volumes
2018-02-22 19:46:11 +01:00
Marek Marczykowski-Górecki
b283ab2b52
Merge remote-tracking branch 'qubesos/pr/195'
* qubesos/pr/195:
  Fix is_running non-boolean
2018-02-22 17:52:09 +01:00
Marek Marczykowski-Górecki
033100fa7e
Merge remote-tracking branch 'qubesos/pr/194'
* qubesos/pr/194:
  reflink: style fix
  storage: typo fix
  lvm_thin: _remove_revisions() on revisions_to_keep==0
  lvm_thin: don't purge one revision too few
  lvm_thin: really remove revision
  lvm_thin: fill in volume's revisions_to_keep from pool
2018-02-22 17:51:37 +01:00
Christopher Laprise
75d8c553f9
Fix is_running non-boolean 2018-02-20 22:30:47 -05:00
Marek Marczykowski-Górecki
68b6f1ec76
qubespolicy: use '@' instead of '$' for policy keywords
Using '$' is easy to misuse in shell scripts, shell commands etc. After
all this years, lets abandon this dangerous character and move to
something safer: '@'. The choice was made after reviewing specifications
of various shells on different operating systems and this is the
character that have no special meaning in none of them.

To preserve compatibility, automatically translate '$' to '@' when
loading policy files.
2018-02-19 03:33:40 +01:00
Rusty Bird
37e1aedfa3
reflink: style fix 2018-02-16 21:47:39 +00:00
Rusty Bird
c871424fb0
storage: typo fix 2018-02-16 21:47:37 +00:00
Rusty Bird
f6542effc5
lvm_thin: _remove_revisions() on revisions_to_keep==0
If revisions_to_keep is 0, it may nevertheless have been > 0 before, so
it makes sense to call _remove_revisions() and hold back none (not all)
of the revisions in this case.
2018-02-16 21:47:36 +00:00
Rusty Bird
ade49d44b7
lvm_thin: don't purge one revision too few
We just added one that must be taken into account.
2018-02-16 21:47:35 +00:00
Rusty Bird
86998b0815
lvm_thin: really remove revision 2018-02-16 21:47:33 +00:00
Rusty Bird
9f468f93f5
lvm_thin: fill in volume's revisions_to_keep from pool
Otherwise, the base class defaults revisions_to_keep to 0.
2018-02-16 21:47:32 +00:00
Marek Marczykowski-Górecki
b00bbb73e4
Merge remote-tracking branch 'qubesos/pr/190'
* qubesos/pr/190:
  Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90
  replaced underscore by dash and update test accordingly
  Updated assert content for test_620_qdb_standalone in TC_90_QubesVM
  Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372
2018-02-14 01:29:08 +01:00
Marek Marczykowski-Górecki
e3d3e149f0
Fix too long line 2018-02-13 11:27:59 +01:00
Marek Marczykowski-Górecki
209af07fd0
Merge remote-tracking branch 'qubesos/pr/188'
* qubesos/pr/188:
  file-reflink, a storage driver optimized for CoW filesystems
  Make AppVM/DispVM root volume rw to avoid CoW-on-CoW
2018-02-13 05:20:52 +01:00
Marek Marczykowski-Górecki
5afd5fb04f
Merge remote-tracking branch 'qubesos/pr/191'
* qubesos/pr/191:
  qubes/tests: fix QubesTestCase.qrexec_policy
2018-02-13 05:20:39 +01:00
Marek Marczykowski-Górecki
717bc4ace3
vm/appvm: forbid changing template if the are children DispVMs
Changing AppVM's template will not update root volume reference in
DispVMs based on it. For now forbid such change.

Fixes QubesOS/qubes-issues#3576
2018-02-13 04:53:39 +01:00
Rusty Bird
1695a732b8
file-reflink, a storage driver optimized for CoW filesystems
This adds the file-reflink storage driver. It is never selected
automatically for pool creation, especially not the creation of
'varlibqubes' (though it can be used if set up manually).

The code is quite small:

               reflink.py  lvm.py      file.py + block-snapshot
    sloccount  334 lines   447 (134%)  570 (171%)

Background: btrfs and XFS (but not yet ZFS) support instant copies of
individual files through the 'FICLONE' ioctl behind 'cp --reflink'.
Which file-reflink uses to snapshot VM image files without an extra
device-mapper layer. All the snapshots are essentially freestanding;
there's no functional origin vs. snapshot distinction.

In contrast to 'file'-on-btrfs, file-reflink inherently avoids
CoW-on-CoW. Which is a bigger issue now on R4.0, where even AppVMs'
private volumes are CoW. (And turning off the lower, filesystem-level
CoW for 'file'-on-btrfs images would turn off data checksums too, i.e.
protection against bit rot.)

Also in contrast to 'file', all storage features are supported,
including

    - any number of revisions_to_keep
    - volume.revert()
    - volume.is_outdated
    - online fstrim/discard

Example tree of a file-reflink pool - *-dirty.img are connected to Xen:

    - /var/lib/testpool/appvms/foo/volatile-dirty.img
    - /var/lib/testpool/appvms/foo/root-dirty.img
    - /var/lib/testpool/appvms/foo/root.img
    - /var/lib/testpool/appvms/foo/private-dirty.img
    - /var/lib/testpool/appvms/foo/private.img
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T03:04:05Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T04:05:06Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T05:06:07Z
    - /var/lib/testpool/appvms/bar/...
    - /var/lib/testpool/appvms/...
    - /var/lib/testpool/template-vms/fedora-26/...
    - /var/lib/testpool/template-vms/...

It looks similar to a 'file' pool tree, and in fact file-reflink is
drop-in compatible:

    $ qvm-shutdown --all --wait
    $ systemctl stop qubesd
    $ sed 's/ driver="file"/ driver="file-reflink"/g' -i.bak /var/lib/qubes/qubes.xml
    $ systemctl start qubesd
    $ sudo rm -f /path/to/pool/*/*/*-cow.img*

If the user tries to create a fresh file-reflink pool on a filesystem
that doesn't support reflinks, qvm-pool will abort and mention the
'setup_check=no' option. Which can be passed to force a fallback on
regular sparse copies, with of course lots of time/space overhead. The
same fallback code is also used when initially cloning a VM from a
foreign pool, or from another file-reflink pool on a different
mountpoint.

'journalctl -fu qubesd' will show all file-reflink copy/rename/remove
operations on VM creation/startup/shutdown/etc.
2018-02-12 21:20:05 +00:00
Rusty Bird
7a75e1090d
Make AppVM/DispVM root volume rw to avoid CoW-on-CoW 2018-02-12 21:20:04 +00:00
Marek Marczykowski-Górecki
340b8dbfe2
tests: add a test for removing expired firewall rules
QubesOS/qubes-issues#1173
2018-02-07 02:48:12 +01:00
Marek Marczykowski-Górecki
5e89b23288
firewall: use asyncio's call_later instead of systemd to reload rules
When some expiring rules are present, it is necessary to reload firewall
when those rules expire. Previously systemd timer was used to trigger
this action, but since we have own daemon now, it isn't necessary
anymore - use this daemon for that.
Additionally automatically removing expired rules was completely broken
in R4.0.

Fixes QubesOS/qubes-issues#1173
2018-02-07 02:48:11 +01:00
Marek Marczykowski-Górecki
e6bb2828f1
storage/lvm: add more details to refused volume shrink message
Include old and new size.
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
4e314c9792
tests: storage/lvm resize
QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
7903dc53f9
storage/lvm: fix resizing not persistent volumes
Even when volume is not persistent (like TemplateBasedVM:root), it
should be resizeable. Just the new size, similarly to the volume
content, will be reverted after qube shutdown.

Additionally, when VM is running, volume resize should affect _only_ its
temporary snapshot. This way resize can be properly reverted together
with actual volume changes (which include resize2fs call).

Fixes QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Wojtek Porczyk
8083d6dece qubes/tests: fix QubesTestCase.qrexec_policy
The context manager left empty file.

fixes QubesOS/qubes-issues#3535
2018-02-04 14:03:12 +01:00
Yassine Ilmi
65d5c41fdb
Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90 2018-02-01 01:03:05 +00:00
Yassine Ilmi
a0d45aac9c
replaced underscore by dash and update test accordingly 2018-02-01 00:50:42 +00:00
Yassine Ilmi
0eb978c59b
Updated assert content for test_620_qdb_standalone in TC_90_QubesVM 2018-02-01 00:33:12 +00:00
Yassine Ilmi
1c3b412ef8
Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372 2018-02-01 00:12:51 +00:00
Marek Marczykowski-Górecki
a6a7efc9a7
vm/mix/net: fix handling network detach/attach on VM startup
- catch both QubesException and libvirtError - do not kill starting VM
just because an error while connecting _other_ VMs to it
- try to detach network first (and do not abort on error) - if
libvirt/libxl will manage to cleanup stale interface this way, the
attach operation below may succeed.

Fixes QubesOS/qubes-issues#3163
2018-01-29 23:06:21 +01:00
Marek Marczykowski-Górecki
86026e364f
Fix starting PCI-having HVMs on early system boot and later
1. Make sure VMs are started after dom0 actual memory usage is reported
to qmemman, otherwise dom0 will hold 4GB, even if just a little over 1GB
is needed at that time.

2. Request only vm.memory MB from qmemman, instead of vm.maxmem. While
HVM with PCI devices indeed do not support populate-on-demand, this is
already handled in libvirt XML.

The later may often cause VM startup fail on systems with 8GB of memory,
because maxmem is 4GB there and with dom0 keeping the other 4GB (see
point 1) there is not enough memory to start any sych VM.

Fixes QubesOS/qubes-issues#3462
2018-01-29 22:57:32 +01:00
Marek Marczykowski-Górecki
2c2b89e2e0
tests: adjust for type=pvh change 2018-01-27 01:34:41 +01:00
Marek Marczykowski-Górecki
99134efc3c
tests: set virt_mode='hvm' for PCI tests 2018-01-26 23:49:11 +01:00
Marek Marczykowski-Górecki
eb846f6647
Merge remote-tracking branch 'qubesos/pr/187'
* qubesos/pr/187:
  Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists
  Make 'qvm-volume revert' really use the latest revision
  Fix wrong mocks of Volume.revisions
2018-01-22 15:39:13 +01:00
Marek Marczykowski-Górecki
e577de7d49
Merge remote-tracking branch 'qubesos/pr/186'
* qubesos/pr/186:
  tests: check if udev do not touch VM's volumes
2018-01-22 15:33:50 +01:00
Marek Marczykowski-Górecki
74eb3f3208
Merge remote-tracking branch 'qubesos/pr/185'
* qubesos/pr/185:
  vm: remove doc for non-existing event `monitor-layout-change`
  vm: include tag/feature name in event name
  events: add support for wildcard event handlers
2018-01-22 15:32:57 +01:00
Rusty Bird
4ae854fdaf
Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists 2018-01-21 22:28:47 +00:00
Rusty Bird
fe77b0ec85
Make 'qvm-volume revert' really use the latest revision
admin.vm.volume.ListSnapshots returned volume revisions in undefined
order, but 'qvm-volume revert' assumes the list to be in chronological
order. Make that assumption true.
2018-01-20 23:20:23 +00:00
Rusty Bird
bf1f1ac5ff
Fix wrong mocks of Volume.revisions
It's a dict, not a list.
2018-01-20 23:20:22 +00:00
Marek Marczykowski-Górecki
ab15d43622
tests: check if udev do not touch VM's volumes
Based on shell version by @rustybird
2018-01-19 18:05:50 +01:00
Marek Marczykowski-Górecki
91639d7933
app: define values for default_dispvm=None, updatevm=None
If those VMs are not set, default to None, instead of throwing
AttributeError.

Fixes QubesOS/qubes-issues#3475
2018-01-19 04:50:04 +01:00
Marek Marczykowski-Górecki
b245bbca6f
tests: update PCI devices tests for core3 API
- use asyncio where needed
- attach now takes DeviceAssignment, not DeviceInfo
- PCI ident have ':' replaced with '_'
2018-01-18 17:36:38 +01:00
Marek Marczykowski-Górecki
edbfd3843e
tests: make tests.extra.VMWrapper hashable
Allow using VMWrapper as dict key, same as QubesVM.
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
46177c7c9f
qmemman: do not close stdout/stderr in daemon mode
Allow exceptions to be logged to syslog/journald
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
ca41ca66cd
qmemman: fix early crash
clear_outdated_error_markers crashes if memory stats are not retrieved
yet. In practice it crashes at the very first call during daemon
startup, making the whole qmemman unusable.

This fixes bf4306b815
    qmemman: clear "not responding" flags when VM require more memory

QubesOS/qubes-issues#3265
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
dce3b609b4
qubesvm: do not try to define libvirt object in offline mode
The idea is to not touch libvirt at all.
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
f2b9be3607
tests: one more missing virt_mode=hvm 2018-01-17 15:23:22 +01:00
Marek Marczykowski-Górecki
7905783861
qubesvm: PVH minor improvements
- use capital letters in acronyms in documentation to match upstream
documentation.
- refuse to start a PVH with without kernel set - provide meaningful
error message
2018-01-16 21:42:20 +01:00
Marek Marczykowski-Górecki
4d59f883a0
tests: minor fixes
- FD leak
- switch to xterm to test also on minimal template
2018-01-16 21:41:38 +01:00
Marek Marczykowski-Górecki
06e82eccb0
tests: add run_service and qrexec_policy wrappers to ExtraTestCase
Provide same API as in core2, especially without exposing asyncio
usage. This allows qubes-usb-proxy and qubes-split-gpg tests to run.
2018-01-16 21:39:22 +01:00
Marek Marczykowski-Górecki
c17b634913
tests: clear PCIDevice cache after each test
This is yet another place where references to VM objects contribute to
object leaks.
2018-01-16 21:32:15 +01:00
Marek Marczykowski-Górecki
d2a7cbb83e
tests: mock vmm.xs
Now it is needed by some unit tests (those calling create_qdb_entries).
2018-01-15 15:58:34 +01:00
Marek Marczykowski-Górecki
ae7031fe7e
tests: explicitly set virt_mode to HVM when needed
HVM is no longer default
2018-01-15 15:57:54 +01:00
Marek Marczykowski-Górecki
241f1d1d3b
tests: do not leak open file 2018-01-15 15:57:30 +01:00
Marek Marczykowski-Górecki
f1a5ca64fd
Merge remote-tracking branch 'qubesos/pr/180'
* qubesos/pr/180:
  vm/qubesvm: default to PVH unless PCI devices are assigned
  vm/qubesvm: expose 'start_time' property over Admin API
  vm/qubesvm: revert backup_timestamp to '%s' format
  doc: link qvm-device man page for qvm-block, qvm-pci, qvm-usb
2018-01-15 04:22:28 +01:00
Marek Marczykowski-Górecki
21760d8ff0
Merge remote-tracking branch 'qubesos/pr/179'
* qubesos/pr/179:
  qmemman: request VMs balloon down with 16MB safety margin
  qmemman: clear "not responding" flags when VM require more memory
  qmemman: slightly improve logging
  qmemman: reformat code, especially comments
2018-01-15 04:21:40 +01:00
Marek Marczykowski-Górecki
b20c3d3458
Merge remote-tracking branch 'qubesos/pr/174'
* qubesos/pr/174:
  tests: fix (system) network tests after switching to ipaddress module
  tests: resurrect extra tests loader
  tests: basic salt integration tests
2018-01-15 04:20:32 +01:00
Marek Marczykowski-Górecki
4ff53879a0
vm/qubesvm: default to PVH unless PCI devices are assigned
Fixes QubesOS/qubes-issues#2185
2018-01-15 03:34:46 +01:00
Marek Marczykowski-Górecki
d9da747ab0
vm/qubesvm: expose 'start_time' property over Admin API
It is useful at least for Qubes Manager.
2018-01-12 05:34:46 +01:00
Marek Marczykowski-Górecki
85e80f2329
vm/qubesvm: revert backup_timestamp to '%s' format
Human readable format `str(datetime.datetime)` is a nightmare for Admin
API level communication. Especially setting the property in a format
that it was read was not supported, and handling such format in
untrusted input handling code is a bad idea. Revert to a simple intiger
format.
2018-01-12 05:34:45 +01:00
Marek Marczykowski-Górecki
bcf42c13fa
storage/lvm: check for LVM LV existence and type when creating ThinPool
Check if requested thin pool exists and really is thin pool.

QubesOS/qubes-issues#3438
2018-01-12 05:12:08 +01:00
Marek Marczykowski-Górecki
377f331d52
storage/lvm: fix size reporting just after creating LV
Force cache refresh after registering new pool - it might be just
created.

QubesOS/qubes-issues#3438
2018-01-11 03:56:30 +01:00
Marek Marczykowski-Górecki
4bca631350
qmemman: request VMs balloon down with 16MB safety margin
It looks like Linux balloon driver do not always precisely respect
requested target memory, but perform some rounding. Also, in some cases
(HVM domains), VM do not see all the memory that Xen have assigned to it
- there are some additional Xen pools for internal usage.
Include 16MB safety margin in memory requests to account for those two
things. This will avoid setting "no_response" flag for most of VMs.

QubesOS/qubes-issues#3265
2018-01-11 03:41:55 +01:00
Marek Marczykowski-Górecki
bf4306b815
qmemman: clear "not responding" flags when VM require more memory
Clear slow_memset_react/no_progress flags when VM request more memory
than it have assigned. If there is some available, it may be given to
such VM, solving the original problem (not reacting to balloon down
request). In any case, qmemman algorithm should not try to take away
memory from under-provisioned VM.

Fixes QubesOS/qubes-issues#3265
2018-01-11 03:41:54 +01:00
Marek Marczykowski-Górecki
4cf6a93b5b
qmemman: slightly improve logging
Add logging more info about each domain state:
 - last requested target
 - no_progress and slow_memset_react flags

This makes it unnecessary to log separately when those flags are cleared.
2018-01-11 03:41:54 +01:00
Marek Marczykowski-Górecki
8e288d9f81
qmemman: reformat code, especially comments
Indent comments to match code indentation to make it readable. Also,
wrap long lines. Fix few typos in comments.

No functional change.
2018-01-07 17:04:25 +01:00
Marek Marczykowski-Górecki
f0fe02998b
vm: remove doc for non-existing event monitor-layout-change 2018-01-06 15:10:54 +01:00
Marek Marczykowski-Górecki
50d34755fa
vm: include tag/feature name in event name
Rename events:
 - domain-feature-set -> domain-feature-set:feature
 - domain-feature-delete -> domain-feature-delete:feature
 - domain-tag-add -> domain-tag-add:tag
 - domain-tag-delete -> domain-tag-delete:tag

Make it consistent with property-* events. It makes more sense to
include tag/feature name in event name, so handler can watch a single
tag/feature - which is the most common case. Otherwise, most handlers
would begin with `if feature == '...'` anyway, wasting time on most
events.

In cases where multiple features/tags should be handled by a single
handler, it is now possible to register a handler with wildcard, for
example `domain-feature-set:*`.
2018-01-06 15:05:34 +01:00
Marek Marczykowski-Górecki
5a39e77708
events: add support for wildcard event handlers
Support registering handlers for more flexible wildcard events: not only
'*', but also 'something*'. This allows to register handlers for
'property-set:*' and such.
2018-01-06 00:40:19 +01:00
Marek Marczykowski-Górecki
a66c9afb18
Merge remote-tracking branch 'qubesos/pr/177'
* qubesos/pr/177:
  Use default_dispvm (not default_template) for new DispVM
2018-01-05 16:29:14 +01:00
Marek Marczykowski-Górecki
d83a07177b
Merge remote-tracking branch 'qubesos/pr/176'
* qubesos/pr/176:
  Do not remove VMs installed via rpm
2018-01-05 16:27:12 +01:00
Rusty Bird
4a2fabc17f
Use default_dispvm (not default_template) for new DispVM
When creating a new VM of type DispVM without specifying any template
(e.g. "qvm-create --class DispVM --label red foo"), use default_dispvm.
Otherwise it would fail saying "Got empty response from qubesd."
2018-01-02 23:19:02 +00:00
Christopher Laprise
47b49c4755
Do not remove VMs installed via rpm 2017-12-29 23:24:41 -05:00
Marek Marczykowski-Górecki
3fa0972317
tests: fix (system) network tests after switching to ipaddress module 2017-12-23 16:43:24 +01:00
Marek Marczykowski-Górecki
3668a73ca2
tests: resurrect extra tests loader
Load integration tests from outside of core-admin repository, through
entry points.
Create wrapper for VM object to keep very basic compatibility with tests
written for core2. This means if test use only basic functionality
(vm.start(), vm.run()), the same test will work for both core2 and
core3. This is especially important for app-* repositories, where the
same version serves multiple Qubes branches.
This also hides asyncio usage from tests writer.

See QubesOS/qubes-issues#1800 for details on original feature.
2017-12-23 02:26:16 +01:00
Marek Marczykowski-Górecki
962742880f
tests: basic salt integration tests
Test base functions of dom0 module (creating VM, setting property) and
configuring system inside of VM (through DispVM). The later is done for
each available template (the process use salt installed in that
template, not copied from dom0).

QubesOS/qubes-issues#3316
2017-12-23 02:26:15 +01:00
Marek Marczykowski-Górecki
32c6083e1c
Make pylint happy
Fix thing detected by updated pylint in Travis-CI
2017-12-21 18:19:10 +01:00
Marek Marczykowski-Górecki
f5fe10e2ad
vm/adminvm: fix type of AdminVM.qid
It is defined as int in QubesVM.qid, make it consistent.
2017-12-21 18:18:12 +01:00
Marek Marczykowski-Górecki
9a4d2abf1f
tests: booting VM from ISO image
Check two cases: if ISO image is in dom0 and if its in a VM.

QubesOS/qubes-issues#3339
2017-12-14 23:26:53 +01:00
Marek Marczykowski-Górecki
96bd734852
storage/lvm: clean -snap LV on volume removal 2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
466bf89aae
Fix starting VM with kernel=None
When dom0 do not provide the kernel, it should also not set kernel
command line in libvirt config. Otherwise qemu in stubdom fails to start
because it get -append option without -kernel, which is illegal
configuration.

Fixes QubesOS/qubes-issues#3339
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
fd45378041
api/admin: make libvirt start error more informative
Point where to look for details.
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
297cb4d012
vm/mix/net: fill QubesDB for already connected VMs
There may be cases when VM providing the network to other VMs is started
later - for example VM restart. While this is rare case (and currently
broken because of QubesOS/qubes-issues#1426), do not assume it will
always be the case.
2017-12-14 23:26:51 +01:00
Marek Marczykowski-Górecki
1187e43697
tests: more vm.create_qdb_entries() tests 2017-12-14 02:09:31 +01:00
Marek Marczykowski-Górecki
97564f014c
vm/mix/net: fix setting QubesDB after converting to ipaddress module 2017-12-14 02:07:57 +01:00
Marek Marczykowski-Górecki
faef890c9a
vm/qubesvm: write QubesDB /qubes-netvm-gateway6 entry when set
This is needed for network-providing VM to actually provide IPv6
connection too.

QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
e12a66f103
vm/mix/net: use ipaddress module for ip and ip6 properties
It has built-in validation, which is much more elegant than custom regex
or socket call.

Suggested by @woju
QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
f3cf58e6f2
tests: add integration tests for IPv6
Run also all IPv4 tests with IPv6 enabled to check for regressions
(broken IPv4 because of enabled IPv6).

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
0786edf8a3
tests: add IPv6-related unit tests
Check produced libvirt XML, and QubesDB entries

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
18f159f8ec
Add IPv6 related VM properties
Add property for IPv6 address ('ip6'). Build default value similarly to
IPv4 - common prefix + QID or Disp ID (for DispVMs).
This all is disabled unless 'ipv6' feature is enabled. It is inherited
from netvm (not template).
Even when enabled, VM may decide to not use it - or simply not support
it.

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
bf59b00f1d
features: add check_with_netvm, similar to check_with_template
Allow using default feature value from netvm, not template. This makes
sense for network-related features like using tor, supporting ipv6 etc.

Similarly to check_with_template, expose it also on Admin API.
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
f223594f92
app: kill default_fw_netvm property
Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)

As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.

Fixes QubesOS/qubes-issues#3247
2017-12-07 01:40:29 +01:00
Marek Marczykowski-Górecki
f2cd7fb226
Merge branch 'tests-and-fixes-20171205' 2017-12-07 01:39:34 +01:00
Marek Marczykowski-Górecki
7b81d7affa
Merge remote-tracking branch 'qubesos/pr/166'
* qubesos/pr/166:
  create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
  change default pool code to be fast
  cache PropertyHolder.property_list and use O(1) property name lookups
  remove unused netid code
  cache isinstance(default, collections.Callable)
  don't access netvm if it's None in visible_gateway/netmask
2017-12-06 00:41:37 +01:00
Marek Marczykowski-Górecki
4d6bfbab4d
tests: improve spoof_ip test
Not only check if full round trip ping (does not) work, but also if just
echo-request get filtered.
2017-12-05 17:40:30 +01:00
Marek Marczykowski-Górecki
379add52ba
tests: skip network tests on whonix-gw and whonix-ws
whonix-ws also have non-standard firewall and require specific tests for
that.
2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
088c255355
tests: add create_qdb_entries() unit test 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
384a792b8a
typo in docstring 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
d54cef5554
app: fix creating dom0 object when not already present in qubes.xml
It's constant properties are now really constant, no need to provide
them explicitly.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
0afee4b05e
Improve checking for netvm loop
There were many cases were the check was missing:
 - changing default_netvm
 - resetting netvm to default value
 - loading already broken qubes.xml

Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...

Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
2d830caab9
app: check DEFAULT_LVM_POOL variable for default pool
Allow to hint default_pool what is default storage pool - especially
useful for tests.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
749e8497e3
api/admin: exclude regex attribute from DeviceInfo structure
DeviceInfo may contain 'regex' attribute - it isn't intended to be
reported through Admin API. Also, mark 'libvirt_regex' attribute as
private.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
da97f4d84c
qubesvm: make initial qmemman request consistent with libvirt config
If HVM have PCI device, it can't use PoD, so need 'maxmem' memory to be
started. Request that much from qmemman.
Note that is is somehow independent of enabling or not dynamic memory
management for the VM (`service.meminfo-writer` feature). Even if VM
initially had assigned maxmem memory, it can be later ballooned down.

QubesOS/qubes-issues#3207
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
19a1579a99
tests: fix deadlock in filecopy test
Error window (where test need to send Enter key) is opened while
qvm-move-to-vm is still running.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
99874a0a25
tests: make waiting for window asyncio aware
For now just replace sleep with asyncio.sleep. Later it may make sense
to change subprocess.call too.
2017-12-05 17:39:32 +01:00
Jean-Philippe Ouellet
b41c118166
Fix typo in volume import end handler 2017-11-21 01:05:06 -05:00
Marek Marczykowski-Górecki
23ee8aeb47
vm/qubesvm: fix handling vm.storage.verify() errors
except section try to access qmemman_client variable - make it defined
at that stage.
2017-11-21 04:16:30 +01:00
Marek Marczykowski-Górecki
a92dd99fbb
Merge branch '20171107-storage'
* 20171107-storage:
  api/admin: add API for changing revisions_to_keep dynamically
  storage/file: move revisions_to_keep restrictions to property setter
  api/admin: hide dd statistics in admin.vm.volume.Import call
  storage/lvm: fix importing different-sized volume from another pool
  storage/file: fix preserving spareness on volume clone
  api/admin: add pool size and usage to admin.pool.Info response
  storage: add size and usage properties to pool object
2017-11-20 22:52:50 +01:00
Marek Marczykowski-Górecki
2c6c766968
Merge branch '20171107-tests-backup-api-misc'
* 20171107-tests-backup-api-misc:
  test: make race condition on xterm close less likely
  tests/backupcompatibility: fix handling 'internal' property
  backup: fix handling target write error (like no disk space)
  tests/backupcompatibility: drop R1 format tests
  backup: use offline_mode for backup collection
  qubespolicy: fix handling '$adminvm' target with ask action
  app: drop reference to libvirt object after undefining it
  vm: always log startup fail
  api: do not log handled errors sent to a client
  tests/backups: convert to new restore handling - using qubesadmin module
  app: clarify error message on failed domain remove (used somewhere)
  Fix qubes-core.service ordering
2017-11-20 22:52:26 +01:00