Commit Graph

1693 Commits

Author SHA1 Message Date
Wojtek Porczyk
ff612a870b qubes/features: check_with_(template_and_)adminvm
- Two new methods: .features.check_with_adminvm() and
  .check_with_template_and_adminvm(). Common code refactored.

- Two new AdminAPI calls to take advantage of the methods:
  - admin.vm.feature.CheckWithAdminVM
  - admin.vm.feature.CheckWithTemplateAndAdminVM

- Features manager moved to separate module in anticipation of features
  on app object in R5.0. The attribute Features.vm renamed to
  Features.subject.

- Documentation, tests.
2018-12-20 18:28:33 +01:00
Marek Marczykowski-Górecki
3728230e3c
Merge branch 'maxmem' 2018-12-09 18:38:21 +01:00
Marek Marczykowski-Górecki
3606010591
Merge branch 'devel-20181205'
* devel-20181205:
  vm/dispvm: fix /qubes-vm-presistence qubesdb entry
  vm/mix/net: prevent setting provides_network=false if qube is still used
  tests: updates-available notification
  tests/network: reduce code duplication
  tests: listen on 'misc' socket too
2018-12-09 18:35:35 +01:00
Marek Marczykowski-Górecki
f66aae220f
vm/dispvm: fix /qubes-vm-presistence qubesdb entry
It should be 'none' for DispVM.

Fixes QubesOS/qubes-issues#4007
2018-12-08 23:13:30 +01:00
AJ Jordan
d4e567cb10
Fix typo 2018-12-06 20:43:39 -05:00
Marek Marczykowski-Górecki
7a6e4b284f
vm/mix/net: prevent setting provides_network=false if qube is still used
Add symmetric check to the one in 'netvm' property.

Fixes QubesOS/qubes-issues#4552
2018-12-06 20:37:50 +01:00
Marek Marczykowski-Górecki
9f6c97596d
tests: updates-available notification
First install test-pkg-1.0, then add test-pkg-1.1 to repo and check if
updates-available flag is set. Then install updates and check if the
flag is cleared.

QubesOS/qubes-issues#2009
2018-12-05 06:19:34 +01:00
Marek Marczykowski-Górecki
cea9f66bc3
tests/network: reduce code duplication 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
2edafacbe4
tests: listen on 'misc' socket too 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
ccd7e285a1
Remove unused core2migration.py
It is part of backup-restore code, but it was moved to
core-admin-client repo.
2018-12-04 16:12:33 +01:00
Marek Marczykowski-Górecki
162196a1f8
tests: fix lookup domain by name 2018-12-04 16:12:09 +01:00
Marek Marczykowski-Górecki
91c9e49612
Drop unnecessary 'pass' to make pylint happy 2018-12-04 16:06:33 +01:00
Marek Marczykowski-Górecki
eee9964b11
Merge remote-tracking branch 'origin/release4.0' 2018-12-04 16:03:33 +01:00
Marek Marczykowski-Górecki
7d1bcaf64c Introduce management_dispvm property
The new property is meant for management stack (Salt) to set which DVM
template should be used to maintain given VM. Since the DispVM based on
it will be given ultimate control over target VM (qubes.VMShell
service), it should be trusted. The one pointed to by default_dispvm
not necessary is one.

The property defaults to the value from the template (if any), and then
to a global management_dispvm property. By default it is set to None.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-12-03 19:18:26 +01:00
Marek Marczykowski-Górecki
d1e20e5178
Merge remote-tracking branch 'origin/pr/242'
* origin/pr/242:
  Drop async future check with python3 version
  tracis-ci: add support for fc29 dom0
2018-11-24 14:43:43 +01:00
fepitre
313a22d4d6 Drop async future check with python3 version
QubesOS/qubes-issues#4225
2018-11-22 19:34:58 +01:00
Marek Marczykowski-Górecki
087a02c7f4
ext/services: add automatic migration meminfo-writer=False -> maxmem=0
Migrate meminfo-writer=False service setting to maxmem=0 as a method to
disable dynamic memory management. Remove the service from vm.features
dict in the process.

Additionally, translate any attempt to set the service.meminfo-writer
feature to either setting maxmem=0 or resetting it to the default (which
is memory balancing enabled if supported by given domain). This is to at
least partially not break existing tools using service.meminfo-writer as
a way to control dynamic memory management. This code does _not_ support
reading service.meminfo-writer feature state to get the current state of
dynamic memory management, as it would require synchronizing with all
the factors affecting its value. One of main reasons for migrating to
maxmem=0 approach is to avoid the need of such synchronization.

QubesOS/qubes-issues#4480
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
62bc462a23
tests: default maxmem 2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
b8052f864a
tests: more cases for libvirt xml generation
Related to automatic mem balance enabling/disabling. Check how it behave
in presence of PCI devices, or explicit disabling it.
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
4dc8631010
Use maxmem=0 to disable qmemman, add more automation to it
Use maxmem=0 for disabling dynamic memory balance, instead of cryptic
service.meminfo-writer feature. Under the hood, meminfo-writer service
is also set based on maxmem property (directly in qubesdb, not
vm.features dict).
Having this as a property (not "feature"), allow to have sensible
handling of default value. Specifically, disable it automatically if
otherwise it would crash a VM. This is the case for:
 - domain with PCI devices (PoD is not supported by Xen then)
 - domain without balloon driver and/or meminfo-writer service

The check for the latter is heuristic (assume presence of 'qrexec' also
can indicate balloon driver support), but it is true for currently
supported systems.

This also allows more reliable control of libvirt config: do not set
memory != maxmem, unless qmemman is enabled.

memory != maxmem only makes sense if qmemman for given domain is
enabled.  Besides wasting some domain resources for extra page tables
etc, for HVM domains this is harmful, because maxmem-memory difference
is made of Popupate-on-Demand pool, which - when depleted - will kill
the domain. This means domain without balloon driver will die as soon
as will try to use more than initial memory - but without balloon driver
it sees maxmem memory and doesn't know about the lower limit.

Fixes QubesOS/qubes-issues#4135
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
35a53840f1
vm: send domain-start-failed event also if some device is missing
Checking device presence wasn't covered with try/except that send the
event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
0eab082d85
ext/core-features: make 'template-postinstall' event async
It makes a lot of sense to call long-running operations in that event
handler, including calling back into the VM. Allow that by using
fire_event_async, not just fire_event.

Also, document the event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
d2585aa871
tests/lvm: fix checking lvm pool existence cont.
Commit 15cf593bc5 "tests/lvm: fix checking
lvm pool existence" attempted to fix handling '-' in pool name by using
/dev/VG/LV symlink. But those are not created for thin pools. Change
back to /dev/mapper, but include '-' mangling.

Related QubesOS/qubes-issues#4332
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
f023b3dd6e
backup: fix naming qubes.xml.000 in the archive
Restore old code for calculating subdir within the archive. The new one
had two problems:
 - set '/' for empty input subdir - which caused qubes.xml.000 to be
 named '/qubes.xml.000' (and then converted to '../../qubes.xml.000');
 among other things, this results in the wrong path used for encryption
 passphrase
 - resolved symlinks, which breaks calculating path for any symlinks
 within VM's directory (symlinks there should be treated as normal files
 to be sure that actual content is included in the backup)

This partially reverts 4e49b951ce.

Fixes QubesOS/qubes-issues#4493
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
328697730b
vm: fix deadlock on qrexec timeout handling
vm.kill() will try to get vm.startup_lock, so it can't be called while
holding it already.
Fix this by extracting vm._kill_locked(), which expect the lock to be
already taken by the caller.
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
68dffb6895
api/admin: fix error message when refusing to create template on template
Fixes QubesOS/qubes-issues#4463
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
64f290c9ba
ext/pci: fix error message about missing device
Print human readable device name, instead of "<PCIDevice at ...".

QubesOS/qubes-issues#4461
2018-11-01 22:28:50 +01:00
Marek Marczykowski-Górecki
00ca0459d9
ext/pci: use correct backend domain for getting PCIDevice instance
In practice backend_domain is ignored (all PCI devices belongs to dom0),
but lets fix this anyway.
2018-11-01 22:21:50 +01:00
Marek Marczykowski-Górecki
15cf593bc5
tests/lvm: fix checking lvm pool existence
If pool or group name have '-', it will be mangled as '--' in
/dev/mapper. Use /dev/VG_NAME/LV_NAME symlink instead.

Related QubesOS/qubes-issues#4332
2018-10-30 01:17:00 +01:00
Marek Marczykowski-Górecki
1ae6abdff5
exc: fix QubesMemoryError constructor
QubesVMError require 'vm' argument.
Fixes 2f3a9847 "exc: Make QubesMemoryError inherit from QubesVMError"
2018-10-30 01:14:58 +01:00
Marek Marczykowski-Górecki
b9a18a819c
Merge remote-tracking branch 'origin/pr/239'
* origin/pr/239:
  storage: fix NotImplementedError message for import_data()
  storage/reflink: make resize()/import_volume() more readable
  storage/reflink: unblock import_data() and import_data_end()
2018-10-29 23:00:04 +01:00
Marek Marczykowski-Górecki
fa2429aae4
Merge remote-tracking branch 'origin/pr/237'
* origin/pr/237:
  progress thresold removed as Marek suggested
  Avoid progress events flooding

Fixes QubesOS/qubes-issues#4406
Fixes QubesOS/qubes-issues#3035
2018-10-29 22:54:05 +01:00
Rusty Bird
7e4812a525
storage: fix NotImplementedError message for import_data() 2018-10-29 20:21:42 +00:00
Rusty Bird
73db2751b8
storage/reflink: make resize()/import_volume() more readable 2018-10-29 20:21:41 +00:00
Rusty Bird
425d993769
storage/reflink: unblock import_data() and import_data_end() 2018-10-29 20:21:39 +00:00
Marek Marczykowski-Górecki
f621e8792c
Merge branch 'master' into devel-no-assert 2018-10-29 20:29:53 +01:00
Marek Marczykowski-Górecki
3740e2d48b
api: make enforce() static
It doesn't use 'self'. And pylint complains.
2018-10-29 20:22:35 +01:00
Marek Marczykowski-Górecki
db6094f397
tests/api: adjust for proper exceptions instead of AssertionError 2018-10-29 20:22:10 +01:00
Marek Marczykowski-Górecki
2b5fc6299e
tests/api: do not test non-existing methods
Remove methods not included in specification (or with different
constraints). Keep commented out methods included in spec but not
implemented.
2018-10-29 20:21:36 +01:00
Marek Marczykowski-Górecki
c1c6dc2acd
Use ValueError in PropertyHolder.property_require()
Specifically do not use AssertionError, but also be consistent with
other value verification.
2018-10-29 20:16:41 +01:00
Marek Marczykowski-Górecki
26a553737f
storage/lvm: minor fix for lvs command building
Do not prepend 'sudo' each time - do a copy of array if that's
necessary.
2018-10-29 05:16:23 +01:00
Marek Marczykowski-Górecki
42061cb194
tests: try to collect qvm-open-in-dvm output if no editor window is shown
Try to collect more details about why the test failed. This will help
only if qvm-open-in-dvm exist early. On the other hand, if it hang, or
remote side fails to find the right editor (which results in GUI error
message), this change will not provide any more details.
2018-10-29 01:20:57 +01:00
Marek Marczykowski-Górecki
84c321b923
tests: increase session startup timeout for whonix-ws based VMs
First boot of whonix-ws based VM take extended period of time, because
a lot of files needs to be copied to private volume. This takes even
more time, when verbose logging through console is enabled. Extend the
timeout for that.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
84d3547f09
tests: adjust extra tests loader to work with nose2
Nose loader do not provide loader.loadTestsFromTestCase(), use
loader.loadTestsFromNames() instead.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
fb14f589cb
tests: wait for full user session before doing rest of the test
Clean VM shutdown may timeout if its initiated before full startup, so
make sure the full startup is completed first.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
4742a630f2
tests: use iptables --wait
QubesOS/qubes-issues#3665 affects also tests...
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
2f3a984742
exc: Make QubesMemoryError inherit from QubesVMError
Same as other vm-related errors.
This helps QubesTestCase.cleanup_traceback() cleanup VM reference.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
8be70c9e4d
ext/services: allow for os=Linux feature request from VM
It's weird to set it for Windows, but not Linux.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
e244c192ae
tests: use /bin/uname instead of /bin/hostname as dummy output generator
Use something included in coreutils installed everywhere.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
f13029219b
vm: disable/enable qubes-vm@ service when domain is removed/created
If domain is set to autostart, qubes-vm@ systemd service is used to
start it at boot. Cleanup the service when domain is removed, and
similarly enable the service when domain is created and already have
autostart=True.

Fixes QubesOS/qubes-issues#4014
2018-10-27 16:44:53 +02:00