Commit Graph

1711 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
f4528296b5
vm: allow StandaloneVM to be a DVM template
Fixes QubesOS/qubes-issues#4670
2019-03-01 05:56:57 +01:00
Marek Marczykowski-Górecki
8041b72e42
vm: do not allow setting template_for_dispvms=False if there are any DispVMs
Setting template_for_dispvms=False will at least prevent starting
(already existing) DispVMs based on it. Those should be first removed.
Add also tests for this case.
2019-03-01 05:56:57 +01:00
Marek Marczykowski-Górecki
4a3772bb57
vm: move DVM template specific code into separate mixin
No functional change.
2019-02-27 15:42:27 +01:00
Marek Marczykowski-Górecki
8962452502
Make DispVMs started from a DispVM to use the same DVM template by default
If a specific DVM template is used for given DispVM, make new DispVMs
called from it use the same DVM template (unless explicitly overridden).
This prevent various isolation bypass cases, like using a chain of
DispVMs to access network.
2019-02-18 19:25:26 +01:00
Marek Marczykowski-Górecki
bfb09f567f
Fix saving 'updates-available' flag reported by TemplateBasedVM
Look for the first updateable template up in the template chain, instead
of going just one level up. Especially this applies to
DispVM -> AppVM -> TemplateVM case.
If DispVM reports available updates, 'updates-available'
flag should be set on relevant TemplateVM, not AppVM (*-dvm).

Include test for the new case.

Fixes QubesOS/qubes-issues#3736
2019-01-19 03:25:21 +01:00
Marek Marczykowski-Górecki
18b0fa9db7
tests: improve mock setup in api_misc
Make self.src.template separate mock instance for better control over
results.
2019-01-19 03:25:20 +01:00
Marek Marczykowski-Górecki
a9ec2bb2c3
vm/qubesvm: fix race condition in failed startup handling
Instead of checking if domain is still running/paused, try to kill it
anyway and ignore appropriate exception. Otherwise domain could die
before the check and killing.
2019-01-19 03:25:20 +01:00
Marek Marczykowski-Górecki
4e5a14774a
storage/lvm: fix listing volume revisions when VM name include volume name
some-vm-root is a valid VM name, and in that case it's volume can be
named some-vm-root-private. Do not let it confuse revision listing,
check for unexpected '-' in volume revision number.

The proper solution would be to use different separator, that is not
allowed in VM names. But that would require migration code that is
undesired in the middle of stable release life cycle.

Fixes QubesOS/qubes-issues#4680
2019-01-19 03:25:19 +01:00
Marek Marczykowski-Górecki
dbd85c75e2
Merge branch 'tests-20181223'
* tests-20181223:
  tests: drop expectedFailure from qubes_desktop_run test
  tests: grub in HVM qubes
  tests: update dom0_update for new updates available flag
  tests: regression test LVM listing code
  tests/extra: wrap ProcessWrapper.wait() to be asyncio-aware
  tests: adjust backupcompat for new maxmem handling
2019-01-19 03:24:29 +01:00
Marek Marczykowski-Górecki
ed6acf4a33
Merge remote-tracking branch 'origin/pr/249'
* origin/pr/249:
  qubes/utils: parse_size: Fix string comparison
2019-01-17 22:28:36 +01:00
M. Vefa Bicakci
d934e8114a
qubes/utils: parse_size: Fix string comparison
This commit resolves a bug which causes strings such as "350MiB" to be
rejected by parse_size, due to the fact that parse_size changes the case
of letters in the input string ("350MiB") to uppercase ("350MIB"), but
fails to do the same for the elements of the units conversion table.

The correction is simple: Apply the same case change to the units
table elements before comparison.
2019-01-16 21:18:50 -05:00
Marek Marczykowski-Górecki
186702646e
tests: drop expectedFailure from qubes_desktop_run test 2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
08621e7225
tests: grub in HVM qubes
Test grub in both PV and HVM. This change will also ease adding grub
test for PVH, when supported.
2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
adc3c4daa4
tests: update dom0_update for new updates available flag
Pending updates for dom0 are now signaled the same way as for templates
- with 'updates-available' feature, not a flag file.
2019-01-12 02:41:22 +01:00
Marek Marczykowski-Górecki
4c4e008368
tests: regression test LVM listing code
The revisions listing code mistakes VM name suffix with volume name.

QubesOS/qubes-issues#4680
2019-01-06 04:33:16 +01:00
Marek Marczykowski-Górecki
b08804e7c8
tests/extra: wrap ProcessWrapper.wait() to be asyncio-aware
The user of ExtraTestCase don't need to know anything about asyncio.
vm.run().wait() normally is a coroutine, but provide a wrapper that
handle asyncio.

This fixes FD leak in input proxy tests.
2019-01-06 04:33:16 +01:00
Marek Marczykowski-Górecki
0099aa1037
tests: adjust backupcompat for new maxmem handling
Since 4dc86310 "Use maxmem=0 to disable qmemman, add more automation to
it" meminfo-writer service is not accessible directly. maxmem property
is used to encode memory management instead.
2019-01-06 04:33:16 +01:00
Wojtek Porczyk
17704c8d0a
qubes/ext/gui: set videoram qdb entries
This is for adjusting videoram size. See
QubesOS/qubes-gui-agent-linux@d222e16003
2018-12-20 18:28:40 +01:00
Wojtek Porczyk
ff612a870b qubes/features: check_with_(template_and_)adminvm
- Two new methods: .features.check_with_adminvm() and
  .check_with_template_and_adminvm(). Common code refactored.

- Two new AdminAPI calls to take advantage of the methods:
  - admin.vm.feature.CheckWithAdminVM
  - admin.vm.feature.CheckWithTemplateAndAdminVM

- Features manager moved to separate module in anticipation of features
  on app object in R5.0. The attribute Features.vm renamed to
  Features.subject.

- Documentation, tests.
2018-12-20 18:28:33 +01:00
Marek Marczykowski-Górecki
3728230e3c
Merge branch 'maxmem' 2018-12-09 18:38:21 +01:00
Marek Marczykowski-Górecki
3606010591
Merge branch 'devel-20181205'
* devel-20181205:
  vm/dispvm: fix /qubes-vm-presistence qubesdb entry
  vm/mix/net: prevent setting provides_network=false if qube is still used
  tests: updates-available notification
  tests/network: reduce code duplication
  tests: listen on 'misc' socket too
2018-12-09 18:35:35 +01:00
Marek Marczykowski-Górecki
f66aae220f
vm/dispvm: fix /qubes-vm-presistence qubesdb entry
It should be 'none' for DispVM.

Fixes QubesOS/qubes-issues#4007
2018-12-08 23:13:30 +01:00
AJ Jordan
d4e567cb10
Fix typo 2018-12-06 20:43:39 -05:00
Marek Marczykowski-Górecki
7a6e4b284f
vm/mix/net: prevent setting provides_network=false if qube is still used
Add symmetric check to the one in 'netvm' property.

Fixes QubesOS/qubes-issues#4552
2018-12-06 20:37:50 +01:00
Marek Marczykowski-Górecki
9f6c97596d
tests: updates-available notification
First install test-pkg-1.0, then add test-pkg-1.1 to repo and check if
updates-available flag is set. Then install updates and check if the
flag is cleared.

QubesOS/qubes-issues#2009
2018-12-05 06:19:34 +01:00
Marek Marczykowski-Górecki
cea9f66bc3
tests/network: reduce code duplication 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
2edafacbe4
tests: listen on 'misc' socket too 2018-12-05 05:42:15 +01:00
Marek Marczykowski-Górecki
ccd7e285a1
Remove unused core2migration.py
It is part of backup-restore code, but it was moved to
core-admin-client repo.
2018-12-04 16:12:33 +01:00
Marek Marczykowski-Górecki
162196a1f8
tests: fix lookup domain by name 2018-12-04 16:12:09 +01:00
Marek Marczykowski-Górecki
91c9e49612
Drop unnecessary 'pass' to make pylint happy 2018-12-04 16:06:33 +01:00
Marek Marczykowski-Górecki
eee9964b11
Merge remote-tracking branch 'origin/release4.0' 2018-12-04 16:03:33 +01:00
Marek Marczykowski-Górecki
7d1bcaf64c Introduce management_dispvm property
The new property is meant for management stack (Salt) to set which DVM
template should be used to maintain given VM. Since the DispVM based on
it will be given ultimate control over target VM (qubes.VMShell
service), it should be trusted. The one pointed to by default_dispvm
not necessary is one.

The property defaults to the value from the template (if any), and then
to a global management_dispvm property. By default it is set to None.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-12-03 19:18:26 +01:00
Marek Marczykowski-Górecki
d1e20e5178
Merge remote-tracking branch 'origin/pr/242'
* origin/pr/242:
  Drop async future check with python3 version
  tracis-ci: add support for fc29 dom0
2018-11-24 14:43:43 +01:00
fepitre
313a22d4d6 Drop async future check with python3 version
QubesOS/qubes-issues#4225
2018-11-22 19:34:58 +01:00
Marek Marczykowski-Górecki
087a02c7f4
ext/services: add automatic migration meminfo-writer=False -> maxmem=0
Migrate meminfo-writer=False service setting to maxmem=0 as a method to
disable dynamic memory management. Remove the service from vm.features
dict in the process.

Additionally, translate any attempt to set the service.meminfo-writer
feature to either setting maxmem=0 or resetting it to the default (which
is memory balancing enabled if supported by given domain). This is to at
least partially not break existing tools using service.meminfo-writer as
a way to control dynamic memory management. This code does _not_ support
reading service.meminfo-writer feature state to get the current state of
dynamic memory management, as it would require synchronizing with all
the factors affecting its value. One of main reasons for migrating to
maxmem=0 approach is to avoid the need of such synchronization.

QubesOS/qubes-issues#4480
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
62bc462a23
tests: default maxmem 2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
b8052f864a
tests: more cases for libvirt xml generation
Related to automatic mem balance enabling/disabling. Check how it behave
in presence of PCI devices, or explicit disabling it.
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
4dc8631010
Use maxmem=0 to disable qmemman, add more automation to it
Use maxmem=0 for disabling dynamic memory balance, instead of cryptic
service.meminfo-writer feature. Under the hood, meminfo-writer service
is also set based on maxmem property (directly in qubesdb, not
vm.features dict).
Having this as a property (not "feature"), allow to have sensible
handling of default value. Specifically, disable it automatically if
otherwise it would crash a VM. This is the case for:
 - domain with PCI devices (PoD is not supported by Xen then)
 - domain without balloon driver and/or meminfo-writer service

The check for the latter is heuristic (assume presence of 'qrexec' also
can indicate balloon driver support), but it is true for currently
supported systems.

This also allows more reliable control of libvirt config: do not set
memory != maxmem, unless qmemman is enabled.

memory != maxmem only makes sense if qmemman for given domain is
enabled.  Besides wasting some domain resources for extra page tables
etc, for HVM domains this is harmful, because maxmem-memory difference
is made of Popupate-on-Demand pool, which - when depleted - will kill
the domain. This means domain without balloon driver will die as soon
as will try to use more than initial memory - but without balloon driver
it sees maxmem memory and doesn't know about the lower limit.

Fixes QubesOS/qubes-issues#4135
2018-11-21 02:13:25 +01:00
Marek Marczykowski-Górecki
35a53840f1
vm: send domain-start-failed event also if some device is missing
Checking device presence wasn't covered with try/except that send the
event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
0eab082d85
ext/core-features: make 'template-postinstall' event async
It makes a lot of sense to call long-running operations in that event
handler, including calling back into the VM. Allow that by using
fire_event_async, not just fire_event.

Also, document the event.
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
d2585aa871
tests/lvm: fix checking lvm pool existence cont.
Commit 15cf593bc5 "tests/lvm: fix checking
lvm pool existence" attempted to fix handling '-' in pool name by using
/dev/VG/LV symlink. But those are not created for thin pools. Change
back to /dev/mapper, but include '-' mangling.

Related QubesOS/qubes-issues#4332
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
f023b3dd6e
backup: fix naming qubes.xml.000 in the archive
Restore old code for calculating subdir within the archive. The new one
had two problems:
 - set '/' for empty input subdir - which caused qubes.xml.000 to be
 named '/qubes.xml.000' (and then converted to '../../qubes.xml.000');
 among other things, this results in the wrong path used for encryption
 passphrase
 - resolved symlinks, which breaks calculating path for any symlinks
 within VM's directory (symlinks there should be treated as normal files
 to be sure that actual content is included in the backup)

This partially reverts 4e49b951ce.

Fixes QubesOS/qubes-issues#4493
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
328697730b
vm: fix deadlock on qrexec timeout handling
vm.kill() will try to get vm.startup_lock, so it can't be called while
holding it already.
Fix this by extracting vm._kill_locked(), which expect the lock to be
already taken by the caller.
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
68dffb6895
api/admin: fix error message when refusing to create template on template
Fixes QubesOS/qubes-issues#4463
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
64f290c9ba
ext/pci: fix error message about missing device
Print human readable device name, instead of "<PCIDevice at ...".

QubesOS/qubes-issues#4461
2018-11-01 22:28:50 +01:00
Marek Marczykowski-Górecki
00ca0459d9
ext/pci: use correct backend domain for getting PCIDevice instance
In practice backend_domain is ignored (all PCI devices belongs to dom0),
but lets fix this anyway.
2018-11-01 22:21:50 +01:00
Marek Marczykowski-Górecki
15cf593bc5
tests/lvm: fix checking lvm pool existence
If pool or group name have '-', it will be mangled as '--' in
/dev/mapper. Use /dev/VG_NAME/LV_NAME symlink instead.

Related QubesOS/qubes-issues#4332
2018-10-30 01:17:00 +01:00
Marek Marczykowski-Górecki
1ae6abdff5
exc: fix QubesMemoryError constructor
QubesVMError require 'vm' argument.
Fixes 2f3a9847 "exc: Make QubesMemoryError inherit from QubesVMError"
2018-10-30 01:14:58 +01:00
Marek Marczykowski-Górecki
b9a18a819c
Merge remote-tracking branch 'origin/pr/239'
* origin/pr/239:
  storage: fix NotImplementedError message for import_data()
  storage/reflink: make resize()/import_volume() more readable
  storage/reflink: unblock import_data() and import_data_end()
2018-10-29 23:00:04 +01:00
Marek Marczykowski-Górecki
fa2429aae4
Merge remote-tracking branch 'origin/pr/237'
* origin/pr/237:
  progress thresold removed as Marek suggested
  Avoid progress events flooding

Fixes QubesOS/qubes-issues#4406
Fixes QubesOS/qubes-issues#3035
2018-10-29 22:54:05 +01:00