Commit Graph

2000 Commits

Author SHA1 Message Date
Frédéric Pierret (fepitre)
660c224914
Allow AudioVM to be ran after any attached qubes 2020-03-08 17:05:34 +01:00
Frédéric Pierret (fepitre)
edfb251413
Allow GuiVM to be ran after any attached qubes 2020-03-08 17:05:34 +01:00
Frédéric Pierret (fepitre)
3f4ee9f8d9
xid: ensure vm is not running 2020-03-08 17:05:34 +01:00
Frédéric Pierret (fepitre)
820500a367
tests: fix missing default audiovm and guivm tags 2020-03-08 17:05:33 +01:00
Frédéric Pierret (fepitre)
a1752ef265
gui, audio: better handling of start/stop guivm/audiovm 2020-03-08 17:05:33 +01:00
Frédéric Pierret (fepitre)
8f67334c4f
gui, audio: ensure guivm and audiovm tag are set 2020-03-08 17:05:33 +01:00
Frédéric Pierret (fepitre)
795ff1233a
Support for AudioVM 2020-03-08 17:05:33 +01:00
Marek Marczykowski-Górecki
b569f5a2b0
tests/firewall: update future time to be really in the future
Mar 4 2020 is no longer future, move it much more into the future.
2020-03-08 16:35:39 +01:00
Frédéric Pierret (fepitre)
bdc3c6588b
services: fixes from Marek's comments 2020-03-08 09:51:30 +01:00
Frédéric Pierret (fepitre)
31c098d1bd
config: fix mistake in path for services 2020-03-08 09:47:09 +01:00
Marek Marczykowski-Górecki
283d251387
Merge remote-tracking branch 'origin/pr/319'
* origin/pr/319:
  gui: set xkb_layout manually
  tests: handle legacy layout
  gui: no check in subprocess.run
  gui: handle legacy keymap setting /qubes-keyboard
2020-03-08 01:29:21 +01:00
Marek Marczykowski-Górecki
6874c7fece
tests/salt: don't depend on initial tags set
Some extensions may add tags at VM creation (guivm-*, audiovm-*). Take
this into account when calculating expected tags.
2020-03-01 22:36:54 +01:00
Marek Marczykowski-Górecki
ec88796129
tests: make audio tests less racy
Wait specifically for pulseaudio to start in the VM, instead of just
waiting few seconds.
Also, improve failure message to distinguish total lack of audio from
just missing few samples.
2020-03-01 22:36:54 +01:00
Marek Marczykowski-Górecki
34e6c2ff34
Merge remote-tracking branch 'origin/pr/323'
* origin/pr/323:
  Add guivm to internal.GetSystemInfo
2020-03-01 03:51:24 +01:00
Paweł Marczewski
941b7f16a0
Add guivm to internal.GetSystemInfo
Needed to move qrexec-policy-agent out to a separate GuiVM.
2020-02-27 18:20:48 +01:00
Frédéric Pierret (fepitre)
a67b8f35ba
gui: set xkb_layout manually 2020-02-25 11:34:02 +01:00
Frédéric Pierret (fepitre)
9a6ff177ce
tests: add/remove services in dom0 2020-02-20 23:07:49 +01:00
Frédéric Pierret (fepitre)
0b8e5400a3
config: specify dom0 services path 2020-02-20 15:56:48 +01:00
Frédéric Pierret (fepitre)
a7e7166f7a
services: handle dom0 write permission errors 2020-02-20 15:55:04 +01:00
Frédéric Pierret (fepitre)
d0a8b49cc9
services: handle /var/run/qubes/'SERVICE NAME' for dom0 2020-02-20 15:55:04 +01:00
Frédéric Pierret (fepitre)
6f2f57caea
services: make PEP8 happier 2020-02-20 15:55:03 +01:00
Marek Marczykowski-Górecki
2f4b42e5cd
Merge remote-tracking branch 'origin/pr/318'
* origin/pr/318:
  Pass network parameters to linux-stubdom

Fixes QubesOS/qubes-issues#5022
2020-02-17 03:56:45 +01:00
Rusty Bird
e8a7a6c82c
qubes/config: reindent varlibqubes pool; add a blank line 2020-02-15 18:33:18 +00:00
Rusty Bird
d527e01136
Remove more core2 code 2020-02-15 17:48:32 +00:00
Rusty Bird
8e581108b7
Remove pylint pragma working around very old false positive 2020-02-15 17:33:35 +00:00
Rusty Bird
1f71a06aed
Remove python2 compat __future__ imports 2020-02-15 17:33:34 +00:00
Frédéric Pierret (fepitre)
65238c187f
tests: handle legacy layout 2020-02-13 22:46:52 +01:00
Frédéric Pierret (fepitre)
2e1290eec1
gui: no check in subprocess.run 2020-02-13 22:31:33 +01:00
Marek Marczykowski-Górecki
86881afeef
storage/lvm: don't try to resize non-existing lvm volumes
Volume with save_on_stop=False and snap_on_start=False exists only when
VM is running and this is already handled under is_dirty() condition.
For non-running VMs, skip LVM resize step and only update internal size
attribute.

Fixes QubesOS/qubes-issues#5645
2020-02-12 21:57:34 +01:00
Marek Marczykowski-Górecki
6b474e39f6
storage/lvm: update volume.size attribute on resize
When volume is resized, underlying LVM is adjusted if exists. For
persistent volumes it is enough, as generally the driver prefers
underlying LVM size than internal attribute. But volatile volume is
removed at vm shutdown, so that value is lost.

Fix it by updating Volume.size attribute too, to be updated in qubes.xml
later too.

Fixes QubesOS/qubes-issues#5645
2020-02-12 21:44:16 +01:00
Frédéric Pierret (fepitre)
dbacd07b43
gui: handle legacy keymap setting /qubes-keyboard 2020-02-10 14:42:30 +01:00
Marek Marczykowski-Górecki
c332de9653
Merge remote-tracking branch 'origin/pr/312'
* origin/pr/312:
  Added handling for 'force' option in vm shutdown
2020-02-06 23:31:43 +01:00
Marek Marczykowski-Górecki
f59fd3d474
Merge remote-tracking branch 'origin/pr/316'
* origin/pr/316:
  storage/reflink: trivial style tweaks
  storage/reflink: comment on _get_size() use in start()
  storage/reflink: don't bother using _get_size() in create()
  storage/reflink: delete all images at beginning of create()
  storage/reflink: get VM dir from less arbitrary-looking path
  storage/reflink: factor out _remove_all_images()
  storage/reflink: _remove_incomplete_{files -> images}()
  storage/reflink: omit redundant comment
2020-02-06 23:29:08 +01:00
Rusty Bird
ada27ee431
storage/reflink: trivial style tweaks
Avoid early return for short and not deeply nested functions.
2020-02-05 17:26:44 +00:00
Rusty Bird
21971d6d0a
storage/reflink: comment on _get_size() use in start() 2020-02-05 17:26:43 +00:00
Marta Marczykowska-Górecka
284fbe1d44
Added handling for 'force' option in vm shutdown
To be more precise: just allowed using it, as the option was
actually implemented previously.

references QubesOS/qubes-issues#5591
2020-01-29 22:12:49 +01:00
Pawel Marczewski
1d1cc10211
Pass network parameters to linux-stubdom
See QubesOS/qubes-issues#5022.
2020-01-29 09:45:05 +01:00
marmot1791
d7b90f92e0 Correct comments in to match current code behavior 2020-01-28 23:02:43 +00:00
Rusty Bird
749ce477df
storage/reflink: don't bother using _get_size() in create()
Only the nominal size is available at this point.
2020-01-28 13:40:15 +00:00
Rusty Bird
6659ed8d39
storage/reflink: delete all images at beginning of create()
Ensure that there are no leftover image files for the volume, e.g. from
an unsuccessful removal of a previous incarnation of this vid, or from
an messily restored pool filesystem backup. We don't want to preserve
any stale data (revisions) or metadata (size) in the new incarnation.
2020-01-28 13:40:14 +00:00
Rusty Bird
56f6a6ef65
storage/reflink: get VM dir from less arbitrary-looking path 2020-01-28 13:40:13 +00:00
Rusty Bird
12d882b355
storage/reflink: factor out _remove_all_images() 2020-01-28 13:40:11 +00:00
Rusty Bird
8f4c90c37a
storage/reflink: _remove_incomplete_{files -> images}() 2020-01-28 13:40:10 +00:00
Rusty Bird
9c9d71c069
storage/reflink: omit redundant comment
The is_dirty() one-liner is defined right above.
2020-01-28 13:40:09 +00:00
Pawel Marczewski
e8baf2a157
Remove leftovers from QubesVM.run_for_stdio() refactoring
We no longer use qubes.VMShell there.
2020-01-24 16:59:42 +01:00
Pawel Marczewski
08d83fb241
Support qubes.VMExec call
The feature is advertised by core-agent so that it can be used
instead of VMShell.

See QubesOS/qubes-issues#4850.
2020-01-24 16:57:13 +01:00
Marek Marczykowski-Górecki
0a66a0c7dd
Merge remote-tracking branch 'origin/pr/313'
* origin/pr/313:
  Fix overlapping block device names
2020-01-24 01:43:33 +01:00
Marek Marczykowski-Górecki
869f963335
Merge remote-tracking branch 'origin/pr/311'
* origin/pr/311:
  Add a test for loading volume config from XML
  Fix ThinVolume.size initialization from string
2020-01-24 01:38:16 +01:00
Marek Marczykowski-Górecki
edf5977b23
Merge remote-tracking branch 'origin/pr/309'
* origin/pr/309:
  import: check exact size of copied data
  Implement new admin.vm.ImportWithSize API call

Fixes QubesOS/qubes-issues#5239
2020-01-24 01:35:00 +01:00
Pawel Marczewski
fe72fe1681
Fix overlapping block device names
This was caused by a change in Jinja template engine that breaks
assignments like {% set i = i + 1 %} in a loop.

Jinja 2.10 introduces a "namespace" object for this use case, but
unfortunately dom0-fc25 uses 2.8.1.

See:

https://github.com/pallets/jinja/issues/641
https://github.com/pallets/jinja/pull/684

Fixes QubesOS/qubes-issues#5551.
2020-01-23 11:27:05 +01:00
Pawel Marczewski
66abc69a33
Add a test for loading volume config from XML 2020-01-23 10:42:26 +01:00
Pawel Marczewski
49f2c1b78d
Fix ThinVolume.size initialization from string
The size config parameter might be a string coming from XML.
The Volume base class handles the conversion to integer already.

Fixes QubesOS/qubes-issues#5219.
2020-01-23 10:39:47 +01:00
Pawel Marczewski
e9b97e42b1
import: check exact size of copied data
The import will error out if there is not enough data, or too
much data provided.
2020-01-23 09:48:58 +01:00
Pawel Marczewski
63ac952803
Implement new admin.vm.ImportWithSize API call
This should allow importing a volume and changing the size at the
same time, without performing the resize operation on original
volume first.

The internal API has been renamed to internal.vm.volume.ImportBegin
to avoid confusion, and for symmetry with ImportEnd.

See QubesOS/qubes-issues#5239.
2020-01-23 09:47:22 +01:00
Marek Marczykowski-Górecki
88037ff289
Merge remote-tracking branch 'origin/pr/310'
* origin/pr/310:
  storage/reflink: fix comment
  storage/reflink: bail out early on most FICLONE errnos
  storage/reflink: pool.setup_check -> pool._setup_check
2020-01-23 04:50:14 +01:00
Marek Marczykowski-Górecki
29f84d5105
Merge remote-tracking branch 'origin/pr/308'
* origin/pr/308:
  Move devices check to on_domain_pre_deleted
  Prevent removing VM if it provides devices in persistent mode
2020-01-23 04:32:39 +01:00
Pawel Marczewski
f1ff6c26d8
Move devices check to on_domain_pre_deleted 2020-01-21 15:35:30 +01:00
Rusty Bird
d54e4b0c6e
storage/reflink: fix comment 2020-01-17 16:45:29 +00:00
Rusty Bird
ba662d2819
storage/reflink: bail out early on most FICLONE errnos
Don't fall back on 'cp' if the FICLONE ioctl gives an errno that's not
plausibly reflink specific, because in such a case any fallback could
theoretically mask real but intermittent system/storage errors.

Looking through ioctl_ficlone(2) and the kernel source, it should be
sufficient to do the fallback only on EBADF/EINVAL/EOPNOTSUPP/EXDEV.
(EISDIR/ETXTBSY don't apply to this storage driver, which will never
legitimately attempt to reflink a directory or an active - in the
storage domain - swap file.)
2020-01-17 15:56:51 +00:00
Rusty Bird
90f25890cf
storage/reflink: pool.setup_check -> pool._setup_check 2020-01-17 15:56:50 +00:00
Marek Marczykowski-Górecki
8dda66d693
tests: fix key generation for dom0 updates tests
gpg2 requires explicit `%no-protection`, otherwise tries to
interactively (through gpg-agent) prompt for the passphrase.
2020-01-17 04:50:31 +01:00
Pawel Marczewski
b09a137b26
Prevent removing VM if it provides devices in persistent mode
Fixes QubesOS/qubes-issues#5136.
2020-01-16 11:13:07 +01:00
Marek Marczykowski-Górecki
309dd11b1d
Merge remote-tracking branch 'origin/pr/307'
* origin/pr/307:
  Implement missing get_cputime() in AdminVM
  Add admin.vm.CurrentState method
2020-01-16 04:12:42 +01:00
Marek Marczykowski-Górecki
161f139f98
Merge remote-tracking branch 'origin/pr/305'
* origin/pr/305:
  Test file copy hang when target machine qrexec is disabled
2020-01-16 04:05:09 +01:00
Marek Marczykowski-Górecki
0c08305f1a
Merge remote-tracking branch 'origin/pr/303'
* origin/pr/303:
  Update tests after adding /connected-ips
  Also reload /connected-ips on firewall change / domain spawn
  Also store /connected-ips6 for machines that have IPv6 addresses
  Don't try to write to qubesdb of an offline VM
  Maintain a list of connected machine IPs in qubesdb
2020-01-16 04:03:44 +01:00
Marek Marczykowski-Górecki
e6aa35fcdf
Merge branch 'tests-leaks'
* tests-leaks:
  ext/pci: handle 'qubes-close' event
  tests: allow extensions to cleanup objects references
2020-01-16 04:00:59 +01:00
Marek Marczykowski-Górecki
51adb434d1
tests: FileVolume.resize()
QubesOS/qubes-issues#5518
2020-01-16 01:38:14 +01:00
Pawel Marczewski
44dc35e972
Implement missing get_cputime() in AdminVM 2020-01-15 16:37:57 +01:00
Pawel Marczewski
62735d9696
Add admin.vm.CurrentState method
For all the transient values that should not be treated as
real properties. To be used in core-admin-client.
2020-01-15 16:06:08 +01:00
Pawel Marczewski
e69bdc5efa
Test file copy hang when target machine qrexec is disabled
See QubesOS/qubes-issues#5347.

Fix in QubesOS/qubes-core-qrexec#15.
2020-01-14 15:07:42 +01:00
Marek Marczykowski-Górecki
74e956e1f1
ext/pci: handle 'qubes-close' event
Move cache cleanup to 'qubes-close' event handler, instead of doing it
specifically in tests.
2020-01-11 03:54:30 +01:00
Marek Marczykowski-Górecki
a89d3f0cae
tests: allow extensions to cleanup objects references
Extension objects are singletons and normally do not require any special
cleanup. But in case of tests, we try to remove all the qubes objects
between tests and the cache in usb extension makes it hard.
Add a 'qubes-close' event that extensions can handle to remove extra
references stored in extension objects themselves.
2020-01-11 01:22:30 +01:00
Pawel Marczewski
70c862fe73
Update tests after adding /connected-ips 2020-01-09 16:41:14 +01:00
Pawel Marczewski
4d8fd0253b
Also reload /connected-ips on firewall change / domain spawn
There are scenarios where it turns out to be necessary, such as
enabling IPv6 on a machine causing it to acquire an IPv6 address.
2020-01-09 16:25:25 +01:00
Pawel Marczewski
40f372ee5f
Also store /connected-ips6 for machines that have IPv6 addresses 2020-01-09 16:18:07 +01:00
Pawel Marczewski
6bf230cf4d
Don't try to write to qubesdb of an offline VM 2020-01-09 12:37:43 +01:00
Pawel Marczewski
7597a50b20
Maintain a list of connected machine IPs in qubesdb
Necessary for anti-spoofing, see QubesOS/qubes-issues#5540.
2020-01-09 12:23:04 +01:00
Marek Marczykowski-Górecki
d181c0f354
storage/file: fix resize
Fixes QubesOS/qubes-issues#5518
2019-12-14 15:48:49 +01:00
Marta Marczykowska-Górecka
35fa733a67
Fixed Exceptions inferiting from KeyError
Due to strangeness of KeyError (it overrrides __str__ method) in some
cases exceptions received superflous quotation marks when inheriting
from it.

fixes QubesOS/qubes-issues#5106
2019-12-09 21:02:24 +01:00
Marek Marczykowski-Górecki
7cb46e6929
Merge remote-tracking branch 'origin/pr/298'
* origin/pr/298:
  tests/network: let xl devd bring the interfaces up
  tests/network: improve error reporting
  api/admin: implement *.property.GetAll methods
  Avoid resetting clocksync service of just enabled clockvm
  doc/tests: extend qubes-specific quirks in tests
  tests: add include and exclude lists for extra tests loader
2019-12-07 05:04:42 +01:00
Marek Marczykowski-Górecki
8965e9a8e4
tests/network: let xl devd bring the interfaces up
xl devd doesn't use startup notify, so when the service is started
(according to systemd) it may still be initializing interfaces. Add a
little sleep for that.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
ffa1a40e6e
tests/network: improve error reporting
Include stdout/stderr of failed command during netvm setup.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
10f99e5c4a
api/admin: implement *.property.GetAll methods
Allow getting all the VM properties with one call. This greatly improve
performance of an applications retrieving many/all of them (qvm-ls,
qubes manager etc)

QubesOS/qubes-issues#5415
Fixes QubesOS/qubes-issues#3293
2019-12-05 23:31:14 +01:00
Rusty Bird
87081d6ee3
storage/reflink: _cleanup() -> _remove_incomplete_files()
"cleanup" sounds related to the concept of a volume being "dirty" - but
it's unrelated. Rename it for clarity.
2019-12-03 18:21:55 +00:00
Rusty Bird
d7478d128b
storage/reflink: document hardcoded sizeof(int) for FICLONE
One alternative would look like

    import ctypes
    sizeof_int = ctypes.sizeof(ctypes.c_int)
    FICLONE = (1073741824 % 256**sizeof_int) | 37897 | (sizeof_int << 16)

but, even if the above really(?) is a 100% correct Python port of

    $ echo FICLONE | cpp -include linux/fs.h | tail -n 1

it still seems more likely that the ctypes package is somehow buggy
somewhere than for Qubes storage to run on an exotic architecture with
non 32 bit ints (in the foreseeable future).

So just document the baked in assumption.
2019-12-03 18:21:54 +00:00
Rusty Bird
3f0286220c
storage/reflink: simplify _replace_file() comment 2019-12-03 18:21:52 +00:00
Rusty Bird
9d5deffb13
storage/reflink: open in binary mode for loopdev resize ioctl
The default (= text) mode for a loop device which contains a VM image
looked weird, even though it didn't make a difference here because the
dev_io object was never actually read from.
2019-12-03 18:21:51 +00:00
Rusty Bird
4cd9e42416
storage/reflink: use a conditional expression 2019-12-03 18:21:50 +00:00
Marek Marczykowski-Górecki
5d77cf2298
Avoid resetting clocksync service of just enabled clockvm
When setting global clockvm property, 'clocksync' service is
automatically added to the new value and then removed from the old one.
But if the new value is the same as the old one, the service gets
removed from the just set new value.
Check for this case explicitly.

Fixes QubesOS/qubes-issues#4939
2019-11-30 05:20:08 +01:00
Marek Marczykowski-Górecki
7c18b187de
tests: add include and exclude lists for extra tests loader
'extra' tests run is getting ridiculously long. Allow splitting it into
several jobs. Since this appears as just one class from the test loader
perspective, implement it as environment variables:
 - QUBES_TEST_EXTRA_INCLUDE - load just selected tests
 - QUBES_TEST_EXTRA_EXCLUDE - skip selected tests (to select "the rest"
   tests)
2019-11-30 04:35:18 +01:00
Marek Marczykowski-Górecki
fd8e89c546
Merge remote-tracking branch 'origin/pr/297'
* origin/pr/297:
  doc: remove useless _static generated by sphinx
  cleanup-dispvms: fix python shebang
  spec: fix missing dependency
  Fix Sphinx 2 new API for Fedora 31+
  doc: Make PEP8 happier
  qmemmand: separate SystemState init xc and xs to a 'init' method
  doc: drop moved elsewhere components
2019-11-27 02:12:37 +01:00
Frédéric Pierret (fepitre)
be1c193e3c
Fix Sphinx 2 new API for Fedora 31+ 2019-11-22 21:39:31 +01:00
Frédéric Pierret (fepitre)
b4177cf7d2
doc: Make PEP8 happier 2019-11-22 21:39:30 +01:00
Frédéric Pierret (fepitre)
0d20639cc1
qmemmand: separate SystemState init xc and xs to a 'init' method
It prevents any connection to xen when initiallizing the class
notably in doc build generation.
2019-11-22 21:38:27 +01:00
Marek Marczykowski-Górecki
6c7af109e5
ext/block: prefer connecting cdrom as xvdd
Only first 4 disks can be emulated as IDE disks by QEMU. Specifically,
CDROM must be one of those first 4 disks, otherwise it will be
ignored. This is especially important if one wants to boot the VM from
that CDROM.
Since xvdd normally is a kernel-related volume (boot image, modules) it
makes perfect sense to re-use it for CDROM. It is either set for kernel
volume (in which case, VM should boot from it and not the CDROM), or
(possibly bootable) CDROM.

This needs to be done in two places:
 - BlockExtension for dynamic attach
 - libvirt xen.xml - for before-boot attach

In theory the latter would be enough, but it would be quite confusing
that device will get different options depending on when it's attached
(in addition to whether the kernel is set - introduced here).

This all also means, xvdd not always is a "system disk". Adjust listing
connected disks accordingly.
2019-11-19 14:03:21 +01:00
Marek Marczykowski-Górecki
9bf0cce11e
tests: extend mock objects in QubesVM tests
- allow TestQubesDB to be populated with initial data
- support list() method
- allow to register pre-created VM instance (useful for AdminVM, which
        don't accept setting qid)
2019-11-18 23:45:02 +01:00
Frédéric Pierret (fepitre)
e092b40350
tests: make PEP8 happier 2019-11-17 01:54:27 +01:00
Frédéric Pierret (fepitre)
3a6ed2d0cd
tests: fix test with not None default_guivm 2019-11-17 01:48:20 +01:00
Frédéric Pierret (fepitre)
85edf511cb
default_guivm: set to dom0 due to migration R4.0->R4.1 problems 2019-11-16 13:04:42 +01:00
Marta Marczykowska-Górecka
783832adde
Correct inconsistent behavior on unavailable usage data
fixes QubesOS/qubes-issues#5463
2019-11-15 20:01:43 +01:00