Commit Graph

812 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
6874c7fece
tests/salt: don't depend on initial tags set
Some extensions may add tags at VM creation (guivm-*, audiovm-*). Take
this into account when calculating expected tags.
2020-03-01 22:36:54 +01:00
Marek Marczykowski-Górecki
ec88796129
tests: make audio tests less racy
Wait specifically for pulseaudio to start in the VM, instead of just
waiting few seconds.
Also, improve failure message to distinguish total lack of audio from
just missing few samples.
2020-03-01 22:36:54 +01:00
Marek Marczykowski-Górecki
34e6c2ff34
Merge remote-tracking branch 'origin/pr/323'
* origin/pr/323:
  Add guivm to internal.GetSystemInfo
2020-03-01 03:51:24 +01:00
Paweł Marczewski
941b7f16a0
Add guivm to internal.GetSystemInfo
Needed to move qrexec-policy-agent out to a separate GuiVM.
2020-02-27 18:20:48 +01:00
Frédéric Pierret (fepitre)
a67b8f35ba
gui: set xkb_layout manually 2020-02-25 11:34:02 +01:00
Frédéric Pierret (fepitre)
9a6ff177ce
tests: add/remove services in dom0 2020-02-20 23:07:49 +01:00
Marek Marczykowski-Górecki
2f4b42e5cd
Merge remote-tracking branch 'origin/pr/318'
* origin/pr/318:
  Pass network parameters to linux-stubdom

Fixes QubesOS/qubes-issues#5022
2020-02-17 03:56:45 +01:00
Rusty Bird
d527e01136
Remove more core2 code 2020-02-15 17:48:32 +00:00
Frédéric Pierret (fepitre)
65238c187f
tests: handle legacy layout 2020-02-13 22:46:52 +01:00
Marek Marczykowski-Górecki
c332de9653
Merge remote-tracking branch 'origin/pr/312'
* origin/pr/312:
  Added handling for 'force' option in vm shutdown
2020-02-06 23:31:43 +01:00
Marta Marczykowska-Górecka
284fbe1d44
Added handling for 'force' option in vm shutdown
To be more precise: just allowed using it, as the option was
actually implemented previously.

references QubesOS/qubes-issues#5591
2020-01-29 22:12:49 +01:00
Pawel Marczewski
1d1cc10211
Pass network parameters to linux-stubdom
See QubesOS/qubes-issues#5022.
2020-01-29 09:45:05 +01:00
Pawel Marczewski
08d83fb241
Support qubes.VMExec call
The feature is advertised by core-agent so that it can be used
instead of VMShell.

See QubesOS/qubes-issues#4850.
2020-01-24 16:57:13 +01:00
Marek Marczykowski-Górecki
0a66a0c7dd
Merge remote-tracking branch 'origin/pr/313'
* origin/pr/313:
  Fix overlapping block device names
2020-01-24 01:43:33 +01:00
Marek Marczykowski-Górecki
869f963335
Merge remote-tracking branch 'origin/pr/311'
* origin/pr/311:
  Add a test for loading volume config from XML
  Fix ThinVolume.size initialization from string
2020-01-24 01:38:16 +01:00
Marek Marczykowski-Górecki
edf5977b23
Merge remote-tracking branch 'origin/pr/309'
* origin/pr/309:
  import: check exact size of copied data
  Implement new admin.vm.ImportWithSize API call

Fixes QubesOS/qubes-issues#5239
2020-01-24 01:35:00 +01:00
Pawel Marczewski
fe72fe1681
Fix overlapping block device names
This was caused by a change in Jinja template engine that breaks
assignments like {% set i = i + 1 %} in a loop.

Jinja 2.10 introduces a "namespace" object for this use case, but
unfortunately dom0-fc25 uses 2.8.1.

See:

https://github.com/pallets/jinja/issues/641
https://github.com/pallets/jinja/pull/684

Fixes QubesOS/qubes-issues#5551.
2020-01-23 11:27:05 +01:00
Pawel Marczewski
66abc69a33
Add a test for loading volume config from XML 2020-01-23 10:42:26 +01:00
Pawel Marczewski
e9b97e42b1
import: check exact size of copied data
The import will error out if there is not enough data, or too
much data provided.
2020-01-23 09:48:58 +01:00
Pawel Marczewski
63ac952803
Implement new admin.vm.ImportWithSize API call
This should allow importing a volume and changing the size at the
same time, without performing the resize operation on original
volume first.

The internal API has been renamed to internal.vm.volume.ImportBegin
to avoid confusion, and for symmetry with ImportEnd.

See QubesOS/qubes-issues#5239.
2020-01-23 09:47:22 +01:00
Marek Marczykowski-Górecki
29f84d5105
Merge remote-tracking branch 'origin/pr/308'
* origin/pr/308:
  Move devices check to on_domain_pre_deleted
  Prevent removing VM if it provides devices in persistent mode
2020-01-23 04:32:39 +01:00
Pawel Marczewski
f1ff6c26d8
Move devices check to on_domain_pre_deleted 2020-01-21 15:35:30 +01:00
Marek Marczykowski-Górecki
8dda66d693
tests: fix key generation for dom0 updates tests
gpg2 requires explicit `%no-protection`, otherwise tries to
interactively (through gpg-agent) prompt for the passphrase.
2020-01-17 04:50:31 +01:00
Pawel Marczewski
b09a137b26
Prevent removing VM if it provides devices in persistent mode
Fixes QubesOS/qubes-issues#5136.
2020-01-16 11:13:07 +01:00
Marek Marczykowski-Górecki
309dd11b1d
Merge remote-tracking branch 'origin/pr/307'
* origin/pr/307:
  Implement missing get_cputime() in AdminVM
  Add admin.vm.CurrentState method
2020-01-16 04:12:42 +01:00
Marek Marczykowski-Górecki
161f139f98
Merge remote-tracking branch 'origin/pr/305'
* origin/pr/305:
  Test file copy hang when target machine qrexec is disabled
2020-01-16 04:05:09 +01:00
Marek Marczykowski-Górecki
0c08305f1a
Merge remote-tracking branch 'origin/pr/303'
* origin/pr/303:
  Update tests after adding /connected-ips
  Also reload /connected-ips on firewall change / domain spawn
  Also store /connected-ips6 for machines that have IPv6 addresses
  Don't try to write to qubesdb of an offline VM
  Maintain a list of connected machine IPs in qubesdb
2020-01-16 04:03:44 +01:00
Marek Marczykowski-Górecki
e6aa35fcdf
Merge branch 'tests-leaks'
* tests-leaks:
  ext/pci: handle 'qubes-close' event
  tests: allow extensions to cleanup objects references
2020-01-16 04:00:59 +01:00
Marek Marczykowski-Górecki
51adb434d1
tests: FileVolume.resize()
QubesOS/qubes-issues#5518
2020-01-16 01:38:14 +01:00
Pawel Marczewski
62735d9696
Add admin.vm.CurrentState method
For all the transient values that should not be treated as
real properties. To be used in core-admin-client.
2020-01-15 16:06:08 +01:00
Pawel Marczewski
e69bdc5efa
Test file copy hang when target machine qrexec is disabled
See QubesOS/qubes-issues#5347.

Fix in QubesOS/qubes-core-qrexec#15.
2020-01-14 15:07:42 +01:00
Marek Marczykowski-Górecki
74e956e1f1
ext/pci: handle 'qubes-close' event
Move cache cleanup to 'qubes-close' event handler, instead of doing it
specifically in tests.
2020-01-11 03:54:30 +01:00
Marek Marczykowski-Górecki
a89d3f0cae
tests: allow extensions to cleanup objects references
Extension objects are singletons and normally do not require any special
cleanup. But in case of tests, we try to remove all the qubes objects
between tests and the cache in usb extension makes it hard.
Add a 'qubes-close' event that extensions can handle to remove extra
references stored in extension objects themselves.
2020-01-11 01:22:30 +01:00
Pawel Marczewski
70c862fe73
Update tests after adding /connected-ips 2020-01-09 16:41:14 +01:00
Marek Marczykowski-Górecki
8965e9a8e4
tests/network: let xl devd bring the interfaces up
xl devd doesn't use startup notify, so when the service is started
(according to systemd) it may still be initializing interfaces. Add a
little sleep for that.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
ffa1a40e6e
tests/network: improve error reporting
Include stdout/stderr of failed command during netvm setup.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
10f99e5c4a
api/admin: implement *.property.GetAll methods
Allow getting all the VM properties with one call. This greatly improve
performance of an applications retrieving many/all of them (qvm-ls,
qubes manager etc)

QubesOS/qubes-issues#5415
Fixes QubesOS/qubes-issues#3293
2019-12-05 23:31:14 +01:00
Marek Marczykowski-Górecki
7c18b187de
tests: add include and exclude lists for extra tests loader
'extra' tests run is getting ridiculously long. Allow splitting it into
several jobs. Since this appears as just one class from the test loader
perspective, implement it as environment variables:
 - QUBES_TEST_EXTRA_INCLUDE - load just selected tests
 - QUBES_TEST_EXTRA_EXCLUDE - skip selected tests (to select "the rest"
   tests)
2019-11-30 04:35:18 +01:00
Marek Marczykowski-Górecki
6c7af109e5
ext/block: prefer connecting cdrom as xvdd
Only first 4 disks can be emulated as IDE disks by QEMU. Specifically,
CDROM must be one of those first 4 disks, otherwise it will be
ignored. This is especially important if one wants to boot the VM from
that CDROM.
Since xvdd normally is a kernel-related volume (boot image, modules) it
makes perfect sense to re-use it for CDROM. It is either set for kernel
volume (in which case, VM should boot from it and not the CDROM), or
(possibly bootable) CDROM.

This needs to be done in two places:
 - BlockExtension for dynamic attach
 - libvirt xen.xml - for before-boot attach

In theory the latter would be enough, but it would be quite confusing
that device will get different options depending on when it's attached
(in addition to whether the kernel is set - introduced here).

This all also means, xvdd not always is a "system disk". Adjust listing
connected disks accordingly.
2019-11-19 14:03:21 +01:00
Marek Marczykowski-Górecki
9bf0cce11e
tests: extend mock objects in QubesVM tests
- allow TestQubesDB to be populated with initial data
- support list() method
- allow to register pre-created VM instance (useful for AdminVM, which
        don't accept setting qid)
2019-11-18 23:45:02 +01:00
Frédéric Pierret (fepitre)
e092b40350
tests: make PEP8 happier 2019-11-17 01:54:27 +01:00
Frédéric Pierret (fepitre)
3a6ed2d0cd
tests: fix test with not None default_guivm 2019-11-17 01:48:20 +01:00
Marta Marczykowska-Górecka
783832adde
Correct inconsistent behavior on unavailable usage data
fixes QubesOS/qubes-issues#5463
2019-11-15 20:01:43 +01:00
Marek Marczykowski-Górecki
ba105e89c6
Merge branch 'devel20191029' 2019-11-12 23:11:30 +01:00
Marek Marczykowski-Górecki
7de9f3e078
tests: is_iommu_supported function
QubesOS/qubes-issues#4689
2019-11-12 22:45:30 +01:00
Marek Marczykowski-Górecki
c79e91d59d
Merge remote-tracking branch 'origin/pr/292'
* origin/pr/292:
  Throw BackupAlreadyRunningError when backup is already running
2019-11-12 22:26:53 +01:00
Marta Marczykowska-Górecka
7a4455bbfa
Throw BackupAlreadyRunningError when backup is already running
Instead of generic PermissionDenied.

requires https://github.com/QubesOS/qubes-core-admin-client/pull/115
references QubesOS/qubes-issues#5432
2019-11-11 19:16:03 +01:00
Marek Marczykowski-Górecki
77cf310c47
storage/kernels: fix listing volumes
Pool.volumes property is implemented in a base class, individual drivers
should provide list_volumes() method as a backend for that property.
Fix this in a LinuxKernel pool.
2019-11-10 01:14:34 +01:00
Marek Marczykowski-Górecki
263f218d40
api/admin: implement admin.pool.volume.List method
Similar to admin.vm.volume.List. There are still other
admin.pool.volume.* methods missing, but lets start with just this one.
Those with both pool name and volume id arguments may need some more
thoughts.
2019-11-09 18:38:56 +01:00
Frédéric Pierret (fepitre)
5f934b43ab
tests: add app guivm tests 2019-11-09 17:06:46 +01:00
Marek Marczykowski-Górecki
598d059c57
tests: check if storage driver adjust the size on import_volume/clone
Regression test for QubesOS/qubes-issues#4821
2019-10-31 01:17:26 +01:00
Marek Marczykowski-Górecki
8603571cbc
tests: skip networking tests for minimal templates
Minimal templates don't have networking packages installed by default.
2019-10-28 03:30:49 +01:00
Marek Marczykowski-Górecki
cc56c6f96a
Merge remote-tracking branch 'origin/pr/278'
* origin/pr/278:
  Added admin.pool.UsageDetails API method
  Add metadata info to LVM AdminAPI
2019-10-23 04:36:49 +02:00
Marta Marczykowska-Górecka
2f6497e48d
Added admin.pool.UsageDetails API method
admin.pool.UsageDetails reports the usage data, unlike
admin.pool.Info, which should report the config/unchangeable data.
At the moment admin.Pool.Info still reports usage, to maintain
compatibility, but once all relevant tools are updated,
it should just return configuration data.
2019-10-23 03:04:30 +02:00
Marta Marczykowska-Górecka
04a215fb83
Add metadata info to LVM AdminAPI
Added usage_details method to Pool class
(returns a dictionary with detailed information
on pool usage) and LVM implementation that returns
metadata info.
Needed for QubesOS/qubes-issues#5053
2019-10-22 17:29:01 +02:00
Marek Marczykowski-Górecki
656e36f1ee
Merge remote-tracking branch 'origin/pr/287'
* origin/pr/287:
  app: fix docstrings PEP8 refactor
  tests: remove iptables_header content in test_622_qdb_keyboard_layout
  tests: add test for guivm and keyboard_layout
  gui: simplify setting guivm xid and keyboard layout
  Make pylint happier
  gui: set keyboard layout from feature
  Handle GuiVM properties
  Make PEP8 happier
2019-10-22 14:10:56 +02:00
Frédéric Pierret (fepitre)
5ee97f4eeb
tests: remove iptables_header content in test_622_qdb_keyboard_layout 2019-10-22 09:26:03 +02:00
Frédéric Pierret (fepitre)
e667639914
tests: add test for guivm and keyboard_layout 2019-10-20 17:36:06 +02:00
Marek Marczykowski-Górecki
a1dabfefa0
tests: fix network re-attach tests
On slow systems (OpenQA), 5s isn't enough. Instead of hardcoding some
timeout, simply wait for the full VM startup.
2019-10-13 05:58:50 +02:00
Marek Marczykowski-Górecki
f0ae8c0454
tests: fix a fix for too short delay in 201_shutdown_event_race test
It's _domain_stopped_lock, not startup_lock.
2019-10-13 05:58:19 +02:00
Marek Marczykowski-Górecki
cc1ac0b859
tests: log stderr of paplay command 2019-09-29 06:43:34 +02:00
Marek Marczykowski-Górecki
7def96c248
tests: register syslog logger, log test start
Move this functionality from our custom runner (qubes.tests.run),
into base test class. This is very useful for correlating logs, so lets
have it with nose2 runner too.
2019-09-29 06:43:34 +02:00
Marek Marczykowski-Górecki
e0e0c7eaf9
tests: remove VM under startup_lock
Prevent starting a VM while it's being removed. Something could try to
start a VM just after it's being killer but before removing it (Whonix
example from previous commit is real-life case). The window specifically
is between kill() call and removing it from collection
(`del app.domains[vm.qid]`). Grab a startup_lock for the whole operation
to prevent it.
2019-09-29 06:14:21 +02:00
Marek Marczykowski-Górecki
6cfda328bf
tests: add workaround for Whonix re-starting VMs
Workaround for https://phabricator.whonix.org/T930
For now, unregister all the VMs to be killed manually.
2019-09-29 06:14:21 +02:00
Marek Marczykowski-Górecki
34e2f3a322
tests: fix sorting kernel version
Debian now has 4.9 and 4.19 kernels installed, so `sort -n` sorts them
wrong.
2019-09-27 16:29:21 +02:00
Marek Marczykowski-Górecki
8ecf00bd0e
tests: add helpful decorator to wait before test cleanup
Allow to manual inspect test environment after test fails. This is
similar to --do-not-clean option we had in R3.2.

The decorator should be used only while debugging and should never be
applied to the code committed into repository.
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
4fa45dbc91
tests: fix (hopefully) too short delay in 201_shutdown_event_race test
Domain shutdown handling may take extended amount of time, especially on
slow machine (all the LVM teardown etc). Take care of it by
synchronizing using vm.startup_lock, instead of increasing constant
delay. This way, the shutdown event handler needs to be started within
3s, not finish in this time.
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
dfa0626cea
tests: check event handler re-registration after libvirt restart
QubesOS/qubes-issues#5303
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
273238bd2a
tests: fix qrexec abort test
Exit code depends on when exactly the other end was terminated.
2019-09-22 23:30:38 +02:00
Marek Marczykowski-Górecki
c5aaf3abd9
tests: make libvirt mockup more robust
If not in offline_mode, return actual mock for libvirt connection object
instead of always raising exception.
2019-09-10 03:34:11 +02:00
Marek Marczykowski-Górecki
05e48748d2
tests: update for not needing custom kernel modules anymore
kernel-devel package isn't needed in VMs anymore.
2019-09-10 03:33:42 +02:00
Marek Marczykowski-Górecki
5d5f102378
Merge remote-tracking branch 'origin/pr/277'
* origin/pr/277:
  admin: add admin.deviceclass.List
  admin: replace single quote to double for docstring
2019-08-08 14:05:00 +02:00
Frédéric Pierret (fepitre)
7ff01b631d
admin: add admin.deviceclass.List
QubesOS/qubes-issues#5213
2019-08-06 11:01:02 +02:00
Marek Marczykowski-Górecki
39d64eabc8
api/stats: improve cpu_usage normalization, add cpu_usage_raw
Give raw cpu_time value, instead of normalized one (to number of vcpus),
as documented.
Move the normalization to cpu_usage calculation. At the same time, add
cpu_usage_raw without it, if anyone needs it.

QubesOS/qubes-issues#4531
2019-08-01 04:51:05 +02:00
Marek Marczykowski-Górecki
9e226ab5cf
Merge remote-tracking branch 'origin/pr/273'
* origin/pr/273:
  tests: check importing empty data into ReflinkVolume
  tests: check importing empty data into ThinVolume
  tests: check importing empty data into FileVolume
  tests: improve cleanup after LVM tests
2019-07-31 15:37:13 +02:00
Marek Marczykowski-Górecki
afb0de43d4
tests: check importing empty data into ReflinkVolume
Verify if it really discards old content.

QubesOS/qubes-issues#5192
2019-07-28 22:08:52 +02:00
Marek Marczykowski-Górecki
19186f7840
tests: check importing empty data into ThinVolume
Verify if it really discards old content.

QubesOS/qubes-issues#5192
2019-07-28 22:08:37 +02:00
Marek Marczykowski-Górecki
790c2ad8cb
tests: check importing empty data into FileVolume
Verify if it really discards old content.

QubesOS/qubes-issues#5192
2019-07-28 22:06:30 +02:00
Marek Marczykowski-Górecki
8414d0153f
tests: improve cleanup after LVM tests
Remove test volumes - this way if a test fails, subsequent tests have a
chance to succeed.
2019-07-28 21:41:04 +02:00
Rusty Bird
1d89acf698
app: setup_pools() must be a coroutine
This is needed as a consequence of d8b6d3ef ("Make add_pool/remove_pool
coroutines, allow Pool.{setup,destroy} as coroutines"), but there hasn't
been any problem so far because no storage driver implemented pool
setup() as a coroutine.
2019-06-28 10:29:26 +00:00
Rusty Bird
fe97a15d11
factor out utils.coro_maybe() 2019-06-28 10:29:24 +00:00
Marek Marczykowski-Górecki
883d324b6c
Revert "tests: do not use lazy unmount"
Revert to use umount -l in storage tests cleanup. With fixed permissions
in 4234fe51 "tests: fix cleanup after reflink tests", it shouldn't cause
issues anymore, but apparently on some systems test cleanup fails
otherwise.

Reported by @rustybird
This reverts commit b6f77ebfa1.
2019-06-25 05:51:33 +02:00
Marek Marczykowski-Górecki
fa0ae0c921
Merge remote-tracking branch 'origin/pr/257'
* origin/pr/257:
  tests: AdminVM.run_service*()
  tests: QubesVM.run_service function
  vm/adminvm: add run_service* functions
2019-06-23 03:43:35 +02:00
Marek Marczykowski-Górecki
c468b29490
tests: AdminVM.run_service*() 2019-06-21 20:45:38 +02:00
Marek Marczykowski-Górecki
e352fc25d7
tests: QubesVM.run_service function 2019-06-21 20:45:38 +02:00
Rusty Bird
30b92f8845
storage/reflink: simplify volume.usage() 2019-06-15 16:03:43 +00:00
Marek Marczykowski-Górecki
b6c4f8456f
api/admin: make admin.vm.Console call go through qubesd
Ask qubesd for admin.vm.Console call. This allows to intercept it with
admin-permission event. While at it, extract tty path extraction to
python, where libvirt domain object is already available.

Fixes QubesOS/qubes-issues#5030
2019-06-09 18:03:18 +02:00
Marek Marczykowski-Górecki
c5aaf8cdd7
Merge branch 'remove-qrexec' 2019-06-07 05:10:54 +02:00
Marek Marczykowski-Górecki
7cc5c66f93
Merge remote-tracking branch 'origin/pr/260'
* origin/pr/260:
  tests: handle /qubes-mac
  mac address: lowercase of default value as in _setter_mac
  qubesdb: add qubes-mac path entry
2019-05-28 23:34:44 +02:00
Frédéric Pierret (fepitre)
1e3e579938
tests: handle /qubes-mac 2019-05-23 11:30:12 +02:00
Frédéric Pierret (fepitre)
eaba6e54ba
Fix tests for hvm with console 2019-05-06 18:56:31 +02:00
Marek Marczykowski-Górecki
33bf3d4506
tests/salt: log more details about failures
Log command output on failure. By default CalledProcessError
presentation do not include it.
2019-04-13 21:51:42 +02:00
Marek Marczykowski-Górecki
5d0f8f8efa
tests/mime: confirm closing Firefox
If Firefox is started for the first time, it will open both requested
page and its welcome page. This means closing the window will trigger a
confirmation about closing multiple tabs. Handle this.
2019-04-13 21:50:34 +02:00
Marek Marczykowski-Górecki
66de41ac26
tests: use pkill instead of killall
killall isn't installed by default
2019-04-13 16:42:37 +02:00
Marek Marczykowski-Górecki
76b807a94b
tests: call qrexec tests
Now they live in separate package, but include them in default tests set
anyway.

QubesOS/qubes-issues#4955
2019-04-11 04:29:17 +02:00
Marek Marczykowski-Górecki
1949b0c777
Remove qrexec related files - moved to core-qrexec repository
QubesOS/qubes-issues#4955
2019-04-11 04:29:04 +02:00
Marek Marczykowski-Górecki
75916dca5d
tests: fix waiting for audio recording to end 2019-04-05 03:16:55 +02:00
Marek Marczykowski-Górecki
fc154fbbad
tests: fix resource leaks - process not waited for 2019-04-02 18:48:43 +02:00
Marek Marczykowski-Górecki
6b93aea5bd
tests: remove unused imports 2019-04-02 18:48:35 +02:00
Marek Marczykowski-Górecki
4234fe5112
tests: fix cleanup after reflink tests
First make the directory accessible again, otherwise os.path.ismount may
not detect it (so umount would not be called) and then rmdir will fail.
2019-03-18 02:21:38 +01:00