Commit Graph

740 Commits

Author SHA1 Message Date
Pawel Marczewski
fe72fe1681
Fix overlapping block device names
This was caused by a change in Jinja template engine that breaks
assignments like {% set i = i + 1 %} in a loop.

Jinja 2.10 introduces a "namespace" object for this use case, but
unfortunately dom0-fc25 uses 2.8.1.

See:

https://github.com/pallets/jinja/issues/641
https://github.com/pallets/jinja/pull/684

Fixes QubesOS/qubes-issues#5551.
2020-01-23 11:27:05 +01:00
Marek Marczykowski-Górecki
8dda66d693
tests: fix key generation for dom0 updates tests
gpg2 requires explicit `%no-protection`, otherwise tries to
interactively (through gpg-agent) prompt for the passphrase.
2020-01-17 04:50:31 +01:00
Marek Marczykowski-Górecki
309dd11b1d
Merge remote-tracking branch 'origin/pr/307'
* origin/pr/307:
  Implement missing get_cputime() in AdminVM
  Add admin.vm.CurrentState method
2020-01-16 04:12:42 +01:00
Marek Marczykowski-Górecki
161f139f98
Merge remote-tracking branch 'origin/pr/305'
* origin/pr/305:
  Test file copy hang when target machine qrexec is disabled
2020-01-16 04:05:09 +01:00
Marek Marczykowski-Górecki
0c08305f1a
Merge remote-tracking branch 'origin/pr/303'
* origin/pr/303:
  Update tests after adding /connected-ips
  Also reload /connected-ips on firewall change / domain spawn
  Also store /connected-ips6 for machines that have IPv6 addresses
  Don't try to write to qubesdb of an offline VM
  Maintain a list of connected machine IPs in qubesdb
2020-01-16 04:03:44 +01:00
Marek Marczykowski-Górecki
e6aa35fcdf
Merge branch 'tests-leaks'
* tests-leaks:
  ext/pci: handle 'qubes-close' event
  tests: allow extensions to cleanup objects references
2020-01-16 04:00:59 +01:00
Marek Marczykowski-Górecki
51adb434d1
tests: FileVolume.resize()
QubesOS/qubes-issues#5518
2020-01-16 01:38:14 +01:00
Pawel Marczewski
62735d9696
Add admin.vm.CurrentState method
For all the transient values that should not be treated as
real properties. To be used in core-admin-client.
2020-01-15 16:06:08 +01:00
Pawel Marczewski
e69bdc5efa
Test file copy hang when target machine qrexec is disabled
See QubesOS/qubes-issues#5347.

Fix in QubesOS/qubes-core-qrexec#15.
2020-01-14 15:07:42 +01:00
Marek Marczykowski-Górecki
74e956e1f1
ext/pci: handle 'qubes-close' event
Move cache cleanup to 'qubes-close' event handler, instead of doing it
specifically in tests.
2020-01-11 03:54:30 +01:00
Marek Marczykowski-Górecki
a89d3f0cae
tests: allow extensions to cleanup objects references
Extension objects are singletons and normally do not require any special
cleanup. But in case of tests, we try to remove all the qubes objects
between tests and the cache in usb extension makes it hard.
Add a 'qubes-close' event that extensions can handle to remove extra
references stored in extension objects themselves.
2020-01-11 01:22:30 +01:00
Pawel Marczewski
70c862fe73
Update tests after adding /connected-ips 2020-01-09 16:41:14 +01:00
Marek Marczykowski-Górecki
8965e9a8e4
tests/network: let xl devd bring the interfaces up
xl devd doesn't use startup notify, so when the service is started
(according to systemd) it may still be initializing interfaces. Add a
little sleep for that.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
ffa1a40e6e
tests/network: improve error reporting
Include stdout/stderr of failed command during netvm setup.
2019-12-05 23:31:15 +01:00
Marek Marczykowski-Górecki
10f99e5c4a
api/admin: implement *.property.GetAll methods
Allow getting all the VM properties with one call. This greatly improve
performance of an applications retrieving many/all of them (qvm-ls,
qubes manager etc)

QubesOS/qubes-issues#5415
Fixes QubesOS/qubes-issues#3293
2019-12-05 23:31:14 +01:00
Marek Marczykowski-Górecki
7c18b187de
tests: add include and exclude lists for extra tests loader
'extra' tests run is getting ridiculously long. Allow splitting it into
several jobs. Since this appears as just one class from the test loader
perspective, implement it as environment variables:
 - QUBES_TEST_EXTRA_INCLUDE - load just selected tests
 - QUBES_TEST_EXTRA_EXCLUDE - skip selected tests (to select "the rest"
   tests)
2019-11-30 04:35:18 +01:00
Marek Marczykowski-Górecki
6c7af109e5
ext/block: prefer connecting cdrom as xvdd
Only first 4 disks can be emulated as IDE disks by QEMU. Specifically,
CDROM must be one of those first 4 disks, otherwise it will be
ignored. This is especially important if one wants to boot the VM from
that CDROM.
Since xvdd normally is a kernel-related volume (boot image, modules) it
makes perfect sense to re-use it for CDROM. It is either set for kernel
volume (in which case, VM should boot from it and not the CDROM), or
(possibly bootable) CDROM.

This needs to be done in two places:
 - BlockExtension for dynamic attach
 - libvirt xen.xml - for before-boot attach

In theory the latter would be enough, but it would be quite confusing
that device will get different options depending on when it's attached
(in addition to whether the kernel is set - introduced here).

This all also means, xvdd not always is a "system disk". Adjust listing
connected disks accordingly.
2019-11-19 14:03:21 +01:00
Marek Marczykowski-Górecki
9bf0cce11e
tests: extend mock objects in QubesVM tests
- allow TestQubesDB to be populated with initial data
- support list() method
- allow to register pre-created VM instance (useful for AdminVM, which
        don't accept setting qid)
2019-11-18 23:45:02 +01:00
Frédéric Pierret (fepitre)
e092b40350
tests: make PEP8 happier 2019-11-17 01:54:27 +01:00
Frédéric Pierret (fepitre)
3a6ed2d0cd
tests: fix test with not None default_guivm 2019-11-17 01:48:20 +01:00
Marta Marczykowska-Górecka
783832adde
Correct inconsistent behavior on unavailable usage data
fixes QubesOS/qubes-issues#5463
2019-11-15 20:01:43 +01:00
Marek Marczykowski-Górecki
ba105e89c6
Merge branch 'devel20191029' 2019-11-12 23:11:30 +01:00
Marek Marczykowski-Górecki
7de9f3e078
tests: is_iommu_supported function
QubesOS/qubes-issues#4689
2019-11-12 22:45:30 +01:00
Marek Marczykowski-Górecki
c79e91d59d
Merge remote-tracking branch 'origin/pr/292'
* origin/pr/292:
  Throw BackupAlreadyRunningError when backup is already running
2019-11-12 22:26:53 +01:00
Marta Marczykowska-Górecka
7a4455bbfa
Throw BackupAlreadyRunningError when backup is already running
Instead of generic PermissionDenied.

requires https://github.com/QubesOS/qubes-core-admin-client/pull/115
references QubesOS/qubes-issues#5432
2019-11-11 19:16:03 +01:00
Marek Marczykowski-Górecki
77cf310c47
storage/kernels: fix listing volumes
Pool.volumes property is implemented in a base class, individual drivers
should provide list_volumes() method as a backend for that property.
Fix this in a LinuxKernel pool.
2019-11-10 01:14:34 +01:00
Marek Marczykowski-Górecki
263f218d40
api/admin: implement admin.pool.volume.List method
Similar to admin.vm.volume.List. There are still other
admin.pool.volume.* methods missing, but lets start with just this one.
Those with both pool name and volume id arguments may need some more
thoughts.
2019-11-09 18:38:56 +01:00
Frédéric Pierret (fepitre)
5f934b43ab
tests: add app guivm tests 2019-11-09 17:06:46 +01:00
Marek Marczykowski-Górecki
598d059c57
tests: check if storage driver adjust the size on import_volume/clone
Regression test for QubesOS/qubes-issues#4821
2019-10-31 01:17:26 +01:00
Marek Marczykowski-Górecki
8603571cbc
tests: skip networking tests for minimal templates
Minimal templates don't have networking packages installed by default.
2019-10-28 03:30:49 +01:00
Marek Marczykowski-Górecki
cc56c6f96a
Merge remote-tracking branch 'origin/pr/278'
* origin/pr/278:
  Added admin.pool.UsageDetails API method
  Add metadata info to LVM AdminAPI
2019-10-23 04:36:49 +02:00
Marta Marczykowska-Górecka
2f6497e48d
Added admin.pool.UsageDetails API method
admin.pool.UsageDetails reports the usage data, unlike
admin.pool.Info, which should report the config/unchangeable data.
At the moment admin.Pool.Info still reports usage, to maintain
compatibility, but once all relevant tools are updated,
it should just return configuration data.
2019-10-23 03:04:30 +02:00
Marta Marczykowska-Górecka
04a215fb83
Add metadata info to LVM AdminAPI
Added usage_details method to Pool class
(returns a dictionary with detailed information
on pool usage) and LVM implementation that returns
metadata info.
Needed for QubesOS/qubes-issues#5053
2019-10-22 17:29:01 +02:00
Marek Marczykowski-Górecki
656e36f1ee
Merge remote-tracking branch 'origin/pr/287'
* origin/pr/287:
  app: fix docstrings PEP8 refactor
  tests: remove iptables_header content in test_622_qdb_keyboard_layout
  tests: add test for guivm and keyboard_layout
  gui: simplify setting guivm xid and keyboard layout
  Make pylint happier
  gui: set keyboard layout from feature
  Handle GuiVM properties
  Make PEP8 happier
2019-10-22 14:10:56 +02:00
Frédéric Pierret (fepitre)
5ee97f4eeb
tests: remove iptables_header content in test_622_qdb_keyboard_layout 2019-10-22 09:26:03 +02:00
Frédéric Pierret (fepitre)
e667639914
tests: add test for guivm and keyboard_layout 2019-10-20 17:36:06 +02:00
Marek Marczykowski-Górecki
a1dabfefa0
tests: fix network re-attach tests
On slow systems (OpenQA), 5s isn't enough. Instead of hardcoding some
timeout, simply wait for the full VM startup.
2019-10-13 05:58:50 +02:00
Marek Marczykowski-Górecki
f0ae8c0454
tests: fix a fix for too short delay in 201_shutdown_event_race test
It's _domain_stopped_lock, not startup_lock.
2019-10-13 05:58:19 +02:00
Marek Marczykowski-Górecki
cc1ac0b859
tests: log stderr of paplay command 2019-09-29 06:43:34 +02:00
Marek Marczykowski-Górecki
7def96c248
tests: register syslog logger, log test start
Move this functionality from our custom runner (qubes.tests.run),
into base test class. This is very useful for correlating logs, so lets
have it with nose2 runner too.
2019-09-29 06:43:34 +02:00
Marek Marczykowski-Górecki
e0e0c7eaf9
tests: remove VM under startup_lock
Prevent starting a VM while it's being removed. Something could try to
start a VM just after it's being killer but before removing it (Whonix
example from previous commit is real-life case). The window specifically
is between kill() call and removing it from collection
(`del app.domains[vm.qid]`). Grab a startup_lock for the whole operation
to prevent it.
2019-09-29 06:14:21 +02:00
Marek Marczykowski-Górecki
6cfda328bf
tests: add workaround for Whonix re-starting VMs
Workaround for https://phabricator.whonix.org/T930
For now, unregister all the VMs to be killed manually.
2019-09-29 06:14:21 +02:00
Marek Marczykowski-Górecki
34e2f3a322
tests: fix sorting kernel version
Debian now has 4.9 and 4.19 kernels installed, so `sort -n` sorts them
wrong.
2019-09-27 16:29:21 +02:00
Marek Marczykowski-Górecki
8ecf00bd0e
tests: add helpful decorator to wait before test cleanup
Allow to manual inspect test environment after test fails. This is
similar to --do-not-clean option we had in R3.2.

The decorator should be used only while debugging and should never be
applied to the code committed into repository.
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
4fa45dbc91
tests: fix (hopefully) too short delay in 201_shutdown_event_race test
Domain shutdown handling may take extended amount of time, especially on
slow machine (all the LVM teardown etc). Take care of it by
synchronizing using vm.startup_lock, instead of increasing constant
delay. This way, the shutdown event handler needs to be started within
3s, not finish in this time.
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
dfa0626cea
tests: check event handler re-registration after libvirt restart
QubesOS/qubes-issues#5303
2019-09-27 16:29:19 +02:00
Marek Marczykowski-Górecki
273238bd2a
tests: fix qrexec abort test
Exit code depends on when exactly the other end was terminated.
2019-09-22 23:30:38 +02:00
Marek Marczykowski-Górecki
c5aaf3abd9
tests: make libvirt mockup more robust
If not in offline_mode, return actual mock for libvirt connection object
instead of always raising exception.
2019-09-10 03:34:11 +02:00
Marek Marczykowski-Górecki
05e48748d2
tests: update for not needing custom kernel modules anymore
kernel-devel package isn't needed in VMs anymore.
2019-09-10 03:33:42 +02:00
Marek Marczykowski-Górecki
5d5f102378
Merge remote-tracking branch 'origin/pr/277'
* origin/pr/277:
  admin: add admin.deviceclass.List
  admin: replace single quote to double for docstring
2019-08-08 14:05:00 +02:00