Commit Graph

776 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
d2585aa871
tests/lvm: fix checking lvm pool existence cont.
Commit 15cf593bc5 "tests/lvm: fix checking
lvm pool existence" attempted to fix handling '-' in pool name by using
/dev/VG/LV symlink. But those are not created for thin pools. Change
back to /dev/mapper, but include '-' mangling.

Related QubesOS/qubes-issues#4332
2018-11-15 18:25:29 +01:00
Marek Marczykowski-Górecki
68dffb6895
api/admin: fix error message when refusing to create template on template
Fixes QubesOS/qubes-issues#4463
2018-11-04 17:05:55 +01:00
Marek Marczykowski-Górecki
15cf593bc5
tests/lvm: fix checking lvm pool existence
If pool or group name have '-', it will be mangled as '--' in
/dev/mapper. Use /dev/VG_NAME/LV_NAME symlink instead.

Related QubesOS/qubes-issues#4332
2018-10-30 01:17:00 +01:00
Marek Marczykowski-Górecki
f621e8792c
Merge branch 'master' into devel-no-assert 2018-10-29 20:29:53 +01:00
Marek Marczykowski-Górecki
db6094f397
tests/api: adjust for proper exceptions instead of AssertionError 2018-10-29 20:22:10 +01:00
Marek Marczykowski-Górecki
2b5fc6299e
tests/api: do not test non-existing methods
Remove methods not included in specification (or with different
constraints). Keep commented out methods included in spec but not
implemented.
2018-10-29 20:21:36 +01:00
Marek Marczykowski-Górecki
42061cb194
tests: try to collect qvm-open-in-dvm output if no editor window is shown
Try to collect more details about why the test failed. This will help
only if qvm-open-in-dvm exist early. On the other hand, if it hang, or
remote side fails to find the right editor (which results in GUI error
message), this change will not provide any more details.
2018-10-29 01:20:57 +01:00
Marek Marczykowski-Górecki
84c321b923
tests: increase session startup timeout for whonix-ws based VMs
First boot of whonix-ws based VM take extended period of time, because
a lot of files needs to be copied to private volume. This takes even
more time, when verbose logging through console is enabled. Extend the
timeout for that.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
84d3547f09
tests: adjust extra tests loader to work with nose2
Nose loader do not provide loader.loadTestsFromTestCase(), use
loader.loadTestsFromNames() instead.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
fb14f589cb
tests: wait for full user session before doing rest of the test
Clean VM shutdown may timeout if its initiated before full startup, so
make sure the full startup is completed first.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
4742a630f2
tests: use iptables --wait
QubesOS/qubes-issues#3665 affects also tests...
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
8be70c9e4d
ext/services: allow for os=Linux feature request from VM
It's weird to set it for Windows, but not Linux.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
e244c192ae
tests: use /bin/uname instead of /bin/hostname as dummy output generator
Use something included in coreutils installed everywhere.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
0b7aa546c6
tests: remove VM reference from QubesVMError
Yet another place wheren object references are leaked.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
a972c61914
tests: use socat instead of nc
socat have only one variant, so one command line syntax to handle. It's
also installed by default in Qubes VMs.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
08ddeee9fb
tests: improve VMs cleanup wrt custom templates
Cleanup VMs in template reverse topological order, not network one.
Network can be set to None to break dependency, but template can't. For
netvm to be changed, kill VMs first (kill doesn't check network
dependency), so netvm change will not trigger side effects (runtime
change, which could fail).

This fixes cleanup for tests creating custom templates - previously
order was undefined and if template was tried removed before its child
VMs, it fails. All the relevant files were removed later anyway, but it
lead to python objects leaks.
2018-10-27 16:44:53 +02:00
Marek Marczykowski-Górecki
4e762788a9
tests: check if qubes-vm@ service is disabled on domain removal
Test for QubesOS/qubes-issues#4014
2018-10-27 01:43:00 +02:00
Marek Marczykowski-Górecki
cf8b6219a9
tests: make use of vm.shutdown(wait=True) 2018-10-27 01:43:00 +02:00
Marek Marczykowski-Górecki
5be003d539
vm/dispvm: fix DispVM cleanup
First unregister the domain from collection, and only then call
remove_from_disk(). Removing it from collection prevent further calls
being made to it. Or if anything else keep a reference to it (for
example as a netvm), then abort the operation.

Additionally this makes it unnecessary to take startup lock when
cleaning it up in tests.
2018-10-26 23:54:08 +02:00
Marek Marczykowski-Górecki
b65fdf9700
storage: convert lvm driver to async version
LVM operations can take significant amount of time. This is especially
visible when stopping a VM (`vm.storage.stop()`) - in that time the
whole qubesd freeze for about 2 seconds.

Fix this by making all the ThinVolume methods a coroutines (where
supported). Each public coroutine is also wrapped with locking on
volume._lock to avoid concurrency-related problems.
This all also require changing internal helper functions to
coroutines. There are two functions that still needs to be called from
non-coroutine call sites:
 - init_cache/reset_cache (initial cache fill, ThinPool.setup())
 - qubes_lvm (ThinVolume.export()

So, those two functions need to live in two variants. Extract its common
code to separate functions to reduce code duplications.

Fixes QubesOS/qubes-issues#4283
2018-10-23 16:53:35 +02:00
Marek Marczykowski-Górecki
299c514647
tests: fix asyncio usage in storage_lvm.TC_01_ThinPool
Both vm.create_on_disk() and vm.start() are coroutines. Tests in this
class didn't run them, so basically didn't test anything.

Wrap couroutine calls with self.loop.run_until_complete().

Additionally, don't fail if LVM pool is named differently.
In that case, the test is rather sily, as it probably use the same pool
for source and destination (operation already tested elsewhere). But it
isn't a reason for failing the test.
2018-10-23 16:53:35 +02:00
Marek Marczykowski-Górecki
d1f5cb5d15
ext/services: mechanism for advertising supported services
Support 'supported-service.*' features requests coming from VMs. Set
such features directly (allow only value '1') and remove any not
reported in given call. This way uninstalling package providing given
service will automatically remove related 'supported-service...'
feature.

Fixes QubesOS/qubes-issues#4402
2018-10-23 16:47:39 +02:00
Marek Marczykowski-Górecki
c01ae06fee
tests: add basic ServicesExtension tests 2018-10-17 17:37:02 +02:00
Marek Marczykowski-Górecki
e8dc6cb916
tests: use smaller root.img in backupcompatibility tests
1GB image easily exceed available space on openQA instances. Use 100MB
instead.
2018-10-15 05:24:24 +02:00
Marek Marczykowski-Górecki
fd9f2e2a6c
tests: type commands into specific found window
Make sure events are sent to specific window found with xdotool search,
not the one having the focus. In case of Whonix, it can be first
connection wizard or whonixcheck report.
2018-10-15 05:18:42 +02:00
Marek Marczykowski-Górecki
9e81087b25
tests: use improved wait_for_window in various tests
Replace manual `xdotool search calls` with wait_for_window(), where
compatible.
2018-10-15 05:16:29 +02:00
Marek Marczykowski-Górecki
f1621c01e9
tests: add search based on window class to wait_for_window
Searching based on class is used in many tests, searching by class, not
only by name in wait_for_window will allow to reduce code duplication.
While at it, improve it additionally:
 - avoid active waiting for window and use `xdotool search --sync` instead
 - return found window id
 - add wait_for_window_coro() for use where coroutine is needed
 - when waiting for window to disappear, check window id once and wait
   for that particular window to disappear (avoid xdotool race
   conditions on window enumeration)

Besides reducing code duplication, this also move various xdotool
imperfections handling into one place.
2018-10-15 05:08:25 +02:00
Marek Marczykowski-Górecki
3f5618dbb0
tests/integ/network: make the tests independend of default netvm
Network tests create own temporary netvm. Make it disconnected from the
real netvm, to not interefere in tests.
2018-10-15 00:39:32 +02:00
Marek Marczykowski-Górecki
15140255d5
tests/integ/network: few more code style improvement
Remove unused imports and unused variables, add some more docstrings.
2018-10-15 00:38:38 +02:00
Marek Marczykowski-Górecki
375688837c
tests/integ/network: add type annotations
Make PyCharm understand what mixin those objects are for.
2018-10-15 00:30:17 +02:00
Marek Marczykowski-Górecki
3b6703f2bd
tests: fix race condition in gui_memory_pinning test
Don't rely on top update timing, pause it updates for taking
screenshots.
2018-10-14 05:48:25 +02:00
Marek Marczykowski-Górecki
9887b925b4
tests: increase timeout for vm shutdown
start_standalone_with_cdrom_vm test essence is somewhere else, let not
fail it for just slow shutdown (LVM cleanup etc).
2018-10-14 05:29:05 +02:00
Marek Marczykowski-Górecki
29a26e7d69
tests: make timeout it shutdown test even longer
Reduce false positives when testing on busy machine.
2018-10-14 03:32:17 +02:00
Marek Marczykowski-Górecki
3e28ccefde
tests: fix cleanup after backup compatibility tests
Allow removing VMs based on multiple prefixes at once. Removing them
separately doesn't handle all the dependencies (default_netvm, netvm)
correctly. This is needed for backup compatibility tests, where VMs are
created with `test-` prefix and `disp-tests-`. Additionally backup code
will create `disp-no-netvm`, which also may need to be removed.
2018-10-14 03:29:30 +02:00
Marek Marczykowski-Górecki
00c0b4c69f
tests: cleanup tracebacks also for expectedFailure exception
Continuation of 5bc0baea "tests: do not leak objects in object leaks
checking function".
2018-10-12 14:41:35 +02:00
Marek Marczykowski-Górecki
d23636fa02
tests: migrate qvm-block tests to core3 2018-10-10 00:44:15 +02:00
Marek Marczykowski-Górecki
1245606453
tests: fix cleanup of dom0_update tests
Reset updatevm to None before removing VMs, otherwise removing updatevm
will fail.
2018-10-10 00:08:57 +02:00
Marek Marczykowski-Górecki
8dab298b89
tests: create testcases on module import if environment variable is set
If QUBES_TEST_TEMPLATES or QUBES_TEST_LOAD_ALL is set, create testcases
on modules import, instead of waiting until `load_tests` is called.
The `QUBES_TEST_TEMPLATES` doesn't require `qubes.xml` access, so it
should be safe to do regardless of the environment. The
`QUBES_TEST_LOAD_ALL` force loading tests (and reading `qubes.xml`)
regardless.

This is useful for test runners not supporting load_tests protocol. Or
with limited support - for example both default `unittest` runner and
`nose2` can either use load_tests protocol _or_ select individual tests.
Setting any of those variable allow to run a single test with those
runners.

With this feature used together load_tests protocol, tests could be
registered twice. Avoid this by not listing already defined test classes
in create_testcases_for_templates (according to load_tests protocol,
those should already be registered).
2018-10-10 00:08:29 +02:00
Marek Marczykowski-Górecki
c8929cfee9
tests: improve handling backups in core3 2018-10-07 19:51:55 +02:00
Marek Marczykowski-Górecki
7a607e3731
tests: add QUBES_TEST_TEMPLATES env variable
Allow easily list templates to be tested, without enumerating all the
test classes. This is especially useful with nose2 runner which can't
use load tests protocol _and_ select subset of tests.
2018-10-07 19:51:55 +02:00
Marek Marczykowski-Górecki
35c66987ab
tests: improve clearing tracebacks from Qubes* objects
Clear also tracebacks of chained exceptions.
2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
7c91e82365
tests: handle KWrite editor in DispVM tests 2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
02f9661169
tests: migrate mime handlers test to core3 2018-10-07 15:46:52 +02:00
Marek Marczykowski-Górecki
5bc0baeafa
tests: do not leak objects in object leaks checking function
If any object is leaked, QubesTestCase.cleanup_gc() raises an exception,
which have leaked objects list referenced in its traceback. This happens
after cleanup_traceback(), so isn't cleaned, causing cleanup_gc() fail
for all the further tests in the same test run.

Avoid this, by dropping list just before checking if any object is
leaked.
2018-09-29 02:40:43 +02:00
Rusty Bird
bee69a98b9
Add default_qrexec_timeout to qubes-prefs
When a VM (or its template) does not explicitly set a qrexec_timeout,
fall back to a global default_qrexec_timeout (with default value 60),
instead of hardcoding the fallback value to 60.

This makes it easy to set a higher timeout for the whole system, which
helps users who habitually launch applications from several (not yet
started) VMs at the same time. 60 seconds can be too short for that.
2018-09-16 18:42:48 +00:00
Marek Marczykowski-Górecki
e26655bc82
tests: fix time sync test
qvm-sync-clock no longer fetches time from the network, by design.
So, lets not break clockvm's time and check only if everything else
correctly synchronize with it.
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
c4a84b3298
tests: wait for DispVM's qubes.VMShell exit
It isn't enough to wait for window to disappear, the service may still
be running. And if it is, test cleanup logic will complain about FD
leak.
To avoid deadlock on some test failure, do it with a timeout.
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
240b1dd75e
tests: exclude whonixcheck and NetworkManager from editor window search
Those may pop up before actual editor is found, which fails the test as
it can't handle such "editor".
2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
ac8b8a3ad4
tests: reenable some qrexec tests, convert them to py3k/asyncio 2018-09-16 04:43:50 +02:00
Marek Marczykowski-Górecki
576bcb158e
tests: skip tests not relevant on Whonix 2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
556a08cb78
tests: improve shutdown timeout handling
Instead of waiting 1sec, wait up to 5sec but skip when vm is shut off.
This fix tests on slow machines, including openQA nested virt.
2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
5a1bf11d0d
tests: drop qvm-prefs tests
Those are moved to qubes-core-admin-client repository.
2018-09-15 05:12:41 +02:00
Rusty Bird
cf1ea5cee1
tests/app: test varlibqubes pool driver selection 2018-09-11 23:50:33 +00:00
Rusty Bird
797bbc43a0
tests/storage_reflink: test some file-reflink helpers
Tested:

    - _copy_file()
    - _create_sparse_file()
    - _resize_file()
    - _update_loopdev_sizes()

Smoke tested by calls from the functions above:

    - _replace_file()
    - _rename_file()
    - _make_dir()
    - _fsync_dir()
2018-09-11 23:50:32 +00:00
Rusty Bird
b82e739346
tests/integ/storage: add file-reflink integration tests 2018-09-11 23:50:31 +00:00
Rusty Bird
8c117549ad
tests/integ/basic: use export() in get_rootimg_checksum()
volume.path and volume.export() refer to the same thing in lvm_thin and
'file', but not in file-reflink (where volume.path is the -dirty.img,
which doesn't exist if the volume is not started).
2018-09-11 23:50:30 +00:00
Rusty Bird
49e7ce025f
tests/integ/backupcompatibility: Storage.verify() is a coro 2018-09-11 23:50:28 +00:00
Rusty Bird
8d1913a8cc
app: create /var/lib/qubes as file-reflink if supported
Use the file-reflink storage driver if /var/lib/qubes is on a filesystem
that supports reflinks, e.g. when the btrfs layout was selected in
Anaconda. If it doesn't support reflinks (or if detection fails, e.g. in
an unprivileged test environment), use 'file' as before.
2018-09-11 23:50:26 +00:00
Rusty Bird
53ef5ed431
app: uncouple pool setup from loading initial configuration
And ensure that setup is called on every type of these pools, not just
lvm_thin.
2018-09-11 23:50:25 +00:00
Marek Marczykowski-Górecki
b2cc605f4b
tests: clean local variables from traceback objects
System tests are fragile for any object leaks, especially those holding
open files. Instead of wrapping all tests with try/finally removing
those local variables (as done in qubes.tests.integ.backup for example),
apply generic solution: clean all traceback objects from local
variables. Those aren't used to generate text report by either test
runner (qubes.tests.run and nose2). If one wants to break into debugger
and inspect tracebacks interactively, needs to comment out call to
cleanup_traceback.
2018-09-09 02:43:17 +02:00
Marek Marczykowski-Górecki
c102fa3d68
tests: add basic audio play/rec tests
QubesOS/qubes-issues#4204
2018-09-08 04:13:24 +02:00
Marek Marczykowski-Górecki
8ce3433406
tests: drop sudo in tests already running as root
Don't spam already trashed log.
2018-09-07 15:12:05 +02:00
Marek Marczykowski-Górecki
1b99bd9ab3
Merge branch 'tests-dispvm-fix'
* tests-dispvm-fix:
  tests: fix DispVM related tests
2018-09-07 01:57:19 +02:00
Marek Marczykowski-Górecki
7f1e2741ec
Merge remote-tracking branch 'qubesos/pr/228'
* qubesos/pr/228:
  storage/lvm: filter out warning about intended over-provisioning
  tests: fix getting kernel package version inside VM
  tests/extra: add start_guid option to VMWrapper
  vm/qubesvm: fire 'domain-start-failed' event even if fail was early
  vm/qubesvm: check if all required devices are available before start
  storage/lvm: fix reporting lvm command error
  storage/lvm: save pool's revision_to_keep property
2018-09-07 01:06:59 +02:00
Marek Marczykowski-Górecki
e6edbabf94
tests: exclude windows templates from linux tests 2018-07-16 22:02:59 +02:00
Marek Marczykowski-Górecki
af2435c0d4
Make some properties default to template's value (if any)
Multiple properties are related to system installed inside the VM, so it
makes sense to have them the same for all the VMs based on the same
template. Modify default value getter to first try get the value from a
template (if any) and only if it fails, fallback to original default
value.
This change is made to those properties:
 - default_user (it was already this way)
 - kernel
 - kernelopts
 - maxmem
 - memory
 - qrexec_timeout
 - vcpus
 - virt_mode

This is especially useful for manually installed templates (like
Windows).

Related to QubesOS/qubes-issues#3585
2018-07-16 22:02:58 +02:00
Marek Marczykowski-Górecki
af7d54d388
Update windows-related feature requests
Handle 'os' feature - if it's Windows, then set rpc-clipboard feature.
Handle 'gui-emulated' feature - request for specifically stubdomain GUI.
With 'gui' feature it is only possible to enable gui-agent based on, or
disable GUI completely.
Handle 'default-user' - verify it for weird characters and set
'default_user' property (if wasn't already set).

QubesOS/qubes-issues#3585
2018-07-16 22:02:58 +02:00
Marek Marczykowski-Górecki
ce87451c73
Merge branch 'lvm-snapshots'
* lvm-snapshots:
  tests: fix handling app.pools iteration
  storage/lvm: add repr(ThinPool) for more meaningful test reports
  tests: adjust for variable volume path
  api/admin: expose volume path in admin.vm.volume.Info
  tests: LVM: import, list_volumes, volatile volume, snapshot volume
  tests: collect all SIGCHLD before cleaning event loop
  storage/lvm: use temporary volume for data import
  tests: ThinVolume.revert()
  tests: LVM volume naming migration, and new naming in general
  storage/lvm: improve handling interrupted commit
2018-07-16 16:40:35 +02:00
Marek Marczykowski-Górecki
c688641363
tests: fix DispVM related tests
- fix regex for editor window search - 'disp*' matches 'disk_space.py'
(a dom0 local widget...)
- increase a timeout for automatic DispVM cleanup
2018-07-15 22:29:22 +02:00
Marek Marczykowski-Górecki
f8d17012c3
tests: fix loading tests for templates
Fix #209
2018-07-15 21:57:34 +02:00
Marek Marczykowski-Górecki
69e3018b94
tests: fix handling app.pools iteration 2018-07-15 21:31:48 +02:00
Marek Marczykowski-Górecki
e644378f18
tests: adjust for variable volume path
LVM volumes now have variable volume path. Compare strip path before
comparing content's hash in tests.
2018-07-12 16:52:48 +02:00
Marek Marczykowski-Górecki
d211a2771a
api/admin: expose volume path in admin.vm.volume.Info
Since (for LVM at least) path is dynamic now, add information about it
to volume info. This is not very useful outside of dom0, but in dom0 it
can be very useful for various scripts.
This will disclose current volume revision id, but it is already
possible to deduce it from snapshots list.
2018-07-12 16:52:48 +02:00
Marek Marczykowski-Górecki
4282a41fcb
tests: LVM: import, list_volumes, volatile volume, snapshot volume 2018-07-12 16:52:47 +02:00
Marek Marczykowski-Górecki
76c872a43a
tests: collect all SIGCHLD before cleaning event loop
On python 3.6.4 apparently it requires two callbacks runs to cleanup
stale SIGCHLD handlers.
2018-07-12 16:52:47 +02:00
Marek Marczykowski-Górecki
2b80f0c044
storage/lvm: use temporary volume for data import
Do not write directly to main volume, instead create temporary volume
and only commit it to the main one when operation is finished. This
solve multiple problems:
 - import operation can be aborted, without data loss
 - importing new data over existing volume will not leave traces of
previous content - especially when importing smaller volume to bigger
one
 - import operation can be reverted - it create separate revision,
similar to start/stop
 - easier to prevent qube from starting during import operation
 - template still can be used when importing new version

QubesOS/qubes-issues#2256
2018-07-12 16:52:46 +02:00
Marek Marczykowski-Górecki
aea0de35ad
tests: ThinVolume.revert() 2018-07-12 16:52:46 +02:00
Marek Marczykowski-Górecki
8cf9264283
tests: LVM volume naming migration, and new naming in general 2018-07-12 16:52:45 +02:00
Marek Marczykowski-Górecki
11c7b4bb51
storage/lvm: improve handling interrupted commit
First rename volume to backup revision, regardless of revisions_to_keep,
then rename -snap to current volume. And only then remove backup
revision (if exceed revisions_to_keep). This way even if commit
operation is interrupted, there is still a volume with the data.
This requires also adjusting few functions to actually fallback to most
recent backup revision if the current volume isn't found - create
_vid_current property for this purpose.
Also, use -snap volume for clone operation and commit it normally later.
This makes it safer to interrupt or even revert.

QubesOS/qubes-issues#2256
2018-07-12 16:10:44 +02:00
Marek Marczykowski-Górecki
39a9e4e422
Merge remote-tracking branch 'qubesos/pr/209'
* qubesos/pr/209:
  qubes/tests: change globals= to module= and fix syntax errors
  qubes/tests: use loadTestsFromNames for nose2 compat
2018-04-30 01:13:11 +02:00
Wojtek Porczyk
843bbdb2c5 qubes/tests: change globals= to module= and fix syntax errors 2018-04-24 18:30:55 +02:00
Marek Marczykowski-Górecki
f9e34eff51
api/admin: fix volume info formatting
Newline was put in a wrong place.

Fixes QubesOS/qubes-issues#3831
2018-04-16 22:17:48 +02:00
Marek Marczykowski-Górecki
4794232745
tests: fix getting kernel package version inside VM
Use `sort -V` instead of `sort -n`.
2018-04-13 21:44:44 +02:00
Marek Marczykowski-Górecki
69f19bb7bb
tests/extra: add start_guid option to VMWrapper
Pass start_guid option to vm.start(), when using core2 compatibility
layer.
2018-04-13 21:43:15 +02:00
Marek Marczykowski-Górecki
b168c4a6d7
api/admin: fix admin.vm.volume.Info when is_outdated() isn't supported
is_outdated() may be not supported by given volume pool driver. In that
case skip is_outdated information, instead of crashing the call.

Fixes QubesOS/qubes-issues#3767
2018-04-08 23:25:34 +02:00
Marek Marczykowski-Górecki
8df90f1a86
tests: yet another fix for integration tests cleanup code
Before waiting for remaining tasks on event loop (including libvirt
events), make sure all destroyed objects are really destroyed. This is
especially important for libvirt connections, which gets cleaned up only
when appropriate destructor (__del__) register a cleanup callback and it
gets called by the loop.
2018-04-05 01:28:55 +02:00
Marek Marczykowski-Górecki
3726692530
tests: skip tests using netcat when it isn't installed 2018-04-05 01:28:54 +02:00
Marek Marczykowski-Górecki
2dee554ab7
vm/mix/net: make vm.gateway6 consistent with vm.gateway
Use VM's actual IP address as a gateway for other VMs, instead of
hardcoded link-local address. This is important for sys-net generated
ICMP diagnostics packets - those must _not_ have link-local source
address, otherwise wouldn't be properly forwarded back to the right VM.
2018-04-03 00:20:06 +02:00
Wojtek Porczyk
dfe7688158 qubes/tests: use loadTestsFromNames for nose2 compat 2018-03-30 03:05:01 +02:00
Marek Marczykowski-Górecki
2c13ba9c48
storage/lvm: force default locale for lvm command cont.
Few more places where lvm commands are called.

Fixes QubesOS/qubes-issues#3753
2018-03-29 20:39:15 +02:00
Marek Marczykowski-Górecki
74c1893ce3
tests: fix firewall tests after 6e8e48e3 "Avoid UTC datetime" 2018-03-22 01:47:41 +01:00
Marek Marczykowski-Górecki
6469705196
Merge branch 'devel-storage-fixes'
* devel-storage-fixes:
  storage/file: use proper exception instead of assert
  storage/file: import data into temporary volume
  storage/lvm: check for LVM LV existence and type when creating ThinPool
  storage/lvm: fix size reporting just after creating LV
2018-03-21 02:06:25 +01:00
Marek Marczykowski-Górecki
510fad9163
storage/file: import data into temporary volume
Similar to LVM changes, this fixes/improves multiple things:
 - no old data visible in the volume
 - failed import do not leave broken volume
 - parially imported data not visible to running VM

QubesOS/qubes-issues#3169
2018-03-21 01:48:07 +01:00
Marek Marczykowski-Górecki
e5413a3036
Merge branch 'storage-properties'
* storage-properties:
  storage: use None for size/usage properties if unknown
  tests: call search_pool_containing_dir with various dirs and pools
  storage: make DirectoryThinPool helper less verbose, add sudo
  api/admin: add 'included_in' to admin.pool.Info call
  storage: add Pool.included_in() method for checking nested pools
  storage: move and generalize RootThinPool helper class
  storage/kernels: refuse changes to 'rw' and 'revisions_to_keep'
  api/admin: implement admin.vm.volume.Set.rw method
  api/admin: include 'revisions_to_keep' and 'is_outdated' in volume info
2018-03-21 01:43:53 +01:00
Marek Marczykowski-Górecki
03dc3e315e
storage: use None for size/usage properties if unknown
Raising NotImplementedError in a _property_ is weird behaviour, better
suited for actions (methods). Use None instead.

QubesOS/qubes-issues#3241
2018-03-20 17:31:16 +01:00
Marek Marczykowski-Górecki
05c80c4531
tests: call search_pool_containing_dir with various dirs and pools
QubesOS/qubes-issues#3241
2018-03-20 16:53:41 +01:00
Marek Marczykowski-Górecki
1bc640f3e0
api/admin: add 'included_in' to admin.pool.Info call
QubesOS/qubes-issues#3240
QubesOS/qubes-issues#3241
2018-03-20 16:53:40 +01:00
Marek Marczykowski-Górecki
376c8ec00d
api/admin: implement admin.vm.volume.Set.rw method
Allow setting 'rw' volume property.
2018-03-20 16:52:06 +01:00
Rusty Bird
a0c5014737
api/admin: fix backup exclude
Bugfix on 59abdeb
2018-03-20 02:41:37 +00:00
Marek Marczykowski-Górecki
2a962c54db
api/admin: include 'revisions_to_keep' and 'is_outdated' in volume info
Since Volume.is_outdated() is a method, not a property, add a function
for handling serialization. And at the same time, fix None serialization
(applicable to 'source' property).

QubesOS/qubes-issues#3256
2018-03-19 04:00:21 +01:00
Marek Marczykowski-Górecki
7c4566ec14
vm/qubesvm: allow 'features-request' to have async handlers
Some handlers may want to call into other VMs (or even the one asking),
but vm.run() functions are coroutines, so needs to be called from
another coroutine. Allow for that.
Also fix typo in documentation.
2018-03-02 01:16:38 +01:00
Marek Marczykowski-Górecki
b1e83f899f
tests: convert pvgrub tests to core3 API
QubesOS/qubes-issues#3563
2018-02-28 04:41:30 +01:00
Marek Marczykowski-Górecki
1774be1c69
storage/kernel: skip modules.img block device if kernel dir doesn't have it
Some kernels (like pvgrub2) may not provide modules.img and it isn't an
error. Don't break VM startup in that case, skip that device instead.

Fixes QubesOS/qubes-issues#3563
2018-02-27 04:35:05 +01:00
Marek Marczykowski-Górecki
92a460526f
tests: fix extra test's VMWrapper.run(wait=False)
The call was ignored.
2018-02-26 02:45:26 +01:00
Marek Marczykowski-Górecki
1562defd42
Merge remote-tracking branch 'qubesos/pr/193'
* qubesos/pr/193:
  tests: add a test for removing expired firewall rules
  firewall: use asyncio's call_later instead of systemd to reload rules
2018-02-22 19:47:37 +01:00
Marek Marczykowski-Górecki
7731a56daa
Merge remote-tracking branch 'qubesos/pr/192'
* qubesos/pr/192:
  storage/lvm: add more details to refused volume shrink message
  tests: storage/lvm resize
  storage/lvm: fix resizing not persistent volumes
2018-02-22 19:46:11 +01:00
Marek Marczykowski-Górecki
b00bbb73e4
Merge remote-tracking branch 'qubesos/pr/190'
* qubesos/pr/190:
  Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90
  replaced underscore by dash and update test accordingly
  Updated assert content for test_620_qdb_standalone in TC_90_QubesVM
  Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372
2018-02-14 01:29:08 +01:00
Marek Marczykowski-Górecki
209af07fd0
Merge remote-tracking branch 'qubesos/pr/188'
* qubesos/pr/188:
  file-reflink, a storage driver optimized for CoW filesystems
  Make AppVM/DispVM root volume rw to avoid CoW-on-CoW
2018-02-13 05:20:52 +01:00
Rusty Bird
1695a732b8
file-reflink, a storage driver optimized for CoW filesystems
This adds the file-reflink storage driver. It is never selected
automatically for pool creation, especially not the creation of
'varlibqubes' (though it can be used if set up manually).

The code is quite small:

               reflink.py  lvm.py      file.py + block-snapshot
    sloccount  334 lines   447 (134%)  570 (171%)

Background: btrfs and XFS (but not yet ZFS) support instant copies of
individual files through the 'FICLONE' ioctl behind 'cp --reflink'.
Which file-reflink uses to snapshot VM image files without an extra
device-mapper layer. All the snapshots are essentially freestanding;
there's no functional origin vs. snapshot distinction.

In contrast to 'file'-on-btrfs, file-reflink inherently avoids
CoW-on-CoW. Which is a bigger issue now on R4.0, where even AppVMs'
private volumes are CoW. (And turning off the lower, filesystem-level
CoW for 'file'-on-btrfs images would turn off data checksums too, i.e.
protection against bit rot.)

Also in contrast to 'file', all storage features are supported,
including

    - any number of revisions_to_keep
    - volume.revert()
    - volume.is_outdated
    - online fstrim/discard

Example tree of a file-reflink pool - *-dirty.img are connected to Xen:

    - /var/lib/testpool/appvms/foo/volatile-dirty.img
    - /var/lib/testpool/appvms/foo/root-dirty.img
    - /var/lib/testpool/appvms/foo/root.img
    - /var/lib/testpool/appvms/foo/private-dirty.img
    - /var/lib/testpool/appvms/foo/private.img
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T03:04:05Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T04:05:06Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T05:06:07Z
    - /var/lib/testpool/appvms/bar/...
    - /var/lib/testpool/appvms/...
    - /var/lib/testpool/template-vms/fedora-26/...
    - /var/lib/testpool/template-vms/...

It looks similar to a 'file' pool tree, and in fact file-reflink is
drop-in compatible:

    $ qvm-shutdown --all --wait
    $ systemctl stop qubesd
    $ sed 's/ driver="file"/ driver="file-reflink"/g' -i.bak /var/lib/qubes/qubes.xml
    $ systemctl start qubesd
    $ sudo rm -f /path/to/pool/*/*/*-cow.img*

If the user tries to create a fresh file-reflink pool on a filesystem
that doesn't support reflinks, qvm-pool will abort and mention the
'setup_check=no' option. Which can be passed to force a fallback on
regular sparse copies, with of course lots of time/space overhead. The
same fallback code is also used when initially cloning a VM from a
foreign pool, or from another file-reflink pool on a different
mountpoint.

'journalctl -fu qubesd' will show all file-reflink copy/rename/remove
operations on VM creation/startup/shutdown/etc.
2018-02-12 21:20:05 +00:00
Rusty Bird
7a75e1090d
Make AppVM/DispVM root volume rw to avoid CoW-on-CoW 2018-02-12 21:20:04 +00:00
Marek Marczykowski-Górecki
340b8dbfe2
tests: add a test for removing expired firewall rules
QubesOS/qubes-issues#1173
2018-02-07 02:48:12 +01:00
Marek Marczykowski-Górecki
4e314c9792
tests: storage/lvm resize
QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Wojtek Porczyk
8083d6dece qubes/tests: fix QubesTestCase.qrexec_policy
The context manager left empty file.

fixes QubesOS/qubes-issues#3535
2018-02-04 14:03:12 +01:00
Yassine Ilmi
65d5c41fdb
Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90 2018-02-01 01:03:05 +00:00
Yassine Ilmi
a0d45aac9c
replaced underscore by dash and update test accordingly 2018-02-01 00:50:42 +00:00
Yassine Ilmi
0eb978c59b
Updated assert content for test_620_qdb_standalone in TC_90_QubesVM 2018-02-01 00:33:12 +00:00
Marek Marczykowski-Górecki
2c2b89e2e0
tests: adjust for type=pvh change 2018-01-27 01:34:41 +01:00
Marek Marczykowski-Górecki
99134efc3c
tests: set virt_mode='hvm' for PCI tests 2018-01-26 23:49:11 +01:00
Marek Marczykowski-Górecki
eb846f6647
Merge remote-tracking branch 'qubesos/pr/187'
* qubesos/pr/187:
  Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists
  Make 'qvm-volume revert' really use the latest revision
  Fix wrong mocks of Volume.revisions
2018-01-22 15:39:13 +01:00
Marek Marczykowski-Górecki
e577de7d49
Merge remote-tracking branch 'qubesos/pr/186'
* qubesos/pr/186:
  tests: check if udev do not touch VM's volumes
2018-01-22 15:33:50 +01:00
Marek Marczykowski-Górecki
74eb3f3208
Merge remote-tracking branch 'qubesos/pr/185'
* qubesos/pr/185:
  vm: remove doc for non-existing event `monitor-layout-change`
  vm: include tag/feature name in event name
  events: add support for wildcard event handlers
2018-01-22 15:32:57 +01:00
Rusty Bird
4ae854fdaf
Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists 2018-01-21 22:28:47 +00:00
Rusty Bird
bf1f1ac5ff
Fix wrong mocks of Volume.revisions
It's a dict, not a list.
2018-01-20 23:20:22 +00:00
Marek Marczykowski-Górecki
ab15d43622
tests: check if udev do not touch VM's volumes
Based on shell version by @rustybird
2018-01-19 18:05:50 +01:00
Marek Marczykowski-Górecki
b245bbca6f
tests: update PCI devices tests for core3 API
- use asyncio where needed
- attach now takes DeviceAssignment, not DeviceInfo
- PCI ident have ':' replaced with '_'
2018-01-18 17:36:38 +01:00
Marek Marczykowski-Górecki
edbfd3843e
tests: make tests.extra.VMWrapper hashable
Allow using VMWrapper as dict key, same as QubesVM.
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
f2b9be3607
tests: one more missing virt_mode=hvm 2018-01-17 15:23:22 +01:00
Marek Marczykowski-Górecki
4d59f883a0
tests: minor fixes
- FD leak
- switch to xterm to test also on minimal template
2018-01-16 21:41:38 +01:00
Marek Marczykowski-Górecki
06e82eccb0
tests: add run_service and qrexec_policy wrappers to ExtraTestCase
Provide same API as in core2, especially without exposing asyncio
usage. This allows qubes-usb-proxy and qubes-split-gpg tests to run.
2018-01-16 21:39:22 +01:00
Marek Marczykowski-Górecki
c17b634913
tests: clear PCIDevice cache after each test
This is yet another place where references to VM objects contribute to
object leaks.
2018-01-16 21:32:15 +01:00
Marek Marczykowski-Górecki
d2a7cbb83e
tests: mock vmm.xs
Now it is needed by some unit tests (those calling create_qdb_entries).
2018-01-15 15:58:34 +01:00
Marek Marczykowski-Górecki
ae7031fe7e
tests: explicitly set virt_mode to HVM when needed
HVM is no longer default
2018-01-15 15:57:54 +01:00
Marek Marczykowski-Górecki
f1a5ca64fd
Merge remote-tracking branch 'qubesos/pr/180'
* qubesos/pr/180:
  vm/qubesvm: default to PVH unless PCI devices are assigned
  vm/qubesvm: expose 'start_time' property over Admin API
  vm/qubesvm: revert backup_timestamp to '%s' format
  doc: link qvm-device man page for qvm-block, qvm-pci, qvm-usb
2018-01-15 04:22:28 +01:00
Marek Marczykowski-Górecki
85e80f2329
vm/qubesvm: revert backup_timestamp to '%s' format
Human readable format `str(datetime.datetime)` is a nightmare for Admin
API level communication. Especially setting the property in a format
that it was read was not supported, and handling such format in
untrusted input handling code is a bad idea. Revert to a simple intiger
format.
2018-01-12 05:34:45 +01:00
Marek Marczykowski-Górecki
50d34755fa
vm: include tag/feature name in event name
Rename events:
 - domain-feature-set -> domain-feature-set:feature
 - domain-feature-delete -> domain-feature-delete:feature
 - domain-tag-add -> domain-tag-add:tag
 - domain-tag-delete -> domain-tag-delete:tag

Make it consistent with property-* events. It makes more sense to
include tag/feature name in event name, so handler can watch a single
tag/feature - which is the most common case. Otherwise, most handlers
would begin with `if feature == '...'` anyway, wasting time on most
events.

In cases where multiple features/tags should be handled by a single
handler, it is now possible to register a handler with wildcard, for
example `domain-feature-set:*`.
2018-01-06 15:05:34 +01:00
Marek Marczykowski-Górecki
5a39e77708
events: add support for wildcard event handlers
Support registering handlers for more flexible wildcard events: not only
'*', but also 'something*'. This allows to register handlers for
'property-set:*' and such.
2018-01-06 00:40:19 +01:00
Marek Marczykowski-Górecki
3fa0972317
tests: fix (system) network tests after switching to ipaddress module 2017-12-23 16:43:24 +01:00
Marek Marczykowski-Górecki
3668a73ca2
tests: resurrect extra tests loader
Load integration tests from outside of core-admin repository, through
entry points.
Create wrapper for VM object to keep very basic compatibility with tests
written for core2. This means if test use only basic functionality
(vm.start(), vm.run()), the same test will work for both core2 and
core3. This is especially important for app-* repositories, where the
same version serves multiple Qubes branches.
This also hides asyncio usage from tests writer.

See QubesOS/qubes-issues#1800 for details on original feature.
2017-12-23 02:26:16 +01:00
Marek Marczykowski-Górecki
962742880f
tests: basic salt integration tests
Test base functions of dom0 module (creating VM, setting property) and
configuring system inside of VM (through DispVM). The later is done for
each available template (the process use salt installed in that
template, not copied from dom0).

QubesOS/qubes-issues#3316
2017-12-23 02:26:15 +01:00
Marek Marczykowski-Górecki
9a4d2abf1f
tests: booting VM from ISO image
Check two cases: if ISO image is in dom0 and if its in a VM.

QubesOS/qubes-issues#3339
2017-12-14 23:26:53 +01:00
Marek Marczykowski-Górecki
466bf89aae
Fix starting VM with kernel=None
When dom0 do not provide the kernel, it should also not set kernel
command line in libvirt config. Otherwise qemu in stubdom fails to start
because it get -append option without -kernel, which is illegal
configuration.

Fixes QubesOS/qubes-issues#3339
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
1187e43697
tests: more vm.create_qdb_entries() tests 2017-12-14 02:09:31 +01:00
Marek Marczykowski-Górecki
e12a66f103
vm/mix/net: use ipaddress module for ip and ip6 properties
It has built-in validation, which is much more elegant than custom regex
or socket call.

Suggested by @woju
QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
f3cf58e6f2
tests: add integration tests for IPv6
Run also all IPv4 tests with IPv6 enabled to check for regressions
(broken IPv4 because of enabled IPv6).

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
0786edf8a3
tests: add IPv6-related unit tests
Check produced libvirt XML, and QubesDB entries

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
18f159f8ec
Add IPv6 related VM properties
Add property for IPv6 address ('ip6'). Build default value similarly to
IPv4 - common prefix + QID or Disp ID (for DispVMs).
This all is disabled unless 'ipv6' feature is enabled. It is inherited
from netvm (not template).
Even when enabled, VM may decide to not use it - or simply not support
it.

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
bf59b00f1d
features: add check_with_netvm, similar to check_with_template
Allow using default feature value from netvm, not template. This makes
sense for network-related features like using tor, supporting ipv6 etc.

Similarly to check_with_template, expose it also on Admin API.
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
f223594f92
app: kill default_fw_netvm property
Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)

As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.

Fixes QubesOS/qubes-issues#3247
2017-12-07 01:40:29 +01:00
Marek Marczykowski-Górecki
f2cd7fb226
Merge branch 'tests-and-fixes-20171205' 2017-12-07 01:39:34 +01:00
Marek Marczykowski-Górecki
7b81d7affa
Merge remote-tracking branch 'qubesos/pr/166'
* qubesos/pr/166:
  create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
  change default pool code to be fast
  cache PropertyHolder.property_list and use O(1) property name lookups
  remove unused netid code
  cache isinstance(default, collections.Callable)
  don't access netvm if it's None in visible_gateway/netmask
2017-12-06 00:41:37 +01:00
Marek Marczykowski-Górecki
4d6bfbab4d
tests: improve spoof_ip test
Not only check if full round trip ping (does not) work, but also if just
echo-request get filtered.
2017-12-05 17:40:30 +01:00
Marek Marczykowski-Górecki
379add52ba
tests: skip network tests on whonix-gw and whonix-ws
whonix-ws also have non-standard firewall and require specific tests for
that.
2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
088c255355
tests: add create_qdb_entries() unit test 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
0afee4b05e
Improve checking for netvm loop
There were many cases were the check was missing:
 - changing default_netvm
 - resetting netvm to default value
 - loading already broken qubes.xml

Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...

Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
19a1579a99
tests: fix deadlock in filecopy test
Error window (where test need to send Enter key) is opened while
qvm-move-to-vm is still running.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
99874a0a25
tests: make waiting for window asyncio aware
For now just replace sleep with asyncio.sleep. Later it may make sense
to change subprocess.call too.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
a92dd99fbb
Merge branch '20171107-storage'
* 20171107-storage:
  api/admin: add API for changing revisions_to_keep dynamically
  storage/file: move revisions_to_keep restrictions to property setter
  api/admin: hide dd statistics in admin.vm.volume.Import call
  storage/lvm: fix importing different-sized volume from another pool
  storage/file: fix preserving spareness on volume clone
  api/admin: add pool size and usage to admin.pool.Info response
  storage: add size and usage properties to pool object
2017-11-20 22:52:50 +01:00
Marek Marczykowski-Górecki
2c6c766968
Merge branch '20171107-tests-backup-api-misc'
* 20171107-tests-backup-api-misc:
  test: make race condition on xterm close less likely
  tests/backupcompatibility: fix handling 'internal' property
  backup: fix handling target write error (like no disk space)
  tests/backupcompatibility: drop R1 format tests
  backup: use offline_mode for backup collection
  qubespolicy: fix handling '$adminvm' target with ask action
  app: drop reference to libvirt object after undefining it
  vm: always log startup fail
  api: do not log handled errors sent to a client
  tests/backups: convert to new restore handling - using qubesadmin module
  app: clarify error message on failed domain remove (used somewhere)
  Fix qubes-core.service ordering
2017-11-20 22:52:26 +01:00
Marek Marczykowski-Górecki
cf92a576ad
Merge remote-tracking branch 'qubesos/pr/159'
* qubesos/pr/159:
  qubes/vm: Improve stopped event handling
2017-11-20 22:49:05 +01:00
Marek Marczykowski-Górecki
3217c3ac4e
Merge remote-tracking branch 'qubesos/pr/156'
* qubesos/pr/156:
  tests: Add unit tests for pvh virt_mode
  Don't allow attached PCI devices and virt_mode = 'pvh'
2017-11-20 22:40:53 +01:00
qubesuser
f2b8ad7d38 remove unused netid code
it's unused and has a netid property with name different than key
that would cause issues in the next commit
2017-11-11 02:37:37 +01:00
Marek Marczykowski-Górecki
45af0913f2
test: make race condition on xterm close less likely
xterm is very fast on closing when application inside terminates. It is
so fast with closing on keydown event that xdotool do not manage to send
keyup event, resulting in xdotool crash. Add a little more time for
that.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
07d4f4b340
tests/backupcompatibility: fix handling 'internal' property 2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
11fd2cf115
tests/backupcompatibility: drop R1 format tests
It is no longer supported.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
31a55dcd18
tests/backups: convert to new restore handling - using qubesadmin module
Besides converting itself, change how the test verify restore
correctness: first collect VM metadata (and hashes of data) into plain
dict, then compare against it. This allow to destroy old VMs objects
before restoring the backup, so avoid having duplicate objects of the
same VM - which results in weird effects like trying to undefine libvirt
object twice.
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
c3afdde3ef
api/admin: add API for changing revisions_to_keep dynamically
This one pool/volume property makes sense to change dynamically. There
may be more such properties, but lets be on the safe side and take
whitelist approach - allow only selected (just one for now), instead of
blacklisting any harmful ones.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
81f455e15d
storage/file: move revisions_to_keep restrictions to property setter
Do not check for accepted value only in constructor, do that in property
setter. This will allow enforcing the limit regardless of how the value
was set.

This is preparation for dynamic revisions_to_keep change.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
f18f4c9bff
api/admin: add pool size and usage to admin.pool.Info response
QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
e76372b934
storage: add size and usage properties to pool object
Add Pool.size and Pool.usage to the API. Implement them for LVM and File
pools. Add appropriate tests.

QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
HW42
c695f3268c linux-stubdom: Set cmdline in HVM config 2017-10-30 16:19:01 +01:00
HW42
b907303a39 qubes/vm: Improve stopped event handling
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.

This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.

Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.

Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
2017-10-21 05:57:57 +02:00
Marek Marczykowski-Górecki
e2eaa57f65
Merge branch 'tests-20171020'
* tests-20171020:
  tests: fix asyncio usage some more places
  tests: fix reporting in network tests
  tests: domain-shutdown event race condition
  tests: improve events tests
  tests: fix logic error in wait_for_window()
  tests: cleanup QubesDB connection on domain remove
2017-10-21 04:33:35 +02:00
Marek Marczykowski-Górecki
42f8619559
tests: fix asyncio usage some more places
gui_memory_pinning tests finally works. Also, fix DispVM and private
volume resize tests.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
511502ae7e
tests: fix reporting in network tests
Do not trash stdout with VM's command output. Log it as part of
(potential) failure message.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
c3430fe5f6
tests: domain-shutdown event race condition
Second tests, for actual events, not storage handling done by its
handler.

cc @HW42

QubesOS/qubes-issues#3164
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
96f2e503db
tests: improve events tests
Check property-related events, including their arguments.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
71fc5217f7
tests: fix logic error in wait_for_window()
Revert change introduced in cbc5dbb9b1
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
43c09467b1
tests: cleanup QubesDB connection on domain remove
If domain got removed during the tests (for example DispVM), vm.close()
wouldn't be called in cleanup and some file descriptors will be
leaked. Add event handler for cleaning this up. Do not use close()
method here, because it is destructive, but the object may still be used
by the test.
2017-10-21 04:06:14 +02:00
Marek Marczykowski-Górecki
8f1bfa65f8
tests: fix typo 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
4809c16e08
admin-api: add admin.*.property.GetDefault
Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
038ca6d5d7
base: add method for getting default property value
Allow to get default value even it isn't set currently. This will allow
(G)UI to present better view, without duplicating logic for default
value.

Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
85c3368a0d
comments fixes 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
c2c3225622
tests: root fs resize tests - for all templates
Check resize on each template separately, because it involves VM's
scripts (either qubes.ResizeDisk service, or some startup script).

QubesOS/qubes-issues#3173
2017-10-21 04:05:14 +02:00
Marek Marczykowski-Górecki
77ce956c2d
tests: improve root volume resize tests
- clone all features, not just qrexec (especially include 'gui')
- do not leak VM reference on failed test
- add test for online root volume resize

QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
27b96cce4c
firewall: skip expired rules
Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).

Related QubesOS/qubes-issues#3020
2017-10-21 03:35:00 +02:00
Marek Marczykowski-Górecki
836d9f902a
Merge branch 'bug3164'
* bug3164:
  tests: add regression test for #3164
  storage/lvm: make sure volume cache is refreshed after changes
  storage/lvm: fix Volume.verify()
  storage/lvm: remove old volume only after successfully cloning new one
2017-10-16 04:28:26 +02:00
Marek Marczykowski-Górecki
c8519a700f
tests: add regression test for #3164
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.

QubesOS/qubes-issues#3164
2017-10-16 00:48:25 +02:00
Marek Marczykowski-Górecki
2164a8d7b8
Change license to LGPL v2.1+
See this thread for reasoning and acceptance from contributors:
https://groups.google.com/d/topic/qubes-devel/G7KzrfU0lWY/discussion
"Changing qubes-core-admin license to LGPL v2.1+"
2017-10-12 00:11:50 +02:00
HW42
4a84d5b222 tests: Add unit tests for pvh virt_mode 2017-10-11 21:11:36 +02:00
Marek Marczykowski-Górecki
af60fbb25d
tests: dispvm tests fixes 2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
6990550884
tests: more tests fixes, drop expectedFailure where not needed anymore 2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
e0be7861f9
tests: extend qrexec_policy context manager - custom action
Allow to use custom action - for example with 'target=' argument.
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
cbc5dbb9b1
tests: various fixes
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
  cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
64c5b2c00f
tests: cleanup DispVMs created during tests
Since those do not have specific prefix (test-) compare list of them
with original qubes.xml
2017-10-04 15:24:51 +02:00
Marek Marczykowski-Górecki
3a5e8482c0
Merge branch 'fixes-20170929'
* fixes-20170929:
  vm: do not start QubesDB watch instance multiple times
  vm: report storage.stop() errors to log
  vm: move comment
  storage: fix method name in LinuxModules volume
  Prevent removing domain that is referenced from anywhere
  vm: add vm.klass property
  Move QubesVM.{name,qid,uuid,label} to BaseVM
  vm: do not allow deleting template property from AppVM and DispVM
  vm/qubesvm: emit event on failed startup
  vm/qubesvm: remove duplicated qmemman_client.close()
  vm/dispvm: cleanup DispVM also on failed startup
  vm/dispvm: fix error message
  ext/block: properly list devtype=cdrom option
  block: fix handling non-existing devices
  block: improve handling device name and description
2017-10-03 12:04:00 +02:00
Marek Marczykowski-Górecki
12b7e22d27
vm: do not start QubesDB watch instance multiple times
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
2017-09-29 12:05:23 +02:00
Marek Marczykowski-Górecki
9172a7708d
tests: fix various object leaks cont
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
f9da1f2dd7
tests: various fixes for networking tests
- Update feature name for enabling network-manager.
- Fix error reporting
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
fdf780aad0
tests/network: match vm names with variable names
Having testvm1 variable for "vm2" is really confusing.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
8b34b9a5e4
tests: hide unneeded messages 2017-09-29 11:53:59 +02:00