Commit Graph

5912 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
576bcb158e
tests: skip tests not relevant on Whonix 2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
556a08cb78
tests: improve shutdown timeout handling
Instead of waiting 1sec, wait up to 5sec but skip when vm is shut off.
This fix tests on slow machines, including openQA nested virt.
2018-09-15 05:12:41 +02:00
Marek Marczykowski-Górecki
5a1bf11d0d
tests: drop qvm-prefs tests
Those are moved to qubes-core-admin-client repository.
2018-09-15 05:12:41 +02:00
Rusty Bird
5756e870bd
storage/reflink: use context managers in is_supported()
Don't rely on garbage collection to close and remove the tempfiles.
2018-09-13 19:46:48 +00:00
Rusty Bird
867baf47d1
api/admin: fix typo 2018-09-13 19:46:46 +00:00
Rusty Bird
b3983f5ef8
'except FileNotFoundError' instead of ENOENT check 2018-09-13 19:46:45 +00:00
AJ Jordan
5aa35a1208
Make log location more explicit in error message
See https://github.com/QubesOS/qubes-issues/issues/4224#issuecomment-414513721.
2018-09-12 22:40:21 -04:00
Marek Marczykowski-Górecki
4dab769934
Merge remote-tracking branch 'qubesos/pr/231'
* qubesos/pr/231: (30 commits)
  tests/app: test varlibqubes pool driver selection
  tests/storage_reflink: test some file-reflink helpers
  tests/integ/storage: add file-reflink integration tests
  tests/integ/basic: use export() in get_rootimg_checksum()
  tests/integ/backupcompatibility: Storage.verify() is a coro
  tests: delete orphaned Makefile
  app: create /var/lib/qubes as file-reflink if supported
  app: uncouple pool setup from loading initial configuration
  tools/qubes-create: fix docstring
  storage: factor out _wait_and_reraise(); fix clone/create
  storage: fix search_pool_containing_dir()
  storage: remove broken default parameter from isodate()
  storage: insert missing NotImplementedError in Volume.stop()
  storage: fix docstrings
  storage/reflink: is_reflink_supported() -> is_supported()
  storage/reflink: run synchronous volume methods in executor
  storage/reflink: native FICLONE in _copy_file() happy path
  storage/reflink: factor out _ficlone()
  storage/reflink: inline and simplify _cmd()
  storage/reflink: _update_loopdev_sizes() without losetup
  ...
2018-09-12 02:42:15 +02:00
Rusty Bird
cf1ea5cee1
tests/app: test varlibqubes pool driver selection 2018-09-11 23:50:33 +00:00
Rusty Bird
797bbc43a0
tests/storage_reflink: test some file-reflink helpers
Tested:

    - _copy_file()
    - _create_sparse_file()
    - _resize_file()
    - _update_loopdev_sizes()

Smoke tested by calls from the functions above:

    - _replace_file()
    - _rename_file()
    - _make_dir()
    - _fsync_dir()
2018-09-11 23:50:32 +00:00
Rusty Bird
b82e739346
tests/integ/storage: add file-reflink integration tests 2018-09-11 23:50:31 +00:00
Rusty Bird
8c117549ad
tests/integ/basic: use export() in get_rootimg_checksum()
volume.path and volume.export() refer to the same thing in lvm_thin and
'file', but not in file-reflink (where volume.path is the -dirty.img,
which doesn't exist if the volume is not started).
2018-09-11 23:50:30 +00:00
Rusty Bird
49e7ce025f
tests/integ/backupcompatibility: Storage.verify() is a coro 2018-09-11 23:50:28 +00:00
Rusty Bird
bc30c6f3e8
tests: delete orphaned Makefile 2018-09-11 23:50:27 +00:00
Rusty Bird
8d1913a8cc
app: create /var/lib/qubes as file-reflink if supported
Use the file-reflink storage driver if /var/lib/qubes is on a filesystem
that supports reflinks, e.g. when the btrfs layout was selected in
Anaconda. If it doesn't support reflinks (or if detection fails, e.g. in
an unprivileged test environment), use 'file' as before.
2018-09-11 23:50:26 +00:00
Rusty Bird
53ef5ed431
app: uncouple pool setup from loading initial configuration
And ensure that setup is called on every type of these pools, not just
lvm_thin.
2018-09-11 23:50:25 +00:00
Rusty Bird
8eb9c64f20
tools/qubes-create: fix docstring 2018-09-11 23:50:24 +00:00
Rusty Bird
d181bf1aa4
storage: factor out _wait_and_reraise(); fix clone/create
_wait_and_reraise() is similar to asyncio.gather(), but it preserves the
current behavior of waiting for all futures and only _then_ reraising
the first exception (if there is any) in line.

Also switch Storage.create() and Storage.clone() to _wait_and_reraise().
Previously, they called asyncio.wait() and implicitly swallowed all
exceptions.
2018-09-11 23:50:22 +00:00
Rusty Bird
d33bd3f2b6
storage: fix search_pool_containing_dir()
Canonicalize both directories, resolving symlink components. Compare
with commonpath() instead of startswith(), because /foo doesn't contain
/foobar.
2018-09-11 23:50:21 +00:00
Rusty Bird
f0ee73e63f
storage: remove broken default parameter from isodate()
With that syntax, the default timestamp would have been from the time of
the function's definition (not invocation). But all callers are passing
an explicit timestamp anyway.
2018-09-11 23:50:20 +00:00
Rusty Bird
44ca78523f
storage: insert missing NotImplementedError in Volume.stop() 2018-09-11 23:50:19 +00:00
Rusty Bird
266d90c2f9
storage: fix docstrings 2018-09-11 23:50:18 +00:00
Rusty Bird
c75fe09814
storage/reflink: is_reflink_supported() -> is_supported() 2018-09-11 23:50:16 +00:00
Rusty Bird
1889c9b75f
storage/reflink: run synchronous volume methods in executor
Convert create(), verify(), remove(), start(), stop(), revert(),
resize(), and import_volume() into coroutine methods, via a decorator
that runs them in the event loop's thread-based default executor.

This reduces UI hangs by unblocking the event loop, and can e.g. speed
up VM starts by starting multiple volumes in parallel.
2018-09-11 23:50:15 +00:00
Rusty Bird
3d986be02a
storage/reflink: native FICLONE in _copy_file() happy path
Avoid a subprocess launch, and distinguish reflink vs. fallback copy in
the log.
2018-09-11 23:50:14 +00:00
Rusty Bird
edda3a1734
storage/reflink: factor out _ficlone() 2018-09-11 23:50:13 +00:00
Rusty Bird
69af0a48ec
storage/reflink: inline and simplify _cmd() 2018-09-11 23:50:12 +00:00
Rusty Bird
fb06a8089a
storage/reflink: _update_loopdev_sizes() without losetup
Factor out a function, and use the LOOP_SET_CAPACITY ioctl instead of
going through losetup.
2018-09-11 23:50:10 +00:00
xaki23
ce794f33d8
add missing /sbin/ to hwclock call (so it will work as cronjob) 2018-09-10 17:24:35 +02:00
Rusty Bird
385ba91772
storage/reflink: resize(): don't look for loopdevs if clean 2018-09-09 20:01:22 +00:00
Rusty Bird
e7b7c253ac
storage/reflink: inline _require_self_on_stop() 2018-09-09 20:01:20 +00:00
Rusty Bird
6e8d7d4201
storage/reflink: no-op import_volume() if not save_on_stop
Instead of raising a NotImplementedError, just return self like 'file'
and lvm_thin. This is needed when Storage.clone() is modified in another
commit* to no longer swallow exceptions.

* "storage: factor out _wait_and_reraise(); fix clone/create"
2018-09-09 20:01:19 +00:00
Rusty Bird
60bf68a748
storage/reflink: add _path_import (don't reuse _path_dirty)
Import volume data to a new _path_import (instead of _path_dirty) before
committing to _path_clean. In case the computer crashes while an import
operation is running, the partially written file should not be attached
to Xen on the next volume startup.

Use <name>-import.img as the filename like 'file' does, to be compatible
with qubes.tests.api_admin/TC_00_VMs/test_510_vm_volume_import.
2018-09-09 20:01:18 +00:00
Rusty Bird
d301aa2e50
storage/reflink: delete stale tempfiles on start and remove
When the AT_REPLACE flag for linkat() finally lands in the Linux kernel,
_replace_file() can be modified to use unnamed (O_TMPFILE) tempfiles.
Until then, make sure stale tempfiles from previous crashes can't hang
around for too long.
2018-09-09 20:01:17 +00:00
Rusty Bird
75a4a1340e
storage/reflink: don't recompute static properties per call 2018-09-09 20:01:15 +00:00
Rusty Bird
ef2698adb4
storage/reflink: make revisions() more readable, use iglob 2018-09-09 20:01:14 +00:00
Rusty Bird
18f9356c2c
storage/reflink: refuse to revert() dirty volume 2018-09-09 20:01:13 +00:00
Rusty Bird
677183d8a6
storage/reflink: add revision even if empty
It's sort of useful to be able to revert a volume that has only ever
been started once to its empty state. And the lvm_thin driver allows it
too, so why not.
2018-09-09 20:01:12 +00:00
Rusty Bird
850778b52a
storage/reflink: remove redundant format specifiers 2018-09-09 20:01:11 +00:00
Marek Marczykowski-Górecki
a371375f1d
version 4.0.29 2018-09-09 06:38:16 +02:00
Marek Marczykowski-Górecki
b2cc605f4b
tests: clean local variables from traceback objects
System tests are fragile for any object leaks, especially those holding
open files. Instead of wrapping all tests with try/finally removing
those local variables (as done in qubes.tests.integ.backup for example),
apply generic solution: clean all traceback objects from local
variables. Those aren't used to generate text report by either test
runner (qubes.tests.run and nose2). If one wants to break into debugger
and inspect tracebacks interactively, needs to comment out call to
cleanup_traceback.
2018-09-09 02:43:17 +02:00
Marek Marczykowski-Górecki
c102fa3d68
tests: add basic audio play/rec tests
QubesOS/qubes-issues#4204
2018-09-08 04:13:24 +02:00
Marek Marczykowski-Górecki
e8d2c4e232
Merge remote-tracking branch 'qubesos/pr/213'
* qubesos/pr/213:
  Fix from Marek's revies
  doc: loading graph
2018-09-07 21:35:44 +02:00
Wojtek Porczyk
b200dc8df2
Fix from Marek's revies
- add load_extras()
- change callouts to arrows for draw.io limitations

Cc: @marmarek
2018-09-07 19:44:29 +02:00
Marek Marczykowski-Górecki
8ce3433406
tests: drop sudo in tests already running as root
Don't spam already trashed log.
2018-09-07 15:12:05 +02:00
Marek Marczykowski-Górecki
1b99bd9ab3
Merge branch 'tests-dispvm-fix'
* tests-dispvm-fix:
  tests: fix DispVM related tests
2018-09-07 01:57:19 +02:00
Marek Marczykowski-Górecki
6a3ecd0170
Merge remote-tracking branch 'qubesos/pr/229'
* qubesos/pr/229:
  Order qubesd before systemd-user-sessions
2018-09-07 01:07:27 +02:00
Marek Marczykowski-Górecki
7f1e2741ec
Merge remote-tracking branch 'qubesos/pr/228'
* qubesos/pr/228:
  storage/lvm: filter out warning about intended over-provisioning
  tests: fix getting kernel package version inside VM
  tests/extra: add start_guid option to VMWrapper
  vm/qubesvm: fire 'domain-start-failed' event even if fail was early
  vm/qubesvm: check if all required devices are available before start
  storage/lvm: fix reporting lvm command error
  storage/lvm: save pool's revision_to_keep property
2018-09-07 01:06:59 +02:00
Rusty Bird
d3a5799245
Order qubesd before systemd-user-sessions
qubes-vm@.service would already cause this ordering, but not every user
has any autostart=True VMs.

Also needed to maybe f*x QubesOS/qubes-issues#3149 at some point.
2018-09-06 16:23:24 +00:00
Marek Marczykowski-Górecki
e1378f70fb
doc: fix libvirt config path 2018-09-06 01:16:50 +02:00