Commit Graph

1520 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
f2b9be3607
tests: one more missing virt_mode=hvm 2018-01-17 15:23:22 +01:00
Marek Marczykowski-Górecki
7905783861
qubesvm: PVH minor improvements
- use capital letters in acronyms in documentation to match upstream
documentation.
- refuse to start a PVH with without kernel set - provide meaningful
error message
2018-01-16 21:42:20 +01:00
Marek Marczykowski-Górecki
4d59f883a0
tests: minor fixes
- FD leak
- switch to xterm to test also on minimal template
2018-01-16 21:41:38 +01:00
Marek Marczykowski-Górecki
06e82eccb0
tests: add run_service and qrexec_policy wrappers to ExtraTestCase
Provide same API as in core2, especially without exposing asyncio
usage. This allows qubes-usb-proxy and qubes-split-gpg tests to run.
2018-01-16 21:39:22 +01:00
Marek Marczykowski-Górecki
c17b634913
tests: clear PCIDevice cache after each test
This is yet another place where references to VM objects contribute to
object leaks.
2018-01-16 21:32:15 +01:00
Marek Marczykowski-Górecki
d2a7cbb83e
tests: mock vmm.xs
Now it is needed by some unit tests (those calling create_qdb_entries).
2018-01-15 15:58:34 +01:00
Marek Marczykowski-Górecki
ae7031fe7e
tests: explicitly set virt_mode to HVM when needed
HVM is no longer default
2018-01-15 15:57:54 +01:00
Marek Marczykowski-Górecki
241f1d1d3b
tests: do not leak open file 2018-01-15 15:57:30 +01:00
Marek Marczykowski-Górecki
f1a5ca64fd
Merge remote-tracking branch 'qubesos/pr/180'
* qubesos/pr/180:
  vm/qubesvm: default to PVH unless PCI devices are assigned
  vm/qubesvm: expose 'start_time' property over Admin API
  vm/qubesvm: revert backup_timestamp to '%s' format
  doc: link qvm-device man page for qvm-block, qvm-pci, qvm-usb
2018-01-15 04:22:28 +01:00
Marek Marczykowski-Górecki
21760d8ff0
Merge remote-tracking branch 'qubesos/pr/179'
* qubesos/pr/179:
  qmemman: request VMs balloon down with 16MB safety margin
  qmemman: clear "not responding" flags when VM require more memory
  qmemman: slightly improve logging
  qmemman: reformat code, especially comments
2018-01-15 04:21:40 +01:00
Marek Marczykowski-Górecki
b20c3d3458
Merge remote-tracking branch 'qubesos/pr/174'
* qubesos/pr/174:
  tests: fix (system) network tests after switching to ipaddress module
  tests: resurrect extra tests loader
  tests: basic salt integration tests
2018-01-15 04:20:32 +01:00
Marek Marczykowski-Górecki
4ff53879a0
vm/qubesvm: default to PVH unless PCI devices are assigned
Fixes QubesOS/qubes-issues#2185
2018-01-15 03:34:46 +01:00
Marek Marczykowski-Górecki
d9da747ab0
vm/qubesvm: expose 'start_time' property over Admin API
It is useful at least for Qubes Manager.
2018-01-12 05:34:46 +01:00
Marek Marczykowski-Górecki
85e80f2329
vm/qubesvm: revert backup_timestamp to '%s' format
Human readable format `str(datetime.datetime)` is a nightmare for Admin
API level communication. Especially setting the property in a format
that it was read was not supported, and handling such format in
untrusted input handling code is a bad idea. Revert to a simple intiger
format.
2018-01-12 05:34:45 +01:00
Marek Marczykowski-Górecki
bcf42c13fa
storage/lvm: check for LVM LV existence and type when creating ThinPool
Check if requested thin pool exists and really is thin pool.

QubesOS/qubes-issues#3438
2018-01-12 05:12:08 +01:00
Marek Marczykowski-Górecki
377f331d52
storage/lvm: fix size reporting just after creating LV
Force cache refresh after registering new pool - it might be just
created.

QubesOS/qubes-issues#3438
2018-01-11 03:56:30 +01:00
Marek Marczykowski-Górecki
4bca631350
qmemman: request VMs balloon down with 16MB safety margin
It looks like Linux balloon driver do not always precisely respect
requested target memory, but perform some rounding. Also, in some cases
(HVM domains), VM do not see all the memory that Xen have assigned to it
- there are some additional Xen pools for internal usage.
Include 16MB safety margin in memory requests to account for those two
things. This will avoid setting "no_response" flag for most of VMs.

QubesOS/qubes-issues#3265
2018-01-11 03:41:55 +01:00
Marek Marczykowski-Górecki
bf4306b815
qmemman: clear "not responding" flags when VM require more memory
Clear slow_memset_react/no_progress flags when VM request more memory
than it have assigned. If there is some available, it may be given to
such VM, solving the original problem (not reacting to balloon down
request). In any case, qmemman algorithm should not try to take away
memory from under-provisioned VM.

Fixes QubesOS/qubes-issues#3265
2018-01-11 03:41:54 +01:00
Marek Marczykowski-Górecki
4cf6a93b5b
qmemman: slightly improve logging
Add logging more info about each domain state:
 - last requested target
 - no_progress and slow_memset_react flags

This makes it unnecessary to log separately when those flags are cleared.
2018-01-11 03:41:54 +01:00
Marek Marczykowski-Górecki
8e288d9f81
qmemman: reformat code, especially comments
Indent comments to match code indentation to make it readable. Also,
wrap long lines. Fix few typos in comments.

No functional change.
2018-01-07 17:04:25 +01:00
Marek Marczykowski-Górecki
f0fe02998b
vm: remove doc for non-existing event monitor-layout-change 2018-01-06 15:10:54 +01:00
Marek Marczykowski-Górecki
50d34755fa
vm: include tag/feature name in event name
Rename events:
 - domain-feature-set -> domain-feature-set:feature
 - domain-feature-delete -> domain-feature-delete:feature
 - domain-tag-add -> domain-tag-add:tag
 - domain-tag-delete -> domain-tag-delete:tag

Make it consistent with property-* events. It makes more sense to
include tag/feature name in event name, so handler can watch a single
tag/feature - which is the most common case. Otherwise, most handlers
would begin with `if feature == '...'` anyway, wasting time on most
events.

In cases where multiple features/tags should be handled by a single
handler, it is now possible to register a handler with wildcard, for
example `domain-feature-set:*`.
2018-01-06 15:05:34 +01:00
Marek Marczykowski-Górecki
5a39e77708
events: add support for wildcard event handlers
Support registering handlers for more flexible wildcard events: not only
'*', but also 'something*'. This allows to register handlers for
'property-set:*' and such.
2018-01-06 00:40:19 +01:00
Marek Marczykowski-Górecki
a66c9afb18
Merge remote-tracking branch 'qubesos/pr/177'
* qubesos/pr/177:
  Use default_dispvm (not default_template) for new DispVM
2018-01-05 16:29:14 +01:00
Marek Marczykowski-Górecki
d83a07177b
Merge remote-tracking branch 'qubesos/pr/176'
* qubesos/pr/176:
  Do not remove VMs installed via rpm
2018-01-05 16:27:12 +01:00
Rusty Bird
4a2fabc17f
Use default_dispvm (not default_template) for new DispVM
When creating a new VM of type DispVM without specifying any template
(e.g. "qvm-create --class DispVM --label red foo"), use default_dispvm.
Otherwise it would fail saying "Got empty response from qubesd."
2018-01-02 23:19:02 +00:00
Christopher Laprise
47b49c4755
Do not remove VMs installed via rpm 2017-12-29 23:24:41 -05:00
Marek Marczykowski-Górecki
3fa0972317
tests: fix (system) network tests after switching to ipaddress module 2017-12-23 16:43:24 +01:00
Marek Marczykowski-Górecki
3668a73ca2
tests: resurrect extra tests loader
Load integration tests from outside of core-admin repository, through
entry points.
Create wrapper for VM object to keep very basic compatibility with tests
written for core2. This means if test use only basic functionality
(vm.start(), vm.run()), the same test will work for both core2 and
core3. This is especially important for app-* repositories, where the
same version serves multiple Qubes branches.
This also hides asyncio usage from tests writer.

See QubesOS/qubes-issues#1800 for details on original feature.
2017-12-23 02:26:16 +01:00
Marek Marczykowski-Górecki
962742880f
tests: basic salt integration tests
Test base functions of dom0 module (creating VM, setting property) and
configuring system inside of VM (through DispVM). The later is done for
each available template (the process use salt installed in that
template, not copied from dom0).

QubesOS/qubes-issues#3316
2017-12-23 02:26:15 +01:00
Marek Marczykowski-Górecki
32c6083e1c
Make pylint happy
Fix thing detected by updated pylint in Travis-CI
2017-12-21 18:19:10 +01:00
Marek Marczykowski-Górecki
f5fe10e2ad
vm/adminvm: fix type of AdminVM.qid
It is defined as int in QubesVM.qid, make it consistent.
2017-12-21 18:18:12 +01:00
Marek Marczykowski-Górecki
9a4d2abf1f
tests: booting VM from ISO image
Check two cases: if ISO image is in dom0 and if its in a VM.

QubesOS/qubes-issues#3339
2017-12-14 23:26:53 +01:00
Marek Marczykowski-Górecki
96bd734852
storage/lvm: clean -snap LV on volume removal 2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
466bf89aae
Fix starting VM with kernel=None
When dom0 do not provide the kernel, it should also not set kernel
command line in libvirt config. Otherwise qemu in stubdom fails to start
because it get -append option without -kernel, which is illegal
configuration.

Fixes QubesOS/qubes-issues#3339
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
fd45378041
api/admin: make libvirt start error more informative
Point where to look for details.
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
297cb4d012
vm/mix/net: fill QubesDB for already connected VMs
There may be cases when VM providing the network to other VMs is started
later - for example VM restart. While this is rare case (and currently
broken because of QubesOS/qubes-issues#1426), do not assume it will
always be the case.
2017-12-14 23:26:51 +01:00
Marek Marczykowski-Górecki
1187e43697
tests: more vm.create_qdb_entries() tests 2017-12-14 02:09:31 +01:00
Marek Marczykowski-Górecki
97564f014c
vm/mix/net: fix setting QubesDB after converting to ipaddress module 2017-12-14 02:07:57 +01:00
Marek Marczykowski-Górecki
faef890c9a
vm/qubesvm: write QubesDB /qubes-netvm-gateway6 entry when set
This is needed for network-providing VM to actually provide IPv6
connection too.

QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
e12a66f103
vm/mix/net: use ipaddress module for ip and ip6 properties
It has built-in validation, which is much more elegant than custom regex
or socket call.

Suggested by @woju
QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
f3cf58e6f2
tests: add integration tests for IPv6
Run also all IPv4 tests with IPv6 enabled to check for regressions
(broken IPv4 because of enabled IPv6).

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
0786edf8a3
tests: add IPv6-related unit tests
Check produced libvirt XML, and QubesDB entries

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
18f159f8ec
Add IPv6 related VM properties
Add property for IPv6 address ('ip6'). Build default value similarly to
IPv4 - common prefix + QID or Disp ID (for DispVMs).
This all is disabled unless 'ipv6' feature is enabled. It is inherited
from netvm (not template).
Even when enabled, VM may decide to not use it - or simply not support
it.

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
bf59b00f1d
features: add check_with_netvm, similar to check_with_template
Allow using default feature value from netvm, not template. This makes
sense for network-related features like using tor, supporting ipv6 etc.

Similarly to check_with_template, expose it also on Admin API.
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
f223594f92
app: kill default_fw_netvm property
Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)

As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.

Fixes QubesOS/qubes-issues#3247
2017-12-07 01:40:29 +01:00
Marek Marczykowski-Górecki
f2cd7fb226
Merge branch 'tests-and-fixes-20171205' 2017-12-07 01:39:34 +01:00
Marek Marczykowski-Górecki
7b81d7affa
Merge remote-tracking branch 'qubesos/pr/166'
* qubesos/pr/166:
  create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
  change default pool code to be fast
  cache PropertyHolder.property_list and use O(1) property name lookups
  remove unused netid code
  cache isinstance(default, collections.Callable)
  don't access netvm if it's None in visible_gateway/netmask
2017-12-06 00:41:37 +01:00
Marek Marczykowski-Górecki
4d6bfbab4d
tests: improve spoof_ip test
Not only check if full round trip ping (does not) work, but also if just
echo-request get filtered.
2017-12-05 17:40:30 +01:00
Marek Marczykowski-Górecki
379add52ba
tests: skip network tests on whonix-gw and whonix-ws
whonix-ws also have non-standard firewall and require specific tests for
that.
2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
088c255355
tests: add create_qdb_entries() unit test 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
384a792b8a
typo in docstring 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
d54cef5554
app: fix creating dom0 object when not already present in qubes.xml
It's constant properties are now really constant, no need to provide
them explicitly.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
0afee4b05e
Improve checking for netvm loop
There were many cases were the check was missing:
 - changing default_netvm
 - resetting netvm to default value
 - loading already broken qubes.xml

Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...

Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
2d830caab9
app: check DEFAULT_LVM_POOL variable for default pool
Allow to hint default_pool what is default storage pool - especially
useful for tests.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
749e8497e3
api/admin: exclude regex attribute from DeviceInfo structure
DeviceInfo may contain 'regex' attribute - it isn't intended to be
reported through Admin API. Also, mark 'libvirt_regex' attribute as
private.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
da97f4d84c
qubesvm: make initial qmemman request consistent with libvirt config
If HVM have PCI device, it can't use PoD, so need 'maxmem' memory to be
started. Request that much from qmemman.
Note that is is somehow independent of enabling or not dynamic memory
management for the VM (`service.meminfo-writer` feature). Even if VM
initially had assigned maxmem memory, it can be later ballooned down.

QubesOS/qubes-issues#3207
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
19a1579a99
tests: fix deadlock in filecopy test
Error window (where test need to send Enter key) is opened while
qvm-move-to-vm is still running.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
99874a0a25
tests: make waiting for window asyncio aware
For now just replace sleep with asyncio.sleep. Later it may make sense
to change subprocess.call too.
2017-12-05 17:39:32 +01:00
Jean-Philippe Ouellet
b41c118166
Fix typo in volume import end handler 2017-11-21 01:05:06 -05:00
Marek Marczykowski-Górecki
23ee8aeb47
vm/qubesvm: fix handling vm.storage.verify() errors
except section try to access qmemman_client variable - make it defined
at that stage.
2017-11-21 04:16:30 +01:00
Marek Marczykowski-Górecki
a92dd99fbb
Merge branch '20171107-storage'
* 20171107-storage:
  api/admin: add API for changing revisions_to_keep dynamically
  storage/file: move revisions_to_keep restrictions to property setter
  api/admin: hide dd statistics in admin.vm.volume.Import call
  storage/lvm: fix importing different-sized volume from another pool
  storage/file: fix preserving spareness on volume clone
  api/admin: add pool size and usage to admin.pool.Info response
  storage: add size and usage properties to pool object
2017-11-20 22:52:50 +01:00
Marek Marczykowski-Górecki
2c6c766968
Merge branch '20171107-tests-backup-api-misc'
* 20171107-tests-backup-api-misc:
  test: make race condition on xterm close less likely
  tests/backupcompatibility: fix handling 'internal' property
  backup: fix handling target write error (like no disk space)
  tests/backupcompatibility: drop R1 format tests
  backup: use offline_mode for backup collection
  qubespolicy: fix handling '$adminvm' target with ask action
  app: drop reference to libvirt object after undefining it
  vm: always log startup fail
  api: do not log handled errors sent to a client
  tests/backups: convert to new restore handling - using qubesadmin module
  app: clarify error message on failed domain remove (used somewhere)
  Fix qubes-core.service ordering
2017-11-20 22:52:26 +01:00
Marek Marczykowski-Górecki
9519f158ab
vm/qubesvm: vm.storage.stop() is a coroutine 2017-11-20 22:49:52 +01:00
Marek Marczykowski-Górecki
cf92a576ad
Merge remote-tracking branch 'qubesos/pr/159'
* qubesos/pr/159:
  qubes/vm: Improve stopped event handling
2017-11-20 22:49:05 +01:00
Marek Marczykowski-Górecki
3217c3ac4e
Merge remote-tracking branch 'qubesos/pr/156'
* qubesos/pr/156:
  tests: Add unit tests for pvh virt_mode
  Don't allow attached PCI devices and virt_mode = 'pvh'
2017-11-20 22:40:53 +01:00
qubesuser
edae7a16b9 create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00 2017-11-11 02:38:16 +01:00
qubesuser
92c452a655 change default pool code to be fast
currently it takes 100ms+ to determine the default pool every time,
including subprocess spawning (!), which is unacceptable since
finding out the default value of properties should be instantaneous

instead of checking every thin pool to see if the root device is in
it, find and cache the name of the root device thin pool and see if
there is a configured pool with that name
2017-11-11 02:38:16 +01:00
qubesuser
d183ab1c18 cache PropertyHolder.property_list and use O(1) property name lookups 2017-11-11 02:38:16 +01:00
qubesuser
f2b8ad7d38 remove unused netid code
it's unused and has a netid property with name different than key
that would cause issues in the next commit
2017-11-11 02:37:37 +01:00
qubesuser
b297ecbcf1 cache isinstance(default, collections.Callable) 2017-11-10 18:14:40 +01:00
qubesuser
9cc86b3be2 don't access netvm if it's None in visible_gateway/netmask
Causes an unnecessary exception
2017-11-10 18:14:31 +01:00
Marek Marczykowski-Górecki
45af0913f2
test: make race condition on xterm close less likely
xterm is very fast on closing when application inside terminates. It is
so fast with closing on keydown event that xdotool do not manage to send
keyup event, resulting in xdotool crash. Add a little more time for
that.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
07d4f4b340
tests/backupcompatibility: fix handling 'internal' property 2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
5e6ba4ff5c
backup: fix handling target write error (like no disk space)
When writing process returns an error, prefer reporting that one,
instead of other process (which in most cases will be canceled, so
the error will be meaningless CancelledError).
Then, include sanitized stderr from VM process in the error message.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
11fd2cf115
tests/backupcompatibility: drop R1 format tests
It is no longer supported.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
abfed95bf2
backup: use offline_mode for backup collection
This object is used solely for manipulating qubes.xml for the backup. Do
not open any connection to hypervisor/QubesDB/libvirt etc. This will
help avoiding various leaks (memory, FD).
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
227378f2b3
app: drop reference to libvirt object after undefining it
Do not try to access that particular object (wrapper) when it got
undefined. If anyone want to access it, appropriate code should do a new
lookup, and probably re-define the object.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
b5e07238a0
vm: always log startup fail 2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
73c33525a0
api: do not log handled errors sent to a client
Those "errors" are already properly handled, and if necessary logged
independently by appropriate function. In some cases, such logs are
misleading (for example QubesNoSuchPropertyError is a normal thing
happening during qvm-ls).

Fixes QubesOS/qubes-issues#3238
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
31a55dcd18
tests/backups: convert to new restore handling - using qubesadmin module
Besides converting itself, change how the test verify restore
correctness: first collect VM metadata (and hashes of data) into plain
dict, then compare against it. This allow to destroy old VMs objects
before restoring the backup, so avoid having duplicate objects of the
same VM - which results in weird effects like trying to undefine libvirt
object twice.
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
59cebd5439
app: clarify error message on failed domain remove (used somewhere)
Point to system logs for more details. Do not include them directly in
the message for privacy reasons (Admin API client may not be given
permission to it).

QubesOS/qubes-issues#3273
QubesOS/qubes-issues#3193
2017-11-07 03:10:32 +01:00
Marek Marczykowski-Górecki
c3afdde3ef
api/admin: add API for changing revisions_to_keep dynamically
This one pool/volume property makes sense to change dynamically. There
may be more such properties, but lets be on the safe side and take
whitelist approach - allow only selected (just one for now), instead of
blacklisting any harmful ones.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
81f455e15d
storage/file: move revisions_to_keep restrictions to property setter
Do not check for accepted value only in constructor, do that in property
setter. This will allow enforcing the limit regardless of how the value
was set.

This is preparation for dynamic revisions_to_keep change.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
439d9b87ff
storage/lvm: fix importing different-sized volume from another pool
Fixes QubesOS/qubes-issues#3257
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
f3455b5d99
storage/file: fix preserving spareness on volume clone
Force creating sparse file, even if source volume is not such file (for
example block device).

Reported by @na--
QubesOS/qubes-issues#3255
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
f18f4c9bff
api/admin: add pool size and usage to admin.pool.Info response
QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
e76372b934
storage: add size and usage properties to pool object
Add Pool.size and Pool.usage to the API. Implement them for LVM and File
pools. Add appropriate tests.

QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
HW42
c695f3268c linux-stubdom: Set cmdline in HVM config 2017-10-30 16:19:01 +01:00
HW42
b907303a39 qubes/vm: Improve stopped event handling
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.

This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.

Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.

Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
2017-10-21 05:57:57 +02:00
Marek Marczykowski-Górecki
e2eaa57f65
Merge branch 'tests-20171020'
* tests-20171020:
  tests: fix asyncio usage some more places
  tests: fix reporting in network tests
  tests: domain-shutdown event race condition
  tests: improve events tests
  tests: fix logic error in wait_for_window()
  tests: cleanup QubesDB connection on domain remove
2017-10-21 04:33:35 +02:00
Marek Marczykowski-Górecki
42f8619559
tests: fix asyncio usage some more places
gui_memory_pinning tests finally works. Also, fix DispVM and private
volume resize tests.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
511502ae7e
tests: fix reporting in network tests
Do not trash stdout with VM's command output. Log it as part of
(potential) failure message.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
c3430fe5f6
tests: domain-shutdown event race condition
Second tests, for actual events, not storage handling done by its
handler.

cc @HW42

QubesOS/qubes-issues#3164
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
96f2e503db
tests: improve events tests
Check property-related events, including their arguments.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
71fc5217f7
tests: fix logic error in wait_for_window()
Revert change introduced in cbc5dbb9b1
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
43c09467b1
tests: cleanup QubesDB connection on domain remove
If domain got removed during the tests (for example DispVM), vm.close()
wouldn't be called in cleanup and some file descriptors will be
leaked. Add event handler for cleaning this up. Do not use close()
method here, because it is destructive, but the object may still be used
by the test.
2017-10-21 04:06:14 +02:00
Marek Marczykowski-Górecki
0b1a0b028b
app: fix setting default_*netvm
1. Fire both property-pre-del:netvm and property-del:netvm - those
events should be fired in pairs - especially one may assume the other
will be called too. This is the case here - one disconnect old netvm,
the other connect the new one.

2. Remove spurious 'newvalue' argument for property-del:netvm event.

3. Fix logic for default_fw_netvm/default_netvm usage. The former is
used if vm.provides_network=True.
2017-10-21 04:05:17 +02:00
Marek Marczykowski-Górecki
4500c4dcab
vm/adminvm: add is_halted() method
It is used in some places, for example attaching devices.

Fixes QubesOS/qubes-issues#3054
2017-10-21 04:05:17 +02:00
Marek Marczykowski-Górecki
433430884d
devices: clarify error on attaching non-persistent device to stopped VM
Fixes QubesOS/qubes-issues#3195
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
9a9452ae3a
vm/net: fix handling netvm reset to default
If there was some netvm set, unset it first (same as with ordinary set).
Otherwise it will try to attach new netvm without detaching the old one
first.
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
98818b0ad9
base: fix property-(pre-)del events arguments
oldvalue should contain the old value, even if it was default one. Same
as in property-(pre-)set events. If event want to check if that is
default value, it is always possible (in pre- event), but in practice
actual value is most useful.

This bug prevented netvm changing events from working when reseting
netvm.
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
8f1bfa65f8
tests: fix typo 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
4809c16e08
admin-api: add admin.*.property.GetDefault
Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
038ca6d5d7
base: add method for getting default property value
Allow to get default value even it isn't set currently. This will allow
(G)UI to present better view, without duplicating logic for default
value.

Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
85c3368a0d
comments fixes 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
c2c3225622
tests: root fs resize tests - for all templates
Check resize on each template separately, because it involves VM's
scripts (either qubes.ResizeDisk service, or some startup script).

QubesOS/qubes-issues#3173
2017-10-21 04:05:14 +02:00
Marek Marczykowski-Górecki
77ce956c2d
tests: improve root volume resize tests
- clone all features, not just qrexec (especially include 'gui')
- do not leak VM reference on failed test
- add test for online root volume resize

QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
49e35e66aa
storage: improve error reporting
- make it clear that failed qubes.ResizeDisk service call means the need
to resize filesystem manually (but not necessarily the volume itself)
- propagate exceptions raised by async storage pool implementations

Related QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
27b96cce4c
firewall: skip expired rules
Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).

Related QubesOS/qubes-issues#3020
2017-10-21 03:35:00 +02:00
Marek Marczykowski-Górecki
836d9f902a
Merge branch 'bug3164'
* bug3164:
  tests: add regression test for #3164
  storage/lvm: make sure volume cache is refreshed after changes
  storage/lvm: fix Volume.verify()
  storage/lvm: remove old volume only after successfully cloning new one
2017-10-16 04:28:26 +02:00
Marek Marczykowski-Górecki
c8519a700f
tests: add regression test for #3164
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.

QubesOS/qubes-issues#3164
2017-10-16 00:48:25 +02:00
Marek Marczykowski-Górecki
08583af1ff
storage/lvm: make sure volume cache is refreshed after changes
Even if start/stop fails, still refresh the cache. The failed operation
might still modified something.
2017-10-16 00:44:52 +02:00
Marek Marczykowski-Górecki
145ccfb34e
storage/lvm: fix Volume.verify()
This function is supposed to raise an exception if something is wrong,
not just return False. Document this.
2017-10-16 00:43:10 +02:00
Marek Marczykowski-Górecki
021047f950
storage/lvm: remove old volume only after successfully cloning new one
In some cases, it may happen that new volume (`self._vid_snap`) does not
exists. This is normally an error, but even in such a case, do not
remove the only remaining instance of volume (`self.vid`). Instead,
rename it temporarily and remove only after new volume is successfully
cloned.

Fixes QubesOS/qubes-issues#3164
2017-10-13 01:11:55 +02:00
Marek Marczykowski-Górecki
2164a8d7b8
Change license to LGPL v2.1+
See this thread for reasoning and acceptance from contributors:
https://groups.google.com/d/topic/qubes-devel/G7KzrfU0lWY/discussion
"Changing qubes-core-admin license to LGPL v2.1+"
2017-10-12 00:11:50 +02:00
HW42
4a84d5b222 tests: Add unit tests for pvh virt_mode 2017-10-11 21:11:36 +02:00
HW42
67c06eb428 Don't allow attached PCI devices and virt_mode = 'pvh' 2017-10-11 21:10:43 +02:00
Marek Marczykowski-Górecki
08640f4779
Merge remote-tracking branch 'qubesos/pr/154'
* qubesos/pr/154:
  Add basic PVHv2 support
2017-10-09 00:08:35 +02:00
Marek Marczykowski-Górecki
b8d45c214d
storage/lvm: don't crash when resizing to the same size
This applies to qvm-template-postprocess, which at the beginning try to
resize root volume to appropriate size. It makes more sense to silently
succeed here, instead of forcing every client-side utility to check if
the volume have already desired size.
2017-10-08 18:37:25 +02:00
Marek Marczykowski-Górecki
bce7b64d4a
Fix calling qubes.SuspendPre/qubes.SuspendPost services
Those should be called as root, not default user.

Thanks @jpouellet for debugging this.
Fixes QubesOS/qubes-issues#3151
Fixes QubesOS/qubes-issues#3142
2017-10-07 02:36:02 +02:00
Marek Marczykowski-Górecki
451cc339c7
ext/block: accept any boolean value for 'read-only' option
Be consistent with other parts of the Admin API. Especially ...Attach
accept "standard" boolean values for persistent= option.
2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
af60fbb25d
tests: dispvm tests fixes 2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
df32acfad9
Allow setting global default_dispvm to None
There is no technical need for having this set to anything. If set to
None, DispVM startup requests will be refused.
2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
6990550884
tests: more tests fixes, drop expectedFailure where not needed anymore 2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
e0be7861f9
tests: extend qrexec_policy context manager - custom action
Allow to use custom action - for example with 'target=' argument.
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
cbc5dbb9b1
tests: various fixes
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
  cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
64c5b2c00f
tests: cleanup DispVMs created during tests
Since those do not have specific prefix (test-) compare list of them
with original qubes.xml
2017-10-04 15:24:51 +02:00
Marek Marczykowski-Górecki
6091bbbe88
storage/lvm: fix size units on resize
lvextend uses MiB, not MB.
2017-10-03 18:46:39 +02:00
Marek Marczykowski-Górecki
3a5e8482c0
Merge branch 'fixes-20170929'
* fixes-20170929:
  vm: do not start QubesDB watch instance multiple times
  vm: report storage.stop() errors to log
  vm: move comment
  storage: fix method name in LinuxModules volume
  Prevent removing domain that is referenced from anywhere
  vm: add vm.klass property
  Move QubesVM.{name,qid,uuid,label} to BaseVM
  vm: do not allow deleting template property from AppVM and DispVM
  vm/qubesvm: emit event on failed startup
  vm/qubesvm: remove duplicated qmemman_client.close()
  vm/dispvm: cleanup DispVM also on failed startup
  vm/dispvm: fix error message
  ext/block: properly list devtype=cdrom option
  block: fix handling non-existing devices
  block: improve handling device name and description
2017-10-03 12:04:00 +02:00
HW42
c583cf5190 Add basic PVHv2 support 2017-10-02 22:23:27 +02:00
Marek Marczykowski-Górecki
12b7e22d27
vm: do not start QubesDB watch instance multiple times
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
2017-09-29 12:05:23 +02:00
Marek Marczykowski-Górecki
9172a7708d
tests: fix various object leaks cont
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
f9da1f2dd7
tests: various fixes for networking tests
- Update feature name for enabling network-manager.
- Fix error reporting
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
fdf780aad0
tests/network: match vm names with variable names
Having testvm1 variable for "vm2" is really confusing.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
8b34b9a5e4
tests: hide unneeded messages 2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
3c54244fdc
tests: fix various object and FD leaks
- Prefer instance attributes over local variables - the former ones do
not leak into traceback object and are cleaned up by tests framework.
- Use 'with' syntax for handling files.
- Use subprocess.DEVNULL instead of open('/dev/null') where applicable
- Delete local variables when not needed anymore.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
d96d4a94e1
tests: various fixes
- Fix str/bytes
- Call skipTest as early as possible - before doing any setup
- Fix networking tests - configuration commands needs to be called as
root (missing user= argument).
- Fix setting firewall - policy is no longer changeable
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
ed3001da99
tests: fix asyncio usage cont...
- Add missing loop.run_until_complete() calls.
- Convert subprocess.Popen to asyncio.create_subprocess_exec where
needed (when called process needs to communicate with qubesd).
- Cleanup processes (call .wait()).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
f0da5b21df
tests: improve "revert template changes" test
There is no more significant difference between PV and HVM. VMs are HVM
by default anyway. More important for this test is difference between
Linux (with Qubes packages installed) and other OS-es. Rename tests
accordingly. The later one is still incomplete.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
4b51c9da07
tests/storage: don't fail the test when directory already exists
/var/tmp/test-pool should normally be removed by previous test cleanup,
but do not fail next test if it wasn't.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
344bb16ae2
tests: fix various object leaks at tests framework level
The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.

Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
a929916ce8
tests/storage: do not depend on system default storage pool
Use just created storage pool for tests, instead of system default
(which depends on system configuration).
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
d8ff85eee6
tests: fix cleanup function
Don't crash when directory to cleanup do not exists.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
3075767bb8
tests: add more complex test for netvm cyclic connection
Something more complex than VM connecting to itself.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
b12fa13f06
vm: report storage.stop() errors to log
Catch exception there and log it. Otherwise asyncio complains about not
retrieved exception. There is no one else to handle this exception,
because shutdown event is triggered from libvirt, not any Admin API.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
97e3dced92
vm: move comment
Place comment describing self.app near self.app definition.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
34125d915b
storage: fix method name in LinuxModules volume
It's `import_volume`, not `clone`.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
9e8c40867b
Prevent removing domain that is referenced from anywhere
Check VM properties and global properties (all of them).

Fixes QubesOS/qubes-issues#3128
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
a90dea34de
vm: add vm.klass property
Allow to get domain class as a property, not using admin.vm.List call.
This makes it unnecessary to call admin.vm.List on the client side to
construct wrapper object.
2017-09-29 11:53:01 +02:00