Commit Graph

2191 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
962742880f
tests: basic salt integration tests
Test base functions of dom0 module (creating VM, setting property) and
configuring system inside of VM (through DispVM). The later is done for
each available template (the process use salt installed in that
template, not copied from dom0).

QubesOS/qubes-issues#3316
2017-12-23 02:26:15 +01:00
Marek Marczykowski-Górecki
32c6083e1c
Make pylint happy
Fix thing detected by updated pylint in Travis-CI
2017-12-21 18:19:10 +01:00
Marek Marczykowski-Górecki
f5fe10e2ad
vm/adminvm: fix type of AdminVM.qid
It is defined as int in QubesVM.qid, make it consistent.
2017-12-21 18:18:12 +01:00
Marek Marczykowski-Górecki
9a4d2abf1f
tests: booting VM from ISO image
Check two cases: if ISO image is in dom0 and if its in a VM.

QubesOS/qubes-issues#3339
2017-12-14 23:26:53 +01:00
Marek Marczykowski-Górecki
96bd734852
storage/lvm: clean -snap LV on volume removal 2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
466bf89aae
Fix starting VM with kernel=None
When dom0 do not provide the kernel, it should also not set kernel
command line in libvirt config. Otherwise qemu in stubdom fails to start
because it get -append option without -kernel, which is illegal
configuration.

Fixes QubesOS/qubes-issues#3339
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
fd45378041
api/admin: make libvirt start error more informative
Point where to look for details.
2017-12-14 23:26:52 +01:00
Marek Marczykowski-Górecki
297cb4d012
vm/mix/net: fill QubesDB for already connected VMs
There may be cases when VM providing the network to other VMs is started
later - for example VM restart. While this is rare case (and currently
broken because of QubesOS/qubes-issues#1426), do not assume it will
always be the case.
2017-12-14 23:26:51 +01:00
Marek Marczykowski-Górecki
1187e43697
tests: more vm.create_qdb_entries() tests 2017-12-14 02:09:31 +01:00
Marek Marczykowski-Górecki
97564f014c
vm/mix/net: fix setting QubesDB after converting to ipaddress module 2017-12-14 02:07:57 +01:00
Marek Marczykowski-Górecki
faef890c9a
vm/qubesvm: write QubesDB /qubes-netvm-gateway6 entry when set
This is needed for network-providing VM to actually provide IPv6
connection too.

QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
e12a66f103
vm/mix/net: use ipaddress module for ip and ip6 properties
It has built-in validation, which is much more elegant than custom regex
or socket call.

Suggested by @woju
QubesOS/qubes-issues#718
2017-12-07 01:40:31 +01:00
Marek Marczykowski-Górecki
f3cf58e6f2
tests: add integration tests for IPv6
Run also all IPv4 tests with IPv6 enabled to check for regressions
(broken IPv4 because of enabled IPv6).

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
0786edf8a3
tests: add IPv6-related unit tests
Check produced libvirt XML, and QubesDB entries

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
18f159f8ec
Add IPv6 related VM properties
Add property for IPv6 address ('ip6'). Build default value similarly to
IPv4 - common prefix + QID or Disp ID (for DispVMs).
This all is disabled unless 'ipv6' feature is enabled. It is inherited
from netvm (not template).
Even when enabled, VM may decide to not use it - or simply not support
it.

QubesOS/qubes-issues#718
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
bf59b00f1d
features: add check_with_netvm, similar to check_with_template
Allow using default feature value from netvm, not template. This makes
sense for network-related features like using tor, supporting ipv6 etc.

Similarly to check_with_template, expose it also on Admin API.
2017-12-07 01:40:30 +01:00
Marek Marczykowski-Górecki
f223594f92
app: kill default_fw_netvm property
Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)

As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.

Fixes QubesOS/qubes-issues#3247
2017-12-07 01:40:29 +01:00
Marek Marczykowski-Górecki
f2cd7fb226
Merge branch 'tests-and-fixes-20171205' 2017-12-07 01:39:34 +01:00
Marek Marczykowski-Górecki
7b81d7affa
Merge remote-tracking branch 'qubesos/pr/166'
* qubesos/pr/166:
  create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
  change default pool code to be fast
  cache PropertyHolder.property_list and use O(1) property name lookups
  remove unused netid code
  cache isinstance(default, collections.Callable)
  don't access netvm if it's None in visible_gateway/netmask
2017-12-06 00:41:37 +01:00
Marek Marczykowski-Górecki
4d6bfbab4d
tests: improve spoof_ip test
Not only check if full round trip ping (does not) work, but also if just
echo-request get filtered.
2017-12-05 17:40:30 +01:00
Marek Marczykowski-Górecki
379add52ba
tests: skip network tests on whonix-gw and whonix-ws
whonix-ws also have non-standard firewall and require specific tests for
that.
2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
088c255355
tests: add create_qdb_entries() unit test 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
384a792b8a
typo in docstring 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
d54cef5554
app: fix creating dom0 object when not already present in qubes.xml
It's constant properties are now really constant, no need to provide
them explicitly.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
0afee4b05e
Improve checking for netvm loop
There were many cases were the check was missing:
 - changing default_netvm
 - resetting netvm to default value
 - loading already broken qubes.xml

Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...

Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
2d830caab9
app: check DEFAULT_LVM_POOL variable for default pool
Allow to hint default_pool what is default storage pool - especially
useful for tests.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
749e8497e3
api/admin: exclude regex attribute from DeviceInfo structure
DeviceInfo may contain 'regex' attribute - it isn't intended to be
reported through Admin API. Also, mark 'libvirt_regex' attribute as
private.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
da97f4d84c
qubesvm: make initial qmemman request consistent with libvirt config
If HVM have PCI device, it can't use PoD, so need 'maxmem' memory to be
started. Request that much from qmemman.
Note that is is somehow independent of enabling or not dynamic memory
management for the VM (`service.meminfo-writer` feature). Even if VM
initially had assigned maxmem memory, it can be later ballooned down.

QubesOS/qubes-issues#3207
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
19a1579a99
tests: fix deadlock in filecopy test
Error window (where test need to send Enter key) is opened while
qvm-move-to-vm is still running.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
99874a0a25
tests: make waiting for window asyncio aware
For now just replace sleep with asyncio.sleep. Later it may make sense
to change subprocess.call too.
2017-12-05 17:39:32 +01:00
Jean-Philippe Ouellet
b41c118166
Fix typo in volume import end handler 2017-11-21 01:05:06 -05:00
Marek Marczykowski-Górecki
23ee8aeb47
vm/qubesvm: fix handling vm.storage.verify() errors
except section try to access qmemman_client variable - make it defined
at that stage.
2017-11-21 04:16:30 +01:00
Marek Marczykowski-Górecki
a92dd99fbb
Merge branch '20171107-storage'
* 20171107-storage:
  api/admin: add API for changing revisions_to_keep dynamically
  storage/file: move revisions_to_keep restrictions to property setter
  api/admin: hide dd statistics in admin.vm.volume.Import call
  storage/lvm: fix importing different-sized volume from another pool
  storage/file: fix preserving spareness on volume clone
  api/admin: add pool size and usage to admin.pool.Info response
  storage: add size and usage properties to pool object
2017-11-20 22:52:50 +01:00
Marek Marczykowski-Górecki
2c6c766968
Merge branch '20171107-tests-backup-api-misc'
* 20171107-tests-backup-api-misc:
  test: make race condition on xterm close less likely
  tests/backupcompatibility: fix handling 'internal' property
  backup: fix handling target write error (like no disk space)
  tests/backupcompatibility: drop R1 format tests
  backup: use offline_mode for backup collection
  qubespolicy: fix handling '$adminvm' target with ask action
  app: drop reference to libvirt object after undefining it
  vm: always log startup fail
  api: do not log handled errors sent to a client
  tests/backups: convert to new restore handling - using qubesadmin module
  app: clarify error message on failed domain remove (used somewhere)
  Fix qubes-core.service ordering
2017-11-20 22:52:26 +01:00
Marek Marczykowski-Górecki
9519f158ab
vm/qubesvm: vm.storage.stop() is a coroutine 2017-11-20 22:49:52 +01:00
Marek Marczykowski-Górecki
cf92a576ad
Merge remote-tracking branch 'qubesos/pr/159'
* qubesos/pr/159:
  qubes/vm: Improve stopped event handling
2017-11-20 22:49:05 +01:00
Marek Marczykowski-Górecki
3217c3ac4e
Merge remote-tracking branch 'qubesos/pr/156'
* qubesos/pr/156:
  tests: Add unit tests for pvh virt_mode
  Don't allow attached PCI devices and virt_mode = 'pvh'
2017-11-20 22:40:53 +01:00
qubesuser
edae7a16b9 create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00 2017-11-11 02:38:16 +01:00
qubesuser
92c452a655 change default pool code to be fast
currently it takes 100ms+ to determine the default pool every time,
including subprocess spawning (!), which is unacceptable since
finding out the default value of properties should be instantaneous

instead of checking every thin pool to see if the root device is in
it, find and cache the name of the root device thin pool and see if
there is a configured pool with that name
2017-11-11 02:38:16 +01:00
qubesuser
d183ab1c18 cache PropertyHolder.property_list and use O(1) property name lookups 2017-11-11 02:38:16 +01:00
qubesuser
f2b8ad7d38 remove unused netid code
it's unused and has a netid property with name different than key
that would cause issues in the next commit
2017-11-11 02:37:37 +01:00
qubesuser
b297ecbcf1 cache isinstance(default, collections.Callable) 2017-11-10 18:14:40 +01:00
qubesuser
9cc86b3be2 don't access netvm if it's None in visible_gateway/netmask
Causes an unnecessary exception
2017-11-10 18:14:31 +01:00
Marek Marczykowski-Górecki
45af0913f2
test: make race condition on xterm close less likely
xterm is very fast on closing when application inside terminates. It is
so fast with closing on keydown event that xdotool do not manage to send
keyup event, resulting in xdotool crash. Add a little more time for
that.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
07d4f4b340
tests/backupcompatibility: fix handling 'internal' property 2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
5e6ba4ff5c
backup: fix handling target write error (like no disk space)
When writing process returns an error, prefer reporting that one,
instead of other process (which in most cases will be canceled, so
the error will be meaningless CancelledError).
Then, include sanitized stderr from VM process in the error message.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
11fd2cf115
tests/backupcompatibility: drop R1 format tests
It is no longer supported.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
abfed95bf2
backup: use offline_mode for backup collection
This object is used solely for manipulating qubes.xml for the backup. Do
not open any connection to hypervisor/QubesDB/libvirt etc. This will
help avoiding various leaks (memory, FD).
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
227378f2b3
app: drop reference to libvirt object after undefining it
Do not try to access that particular object (wrapper) when it got
undefined. If anyone want to access it, appropriate code should do a new
lookup, and probably re-define the object.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
b5e07238a0
vm: always log startup fail 2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
73c33525a0
api: do not log handled errors sent to a client
Those "errors" are already properly handled, and if necessary logged
independently by appropriate function. In some cases, such logs are
misleading (for example QubesNoSuchPropertyError is a normal thing
happening during qvm-ls).

Fixes QubesOS/qubes-issues#3238
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
31a55dcd18
tests/backups: convert to new restore handling - using qubesadmin module
Besides converting itself, change how the test verify restore
correctness: first collect VM metadata (and hashes of data) into plain
dict, then compare against it. This allow to destroy old VMs objects
before restoring the backup, so avoid having duplicate objects of the
same VM - which results in weird effects like trying to undefine libvirt
object twice.
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
59cebd5439
app: clarify error message on failed domain remove (used somewhere)
Point to system logs for more details. Do not include them directly in
the message for privacy reasons (Admin API client may not be given
permission to it).

QubesOS/qubes-issues#3273
QubesOS/qubes-issues#3193
2017-11-07 03:10:32 +01:00
Marek Marczykowski-Górecki
c3afdde3ef
api/admin: add API for changing revisions_to_keep dynamically
This one pool/volume property makes sense to change dynamically. There
may be more such properties, but lets be on the safe side and take
whitelist approach - allow only selected (just one for now), instead of
blacklisting any harmful ones.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
81f455e15d
storage/file: move revisions_to_keep restrictions to property setter
Do not check for accepted value only in constructor, do that in property
setter. This will allow enforcing the limit regardless of how the value
was set.

This is preparation for dynamic revisions_to_keep change.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
439d9b87ff
storage/lvm: fix importing different-sized volume from another pool
Fixes QubesOS/qubes-issues#3257
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
f3455b5d99
storage/file: fix preserving spareness on volume clone
Force creating sparse file, even if source volume is not such file (for
example block device).

Reported by @na--
QubesOS/qubes-issues#3255
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
f18f4c9bff
api/admin: add pool size and usage to admin.pool.Info response
QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
e76372b934
storage: add size and usage properties to pool object
Add Pool.size and Pool.usage to the API. Implement them for LVM and File
pools. Add appropriate tests.

QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
HW42
c695f3268c linux-stubdom: Set cmdline in HVM config 2017-10-30 16:19:01 +01:00
HW42
b907303a39 qubes/vm: Improve stopped event handling
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.

This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.

Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.

Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
2017-10-21 05:57:57 +02:00
Marek Marczykowski-Górecki
e2eaa57f65
Merge branch 'tests-20171020'
* tests-20171020:
  tests: fix asyncio usage some more places
  tests: fix reporting in network tests
  tests: domain-shutdown event race condition
  tests: improve events tests
  tests: fix logic error in wait_for_window()
  tests: cleanup QubesDB connection on domain remove
2017-10-21 04:33:35 +02:00
Marek Marczykowski-Górecki
42f8619559
tests: fix asyncio usage some more places
gui_memory_pinning tests finally works. Also, fix DispVM and private
volume resize tests.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
511502ae7e
tests: fix reporting in network tests
Do not trash stdout with VM's command output. Log it as part of
(potential) failure message.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
c3430fe5f6
tests: domain-shutdown event race condition
Second tests, for actual events, not storage handling done by its
handler.

cc @HW42

QubesOS/qubes-issues#3164
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
96f2e503db
tests: improve events tests
Check property-related events, including their arguments.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
71fc5217f7
tests: fix logic error in wait_for_window()
Revert change introduced in cbc5dbb9b1
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
43c09467b1
tests: cleanup QubesDB connection on domain remove
If domain got removed during the tests (for example DispVM), vm.close()
wouldn't be called in cleanup and some file descriptors will be
leaked. Add event handler for cleaning this up. Do not use close()
method here, because it is destructive, but the object may still be used
by the test.
2017-10-21 04:06:14 +02:00
Marek Marczykowski-Górecki
0b1a0b028b
app: fix setting default_*netvm
1. Fire both property-pre-del:netvm and property-del:netvm - those
events should be fired in pairs - especially one may assume the other
will be called too. This is the case here - one disconnect old netvm,
the other connect the new one.

2. Remove spurious 'newvalue' argument for property-del:netvm event.

3. Fix logic for default_fw_netvm/default_netvm usage. The former is
used if vm.provides_network=True.
2017-10-21 04:05:17 +02:00
Marek Marczykowski-Górecki
4500c4dcab
vm/adminvm: add is_halted() method
It is used in some places, for example attaching devices.

Fixes QubesOS/qubes-issues#3054
2017-10-21 04:05:17 +02:00
Marek Marczykowski-Górecki
433430884d
devices: clarify error on attaching non-persistent device to stopped VM
Fixes QubesOS/qubes-issues#3195
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
9a9452ae3a
vm/net: fix handling netvm reset to default
If there was some netvm set, unset it first (same as with ordinary set).
Otherwise it will try to attach new netvm without detaching the old one
first.
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
98818b0ad9
base: fix property-(pre-)del events arguments
oldvalue should contain the old value, even if it was default one. Same
as in property-(pre-)set events. If event want to check if that is
default value, it is always possible (in pre- event), but in practice
actual value is most useful.

This bug prevented netvm changing events from working when reseting
netvm.
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
8f1bfa65f8
tests: fix typo 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
4809c16e08
admin-api: add admin.*.property.GetDefault
Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
038ca6d5d7
base: add method for getting default property value
Allow to get default value even it isn't set currently. This will allow
(G)UI to present better view, without duplicating logic for default
value.

Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
85c3368a0d
comments fixes 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
c2c3225622
tests: root fs resize tests - for all templates
Check resize on each template separately, because it involves VM's
scripts (either qubes.ResizeDisk service, or some startup script).

QubesOS/qubes-issues#3173
2017-10-21 04:05:14 +02:00
Marek Marczykowski-Górecki
77ce956c2d
tests: improve root volume resize tests
- clone all features, not just qrexec (especially include 'gui')
- do not leak VM reference on failed test
- add test for online root volume resize

QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
49e35e66aa
storage: improve error reporting
- make it clear that failed qubes.ResizeDisk service call means the need
to resize filesystem manually (but not necessarily the volume itself)
- propagate exceptions raised by async storage pool implementations

Related QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
27b96cce4c
firewall: skip expired rules
Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).

Related QubesOS/qubes-issues#3020
2017-10-21 03:35:00 +02:00
Marek Marczykowski-Górecki
836d9f902a
Merge branch 'bug3164'
* bug3164:
  tests: add regression test for #3164
  storage/lvm: make sure volume cache is refreshed after changes
  storage/lvm: fix Volume.verify()
  storage/lvm: remove old volume only after successfully cloning new one
2017-10-16 04:28:26 +02:00
Marek Marczykowski-Górecki
c8519a700f
tests: add regression test for #3164
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.

QubesOS/qubes-issues#3164
2017-10-16 00:48:25 +02:00
Marek Marczykowski-Górecki
08583af1ff
storage/lvm: make sure volume cache is refreshed after changes
Even if start/stop fails, still refresh the cache. The failed operation
might still modified something.
2017-10-16 00:44:52 +02:00
Marek Marczykowski-Górecki
145ccfb34e
storage/lvm: fix Volume.verify()
This function is supposed to raise an exception if something is wrong,
not just return False. Document this.
2017-10-16 00:43:10 +02:00
Marek Marczykowski-Górecki
021047f950
storage/lvm: remove old volume only after successfully cloning new one
In some cases, it may happen that new volume (`self._vid_snap`) does not
exists. This is normally an error, but even in such a case, do not
remove the only remaining instance of volume (`self.vid`). Instead,
rename it temporarily and remove only after new volume is successfully
cloned.

Fixes QubesOS/qubes-issues#3164
2017-10-13 01:11:55 +02:00
Marek Marczykowski-Górecki
2164a8d7b8
Change license to LGPL v2.1+
See this thread for reasoning and acceptance from contributors:
https://groups.google.com/d/topic/qubes-devel/G7KzrfU0lWY/discussion
"Changing qubes-core-admin license to LGPL v2.1+"
2017-10-12 00:11:50 +02:00
HW42
4a84d5b222 tests: Add unit tests for pvh virt_mode 2017-10-11 21:11:36 +02:00
HW42
67c06eb428 Don't allow attached PCI devices and virt_mode = 'pvh' 2017-10-11 21:10:43 +02:00
Marek Marczykowski-Górecki
08640f4779
Merge remote-tracking branch 'qubesos/pr/154'
* qubesos/pr/154:
  Add basic PVHv2 support
2017-10-09 00:08:35 +02:00
Marek Marczykowski-Górecki
b8d45c214d
storage/lvm: don't crash when resizing to the same size
This applies to qvm-template-postprocess, which at the beginning try to
resize root volume to appropriate size. It makes more sense to silently
succeed here, instead of forcing every client-side utility to check if
the volume have already desired size.
2017-10-08 18:37:25 +02:00
Marek Marczykowski-Górecki
bce7b64d4a
Fix calling qubes.SuspendPre/qubes.SuspendPost services
Those should be called as root, not default user.

Thanks @jpouellet for debugging this.
Fixes QubesOS/qubes-issues#3151
Fixes QubesOS/qubes-issues#3142
2017-10-07 02:36:02 +02:00
Marek Marczykowski-Górecki
451cc339c7
ext/block: accept any boolean value for 'read-only' option
Be consistent with other parts of the Admin API. Especially ...Attach
accept "standard" boolean values for persistent= option.
2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
af60fbb25d
tests: dispvm tests fixes 2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
df32acfad9
Allow setting global default_dispvm to None
There is no technical need for having this set to anything. If set to
None, DispVM startup requests will be refused.
2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
6990550884
tests: more tests fixes, drop expectedFailure where not needed anymore 2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
e0be7861f9
tests: extend qrexec_policy context manager - custom action
Allow to use custom action - for example with 'target=' argument.
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
cbc5dbb9b1
tests: various fixes
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
  cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
64c5b2c00f
tests: cleanup DispVMs created during tests
Since those do not have specific prefix (test-) compare list of them
with original qubes.xml
2017-10-04 15:24:51 +02:00
Marek Marczykowski-Górecki
6091bbbe88
storage/lvm: fix size units on resize
lvextend uses MiB, not MB.
2017-10-03 18:46:39 +02:00
Marek Marczykowski-Górecki
3a5e8482c0
Merge branch 'fixes-20170929'
* fixes-20170929:
  vm: do not start QubesDB watch instance multiple times
  vm: report storage.stop() errors to log
  vm: move comment
  storage: fix method name in LinuxModules volume
  Prevent removing domain that is referenced from anywhere
  vm: add vm.klass property
  Move QubesVM.{name,qid,uuid,label} to BaseVM
  vm: do not allow deleting template property from AppVM and DispVM
  vm/qubesvm: emit event on failed startup
  vm/qubesvm: remove duplicated qmemman_client.close()
  vm/dispvm: cleanup DispVM also on failed startup
  vm/dispvm: fix error message
  ext/block: properly list devtype=cdrom option
  block: fix handling non-existing devices
  block: improve handling device name and description
2017-10-03 12:04:00 +02:00
HW42
c583cf5190 Add basic PVHv2 support 2017-10-02 22:23:27 +02:00
Marek Marczykowski-Górecki
12b7e22d27
vm: do not start QubesDB watch instance multiple times
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
2017-09-29 12:05:23 +02:00
Marek Marczykowski-Górecki
9172a7708d
tests: fix various object leaks cont
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
f9da1f2dd7
tests: various fixes for networking tests
- Update feature name for enabling network-manager.
- Fix error reporting
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
fdf780aad0
tests/network: match vm names with variable names
Having testvm1 variable for "vm2" is really confusing.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
8b34b9a5e4
tests: hide unneeded messages 2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
3c54244fdc
tests: fix various object and FD leaks
- Prefer instance attributes over local variables - the former ones do
not leak into traceback object and are cleaned up by tests framework.
- Use 'with' syntax for handling files.
- Use subprocess.DEVNULL instead of open('/dev/null') where applicable
- Delete local variables when not needed anymore.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
d96d4a94e1
tests: various fixes
- Fix str/bytes
- Call skipTest as early as possible - before doing any setup
- Fix networking tests - configuration commands needs to be called as
root (missing user= argument).
- Fix setting firewall - policy is no longer changeable
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
ed3001da99
tests: fix asyncio usage cont...
- Add missing loop.run_until_complete() calls.
- Convert subprocess.Popen to asyncio.create_subprocess_exec where
needed (when called process needs to communicate with qubesd).
- Cleanup processes (call .wait()).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
f0da5b21df
tests: improve "revert template changes" test
There is no more significant difference between PV and HVM. VMs are HVM
by default anyway. More important for this test is difference between
Linux (with Qubes packages installed) and other OS-es. Rename tests
accordingly. The later one is still incomplete.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
4b51c9da07
tests/storage: don't fail the test when directory already exists
/var/tmp/test-pool should normally be removed by previous test cleanup,
but do not fail next test if it wasn't.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
344bb16ae2
tests: fix various object leaks at tests framework level
The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.

Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
a929916ce8
tests/storage: do not depend on system default storage pool
Use just created storage pool for tests, instead of system default
(which depends on system configuration).
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
d8ff85eee6
tests: fix cleanup function
Don't crash when directory to cleanup do not exists.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
3075767bb8
tests: add more complex test for netvm cyclic connection
Something more complex than VM connecting to itself.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
b12fa13f06
vm: report storage.stop() errors to log
Catch exception there and log it. Otherwise asyncio complains about not
retrieved exception. There is no one else to handle this exception,
because shutdown event is triggered from libvirt, not any Admin API.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
97e3dced92
vm: move comment
Place comment describing self.app near self.app definition.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
34125d915b
storage: fix method name in LinuxModules volume
It's `import_volume`, not `clone`.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
9e8c40867b
Prevent removing domain that is referenced from anywhere
Check VM properties and global properties (all of them).

Fixes QubesOS/qubes-issues#3128
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
a90dea34de
vm: add vm.klass property
Allow to get domain class as a property, not using admin.vm.List call.
This makes it unnecessary to call admin.vm.List on the client side to
construct wrapper object.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
9f88fa7f0c
Move QubesVM.{name,qid,uuid,label} to BaseVM
Reduce strange code in BaseVM (accessing non-existing self.name) and
code duplication.
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
dc0e1a5481
vm: do not allow deleting template property from AppVM and DispVM
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.

Fixes QubesOS/qubes-issues#3115
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
efe600537e
vm/qubesvm: emit event on failed startup
If VM startup failed before starting anything (even in paused state),
there will be no further event, not even domain-shutdown. This makes it
hard for event-listening applications (like domains tray) to account
domain state. Fix this by emiting domain-start-failed event in every
case of failed startup after emiting domain-pre-start.

Related QubesOS/qubes-issues#3100
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
8c847faacc
vm/qubesvm: remove duplicated qmemman_client.close() 2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
e38e227503
vm/dispvm: cleanup DispVM also on failed startup
If dispvm.auto_cleanup is set, cleanup it also after failed startup
(like not enough memory).

Fixes QubesOS/qubes-issues#3045
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
7c6b04677f
vm/dispvm: fix error message
Fixes QubesOS/qubes-issues#3114
2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
3548ee1163
ext/block: properly list devtype=cdrom option 2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
fd5aaa8866
block: fix handling non-existing devices
Don't yield None as DeviceInfo object. The device-get: event handlers
are expecte to yield anything only when there is a device.
2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
f6d10ec243
block: improve handling device name and description
Don't fail when device have no description. Also, handle device name
consistently - there is already name_re defined.
2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
e0cadfdd67
Merge remote-tracking branch 'qubesos/pr/150'
* qubesos/pr/150:
  qubes/tests: moar fixes
  test-packages: add missing libvirt classes
  qubes/tests: do not deadlock on .drain()
  qubes/vm: put name= first in __repr__
  tests: fix some memory leaks
  tests: complain about memory leaks
  tests: use one event loop and one libvirtaio impl
2017-09-22 20:54:11 +02:00
Wojtek Porczyk
589d567ebc qubes/tests: moar fixes 2017-09-21 14:18:35 +02:00
Wojtek Porczyk
b78ad1993c qubes/tests: do not deadlock on .drain() 2017-09-21 14:17:36 +02:00
Wojtek Porczyk
71eefbea14 qubes/vm: put name= first in __repr__
This is to have it in case the repr is truncated.
2017-09-19 17:02:19 +02:00
Wojtek Porczyk
bd123a3812 tests: fix some memory leaks
This is work in progress. There are still more leaks.
2017-09-19 17:01:29 +02:00
Wojtek Porczyk
5aa9fa2db4 tests: complain about memory leaks
Currently this detects leaking VM (and all subclasses), Qubes and
libvirt-related objects. Usable only with --failfast.
2017-09-19 16:59:09 +02:00
Wojtek Porczyk
7df8f51011 tests: use one event loop and one libvirtaio impl
Recently libvirt removed support for changing event implementation.
Therefore we have to use a single, global one and we check if it is
empty between tests.
2017-09-19 16:47:14 +02:00
HW42
c6c6f681ae tests: Fix expected HVM xml
The last two commits (intentionally) changed the libxl xml. So adapt the
expected value in the test.
2017-09-15 16:19:10 +02:00
Marek Marczykowski-Górecki
86681ba996
Merge remote-tracking branch 'qubesos/pr/149'
* qubesos/pr/149:
  lvm->init_cache: fix incorrect max splits
  Fix service qubesd not working with some locales
2017-09-14 00:15:33 +02:00
Marek Marczykowski-Górecki
55208a0c7c
Merge branch 'dispvm-policy' 2017-09-14 00:15:29 +02:00
Nicolai Søborg
9d797cc864
lvm->init_cache: fix incorrect max splits 2017-09-13 18:50:02 +02:00
Nicolai Søborg
c30131633b Fix service qubesd not working with some locales
On certain locales (e.g. danish) `usage_percent` will output a comma-separated number, which will make `attr` point the last two decimal points, s.t. `return vol_info['attr'][4] == 'a'` (in the `verify` func) will fail and `qubesd` wont run.
2017-09-12 01:57:49 +02:00
Marek Marczykowski-Górecki
0d59965a7b
Merge branch 'cdrom-boot'
* cdrom-boot:
  devices: fix error reporting
  api/admin: implement admin.vm.device....Set.persistent
  devices: implement DeviceCollection.update_persistent()
  devices: move DeviceInfo definition earlier
  api: do not fail events when listener is cancelled in the meantime
2017-09-07 13:02:04 +02:00
Marek Marczykowski-Górecki
99bd193688
Rename 'dispvm_allowed' to 'template_for_dispvms'
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.

Fixes QubesOS/qubes-issues#3047
2017-09-05 02:07:26 +02:00
Marek Marczykowski-Górecki
c247ddff72
dispvm: fix setting up new DispVM
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).

Fixes QubesOS/qubes-issues#3057
2017-09-05 02:07:26 +02:00
Marek Marczykowski-Górecki
687ca4092c
tests: various fixes to integration tests
Mostly add missing self.loop.run_until_complete().
2017-09-05 02:01:58 +02:00
Marek Marczykowski-Górecki
d676576600
tests: fix qvm-sync-clock test for asyncio and new time sync API
qvm-sync-clock in dom0 now synchronize only dom0 time. For VM time,
qvm-sync-clock needs to be called in VM. Also, both will communicate
with qubesd, so must be called asynchronously from tests.
2017-09-05 02:01:58 +02:00
Marek Marczykowski-Górecki
32b57deb3a
devices: fix error reporting 2017-09-05 01:33:47 +02:00
Marek Marczykowski-Górecki
cea70748a6
api/admin: implement admin.vm.device....Set.persistent
This will allow converting persistent device (used to boot VM for
example) to non-persistent.

QubesOS/qubes-issues#3055
2017-09-05 01:33:47 +02:00
Marek Marczykowski-Górecki
9d062c4c66
devices: implement DeviceCollection.update_persistent()
Allow attached device to be converted from persistent to non-persistent
and the other way around.
This is to allow starting a VM with some device attached temporarily.
When VM is not running, it is possible to attach device only
persistently, so this change will allow to do that, then, after starting
the VM, change it to non-persistent - so it will not be attached again
at further startups.

QubesOS/qubes-issues#3055
2017-09-05 01:33:47 +02:00
Marek Marczykowski-Górecki
1f5d43a094
devices: move DeviceInfo definition earlier
Make it in logical order. And allow DeviceAssignment and
DeviceCollection use DeviceClass type, for example in annotations.
2017-09-05 01:33:46 +02:00
Marek Marczykowski-Górecki
8c1ba905a0
api: do not fail events when listener is cancelled in the meantime
Raising any error from event handler will abort further handlers of this
event.
2017-09-05 01:33:46 +02:00
Wojtek Porczyk
6a93137bbe qubes/app: remove .app attr from VMMConnection 2017-09-01 19:46:28 +02:00
Wojtek Porczyk
029ec9cb38 fix some more fd leaks 2017-08-31 20:30:24 +02:00
Wojtek Porczyk
6ff1bfdc16 qubes/tests: convert some tearDown into addCleanup
This is because .tearDown() is not executed if the exception occurs in
setUp() [for example self.skipTest() raises an exception]. The lower
levels of .tearDown() being executed are critical to not leaking file
descriptors.
2017-08-31 20:30:24 +02:00
Wojtek Porczyk
d0f2fdba55 qubes/tests: fix some of the fd leaks
This is WIP, there are more fds leaking.
2017-08-31 20:30:24 +02:00
Wojtek Porczyk
f1334e5095 qubes/tests/run: --break-to-repl 2017-08-31 20:30:24 +02:00
Wojtek Porczyk
25670fbd61 qubes.tests.list_templates(), and .close() the app 2017-08-31 20:30:23 +02:00
Wojtek Porczyk
de8ff20976 qubes: Allow for explicit closing of objects 2017-08-31 20:30:23 +02:00
Wojtek Porczyk
8547d06cc9 qubes/tests: skip env-dependent tests using decorator
Gtk segfaults without X11 display. Some tests need dedicated PCI device.
2017-08-31 20:29:34 +02:00
Wojtek Porczyk
044e10a6ec Merge remote-tracking branches 'origin/pull/142/head', 'origin/pull/143/head' and 'origin/pull/144/head' 2017-08-14 16:32:56 +02:00
Marek Marczykowski-Górecki
3f33a7bc2c
fix and enable ServicesExtension
This extension is responsible for communicating service.* features to
VMs - in other words, qvm-service framework

Fixes QubesOS/qubes-issues#3019
2017-08-14 02:30:52 +02:00
Marek Marczykowski-Górecki
7a9e5c3650
tests: qrexec-policy cli tool tests 2017-08-14 02:24:32 +02:00
Marek Marczykowski-Górecki
9f7915d998
Forbid creating VM with name 'none' or 'default'
Those were special names in Qubes 3.x, don't allow such VMs to avoid
(potentially fatal) confusion.

Fixes QubesOS/qubes-issues#3002
2017-08-14 02:24:32 +02:00
Marek Marczykowski-Górecki
6e5fe58128
storage/lvm: fix Volume() instance init when physical volume is unavailable
First, cache objects created with init_volume - this is the only place
where we have full volume configuration (including snap_on_start and
save_on_stop properties).
But also implement get_volume method, to get a volume instance for given
volume id. Such volume instance may be incomplete (other attributes are
available only in owning domain configuration), but it will be enough
for basic operations - like cheching and changing its size, cloning
etc.
Listing volumes still use list of physically present volumes.

This makes it possible to start qubesd service, without physical
presence of some storage devices. Starting VMs using such storage would
still fail, of course.

Fixes QubesOS/qubes-issues#2960
2017-08-12 22:44:03 +02:00
Marek Marczykowski-Górecki
9ad85a3dff
storage: fix getting list of pool driver parameters
function.func_code existed in python2. Use inspect module (instead of
__code__ attribute).
2017-08-12 22:42:30 +02:00
Marek Marczykowski-Górecki
3338854b1c
storage: fix storage.import_data_end when given Volume instance
Leftover from '0f12870 storage: use direct object references, not only
identifiers'
2017-08-12 22:39:41 +02:00
Marek Marczykowski-Górecki
0cd25b5350
firewall: fix Firewall.clone()
New rule require action to be defined, even if will be overwritten a
moment later.
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
a833692c63
storage/lvm: extend volume snapshot of running VM too
Extend both backing volume as well as current snapshot, so the increased
size is visible immediately to the user.

Fixes QubesOS/qubes-issues#2982
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
8488d5dc72
vm/adminvm: add default_dispvm property to AdminVM
This is useful to select default DispVM template for VMs started
directly by the user. This makes sense as long as AdminVM == GUIVM.

QubesOS/qubes-issues#2974
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
971c7d4ac9
api/admin: add admin.vm.CreateDisposable in place of internal.vm.Create.DispVM
Add public Admin API call to create Disposable VM that would be
automatically destroyed after shutdown. Do not keep this functionality
for qrexec-policy tool only.
Also, use admin.vm.Start there, instead of internal.vm.Start and
admin.vm.Kill instead of internal.vm.CleanupDispVM (this is enough,
because DispVM now have auto_cleanup property).

QubesOS/qubes-issues#2974
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
691a6f4d8c
vm/dispvm: add auto_cleanup property, unify creating new DispVM
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.

New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).

Tests included.

QubesOS/qubes-issues#2974
2017-08-06 20:54:09 +02:00
Marek Marczykowski-Górecki
22f2fe6d69
block: add support for devtype option
QubesOS/qubes-issues#2951
2017-08-01 15:20:36 +02:00
Marek Marczykowski-Górecki
a407e34931
devices: always pass persistent parameter to device-list-attached event 2017-08-01 15:20:19 +02:00
Marek Marczykowski-Górecki
b9c217704c
storage/file: fix disk usage reporting
Do not report root image usage in template-based VMs. This space is used
only once - by template.
2017-07-30 18:36:23 +02:00
Marek Marczykowski-Górecki
8a8674bb57
ext/core_features: add handling 'qubes-firewall' feature request
VM (template) can announce whether it support enforcing firewall rules
or not.

Fixes QubesOS/qubes-issues#2003
2017-07-30 18:34:43 +02:00
Marek Marczykowski-Górecki
71a1be30e7
app: fix LVM default storage pool 2017-07-30 14:57:26 +02:00
Marek Marczykowski-Górecki
36f1a3abaf
Merge branch 'services'
* services:
  tests: check clockvm-related handlers
  doc: include list of extensions
  qubesvm: fix docstring
  ext/services: move exporting 'service.*' features to extensions
  app: update handling features/service os ClockVM
2017-07-29 05:09:32 +02:00
Marek Marczykowski-Górecki
34ff40c721
tests: fix AdminVM test 2017-07-29 05:07:45 +02:00
Marek Marczykowski-Górecki
c8c32d4956
Merge branch 'stats'
* stats:
  tests: admin.vm.Stats
  api: fix handling interrupted calls
  api/admin: implement admin.vm.Stats
  app: refresh getting VM statistics, rename to QubesHost.get_vm_stats
  api: use str(subject) instead of explicit subject.name
2017-07-29 05:05:32 +02:00
Marek Marczykowski-Górecki
76640df091
Merge branch 'tests-storage'
* tests-storage:
  tests: register libvirt events
  tests: even more agressive cleanup in tearDown
  app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
  api: keep track of established connections
  tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
  storage: fix Storage.clone and Storage.clone_volume
  tests: more tests fixes
  firewall: raise ValueError on invalid hostname in dsthost=
  qmemman: don't load qubes.xml
  tests: fix AdminVM test
  tests: create temporary files in /tmp
  tests: remove renaming test - it isn't supported anymore
  tests: various fixes for storage tests
  tests: fix removing LVM volumes
  tests: fix asyncio usage in some tests
  tests: minor fixes to api/admin tests
  storage/file: create -cow.img only when needed
  storage: move volume_config['source'] filling to one place
  app: do not create 'default' storage pool
  app: add missing setters for default_pool* global properties
2017-07-29 05:01:42 +02:00
Marek Marczykowski-Górecki
639fa26079
Merge branch 'qdb-watch'
* qdb-watch:
  tests: add qdb_watch test
  ext/block: make use of QubesDB watch
  vm: add API for watching changes in QubesDB
  vm: optimize imports
  api/admin: don't send internal events in admin.Events
  Add explanation why admin.vm.volume.Import is a custom script
  Follow change of qubesdb path return type
  Rename vm.qdb to vm.untrusted_qdb
2017-07-29 05:01:13 +02:00
Marek Marczykowski-Górecki
46bfa14d08
storage/lvm: make sure that volumes have /dev entries when export() is called
In some cases (even manual manipulation) those files may not be present
- especially because thin snapshots do not have nodes in /dev by default.
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
27ce27b6e9
backup: improve error reporting about passphrase retrieval
QubesOS/qubes-issues#2931
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
bde8d45cca
backup: update last backup timestamp only when backup succeed
QubesOS/qubes-issues#2931
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
1556814f59
backup: fix handling already encoded passphrase
When passphrase is retrieved from VM, it is already encoded. Do not try
to encode it again.

QubesOS/qubes-issues#2931
2017-07-29 04:47:43 +02:00
Marek Marczykowski-Górecki
73fb16dbda
tests: check clockvm-related handlers 2017-07-28 16:32:48 +02:00
Marek Marczykowski-Górecki
5321e11003
qubesvm: fix docstring 2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
1a6728cb12
ext/services: move exporting 'service.*' features to extensions
This means core code will not publish any features by default.
2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
e738a7596d
app: update handling features/service os ClockVM
Threis no more ntpd service used - new approach do not conflict with
ntpd. Because of this, new feature is named 'service.clocksync', and
should be _enabled_ in ClockVM ('ntpd' was disabled there).

QubesOS/qubes-issues#1230
2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
9bbefa0edb
tests: admin.vm.Stats
QubesOS/qubes-issues#853
2017-07-28 03:02:33 +02:00
Marek Marczykowski-Górecki
2f4b4d97e7
api: fix handling interrupted calls
When an API call is interrupted, the relevant coroutine is cancelled -
which means it may throw CancelledError. At the same time, cancelled
call have related socket already closed (and transport set to None). But
QubesDaemonProtocol.respond try to close the transport again, which
fails. Fix handling this case.
2017-07-28 03:02:33 +02:00
Marek Marczykowski-Górecki
147bca1648
api/admin: implement admin.vm.Stats
QubesOS/qubes-issues#853
2017-07-28 03:02:32 +02:00
Marek Marczykowski-Górecki
e8b875f552
app: refresh getting VM statistics, rename to QubesHost.get_vm_stats
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.

QubesOS/qubes-issues#853
2017-07-27 22:16:03 +02:00
Marek Marczykowski-Górecki
0d9574d9fc
api: use str(subject) instead of explicit subject.name
This allows better flexibility, when subject is not necessary a VM
object.
2017-07-27 22:15:02 +02:00
Marek Marczykowski-Górecki
6a85592b51
tests: register libvirt events
This is especially needed to have proper domain-shutdown event.
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
c53582b285
tests: even more agressive cleanup in tearDown
Remove some more references to objects holding (possibly indirectly)
reference to libvirt connection:
 - local variables in tearDown function
 - running Admin API calls (especially admin.Events)
 - vmm._libvirt_conn directly, in case some reference to Qubes()
   is still there
 - any instance attribute that is an object from 'qubes' python package
   (instead of just those descending from BaseVM)
 - do not create new Qubes() instance for removing VMs - if we already
   have one in self.app

Then trigger garbage collector to really cleanup those objects (and
close relevant file descriptors). It's important do do this before
closing event loop, because some of descructors may try to use it (for
example remove registered handlers).
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
d57120d809
app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
close() function purpose is to close the connection...
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
074c705f77
api: keep track of established connections
This will be needed to gracefuly cleanup them between individual tests.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
4be325a383
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
When tearDownClass is executed, event loop is already closed. Since no
test really need it right now, drop support for test class-wide VMs and
convert those methods back to instance methods.
Also put coroutines (vm.remove_from_disk, vm.kill) onto event loop.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
910f793c61
storage: fix Storage.clone and Storage.clone_volume
1. Volume.import_volume expect volume to be created first.
2. asyncio.wait do not accept generator, it must be concrete iterable.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
1af7034d57
tests: more tests fixes 2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
214c646417
firewall: raise ValueError on invalid hostname in dsthost=
...instead of AssertionError
2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
625c94d1f6
qmemman: don't load qubes.xml
Only qubesd should load qubes.xml directly. Put a TODO comments for now
in place of slow VM reporting, invent some better mechanism later.

This loading of qubes.xml caused deadlocks, because qmemnan kept open
file descriptor (in locked state).
2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
5d9bc00885
tests: fix AdminVM test
Since it is no longer child of QubesVM, constructor do not take 'qid'
and 'name' arguments.
Also:
- remove other dropped properties tests (netvm, storage related)
- make the test working in non-dom0
2017-07-26 03:26:40 +02:00
Marek Marczykowski-Górecki
a9934316c1
tests: create temporary files in /tmp 2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8bae2d39c2
tests: remove renaming test - it isn't supported anymore
QubesOS/qubes-issues#2868
2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8b2db94b41
tests: various fixes for storage tests
- improve TestPool mock - init_volume now return appropriate mock type,
   instead of TestPool
 - improve patching base directory (/var/lib/qubes) - it is stored in
   more than one place...
 - fix inheritance in TC_01_ThinPool class
 - fix expected LVM volume names ('vm-' prefix)
 - fix cleanup after FilePool tests - remove temporary qubes.xml
 - asyncio usage
 - better reporting in integ.storage - include error message in the
   report, not only as a comment in code
2017-07-26 03:26:13 +02:00
Marek Marczykowski-Górecki
a8e2f3111d
tests: fix removing LVM volumes
VM volumes have 'vm-' name prefix now.
This is continuation of fd5386c storage/lvm: prefix VM LVM volumes with 'vm-'
2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
75394a1348
tests: fix asyncio usage in some tests 2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
ba2b034843
tests: minor fixes to api/admin tests
Drop debug print, fix mocking Storage.remove (it is a coroutine).
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
4208897146
storage/file: create -cow.img only when needed
Do not create it at volume creation time. It it needed only when VM is
running, so create it just before startup only.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
c5667791e8
storage: move volume_config['source'] filling to one place
Don't set 'source' volume in various places (each VM class constructor
etc), do it as part of volume initialization. And when it needs to be
re-calculated, call storage.init_volume again.

This code was duplicated, and as usual in such a case, those copies
were different - one have set 'size', the other one not.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
3a21e1f1b3
app: do not create 'default' storage pool
Since we have app.default_pool* properties, create appropriately named
pool and let those properties choose the right pool. This also means we
don't need to specify pool name in default volume config anymore

QubesOS/qubes-issues#2256
2017-07-25 06:20:42 +02:00
Marek Marczykowski-Górecki
73a78ca0db
app: add missing setters for default_pool* global properties
Function itself was already written, but not set in the properties.

QubesOS/qubes-issues#2256
2017-07-25 05:21:32 +02:00
Marek Marczykowski-Górecki
1933a720d0
tests: add qdb_watch test
Use dom0 instance of QubesDB, to be independent of working (or not) VMs
at this stage.

QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
67c382c8b0
ext/block: make use of QubesDB watch
Actually use just introduced API.

Also document new `device-list-change:class` event.

QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
b7f0cf7d82
vm: add API for watching changes in QubesDB
Provide an API for use QubesDB.watch() inside of qubesd.

Fixes QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
d5b94d1cbd
vm: optimize imports
Remove unused, sort others.
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
354aa14c53
api/admin: don't send internal events in admin.Events
Some events are internal for a sole purpose of getting some data from
extension. Since listeners of admin.Events cannot return anything, there
is no sense in sending those events there.
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
0fb7c1fbed
Follow change of qubesdb path return type
It's str/unicode, not bytes now.

QubesOS/qubes-issues#2937
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
1759bca00f
Rename vm.qdb to vm.untrusted_qdb
QubesDB can be freely modified by a VM, so one should take care when
reading any data retrieved from it.

Fixes QubesOS/qubes-issues#2934
2017-07-24 13:01:55 +02:00
Marek Marczykowski-Górecki
397a8263bd
tarwriter: use new PAX format, instead of old GNU for sparse files
The old format have many issues and is discouraged by tar developers. In
this case the most important one is header with possible non-ASCII
characters, which will result in UnicodeDecodeError (tarfile module
require header parts in utf-8).
PAX format is much cleaner, as it use standard mechanism for extended
headers.
2017-07-21 03:14:06 +02:00
Marek Marczykowski-Górecki
abdad8c2b2
backup: allow backup running VMs
Since we have LVM by default, it is possible to backup VMs while they
are running. For now it will include its state from before startup, but
later we may implement some other logic (a snapshot of running VM).
2017-07-21 03:12:23 +02:00
Marek Marczykowski-Górecki
1ec0060577
backup: check inner archiver exit code
If inner tar returns with non-zero code, mostly like the backup is
incomplete, so it is very important to report it.
2017-07-21 03:11:02 +02:00
Marek Marczykowski-Górecki
82918ee704
api/admin: gracefuly report cancelled backup operation
QubesOS/qubes-issues#2931
2017-07-21 03:10:26 +02:00
Marek Marczykowski-Górecki
088c838ac3
storage: typo fix in comment 2017-07-20 12:19:02 +02:00
Marek Marczykowski-Górecki
dbea03ad1f
tests: admin.backup.*
Those tests are only about Admin API integration, not about actual backup
backend code

QubesOS/qubes-issues#2931
2017-07-20 12:19:01 +02:00
Marek Marczykowski-Górecki
74375146e8
tools/qubesd: increase loglevel with --debug option 2017-07-20 12:19:01 +02:00
Marek Marczykowski-Górecki
6dbce8259f
api/admin: plug backup into Admin API
Fixes QubesOS/qubes-issues#2931
2017-07-20 12:19:01 +02:00
Marek Marczykowski-Górecki
81246cac64
backup: fix get_backup_summary
QubesOS/qubes-issues#2931
2017-07-20 12:09:17 +02:00
Marek Marczykowski-Górecki
579d90e174
backup: use proper API for getting volumes list and size
Do not assume static list of volume (although it is true for now), and
also use proper API for getting volume size, instead of assuming it's a
normal file.
2017-07-20 12:09:17 +02:00
Marek Marczykowski-Górecki
9dd0cfe6c5
backup: drop BackupHeader loading code
It is useless here since backup restore is moved to
qubes-core-admin-client.
2017-07-20 12:09:16 +02:00
Marek Marczykowski-Górecki
d4e9120903
backup: convert from multiprocessing to asyncio
QubesOS/qubes-issues#2931
2017-07-20 02:48:44 +02:00
Marek Marczykowski-Górecki
801d6acf5c
Rename 'net/*' features to 'net.*'
Same reasoning as with 'service/*' - '/' is not allowed in qrexec (so
Admin API) call argument.

Related to QubesOS/qubes-issues#2906
2017-07-18 03:50:02 +02:00
Marek Marczykowski-Górecki
9e4f2d0907
Merge remote-tracking branch 'qubesos/pr/131'
* qubesos/pr/131:
  When running tests clean up and remove qubesd.socket
  Do not display gi Gtk warning when running tests
2017-07-18 02:19:34 +02:00
Marek Marczykowski-Górecki
c6e09b5a99
storage: fix resize and revert handling
- fix coroutine calling from Admin API
 - fix off-by-one error in resize
 - adjust tests
2017-07-18 01:42:37 +02:00
Marek Marczykowski-Górecki
7f85d81197
Remove qvm-backup tool
The tool itself should live in qubes-core-admin-client and communicate
with actual backup code over Admin API.
2017-07-18 01:42:37 +02:00
Marek Marczykowski-Górecki
2ace5901df
backup: make pylint happy, remove dead settings
After removing backup-restore code the remaining part of backup.py is
not that bad - lets fix it and no longer exclude it from pylint.
2017-07-18 01:42:36 +02:00
Marek Marczykowski-Górecki
61519014cb
Remove qvm-backup-restore and its backend code
This functionality is moved to qubes-core-admin-client, which does it
over Admin API

QubesOS/qubes-issues#1214
2017-07-18 01:30:39 +02:00
Marek Marczykowski-Górecki
4037bf9abc
vm/qubesvm: do not assign all host vCPUs to a qube
Fixes QubesOS/qubes-issues#2900
2017-07-17 12:33:52 +02:00
Marek Marczykowski-Górecki
11fac7be56
vm/qubesvm: minor fix 2017-07-17 12:29:25 +02:00
Marek Marczykowski-Górecki
ef756eb05d
tests: libvirt xml rendering unit test 2017-07-17 12:28:24 +02:00
Marek Marczykowski-Górecki
9ba84ecdad
Add code and test for migrating vm.hvm into vm.virt_mode
This will allow to load old qubes.xml - otherwise previous change render
the system unusable (until manual qubes.xml edit).

QubesOS/qubes-issues#2912
2017-07-17 12:27:17 +02:00
Marek Marczykowski-Górecki
21940bef90
Change vm.hvm into vm.virt_mode
Fixes QubesOS/qubes-issues#2912
2017-07-17 12:26:10 +02:00
Marek Marczykowski-Górecki
8cb831da29
ext/admin: allow setting 'created-by-*' tags from dom0
Add an exception for this limit - if for nothing else, to allow full
backup restore (non-paranoid mode).
2017-07-17 02:42:36 +02:00
Marek Marczykowski-Górecki
cbc7241a93
api/admin: rename mgmt-permission:* event to admin-permission:*
Be more consistent with Admin API name and method names.
2017-07-17 02:41:42 +02:00
Bahtiar `kalkin-` Gadimov
3be7b23f46
When running tests clean up and remove qubesd.socket
Changed the inheritance hierarchy:

1. Renamed `SystemTestsMixin` to `SystemTestCase`
2. `SystemTestCase` is a child of `QubesTestCase`
3. All classes extending the prior `SystemTestsMixin` now just extend `object`
2017-07-14 14:54:04 +02:00
Marek Marczykowski-Górecki
ab7ed776f9
Merge branch 'dispvm-fix'
* dispvm-fix:
  storage: add comment about source volume lookup
  vm/dispvm: convert DispVM related function to coroutines
  vm/dispvm: fix DispVM storage definition
2017-07-14 01:13:20 +02:00
Marek Marczykowski-Górecki
a42e500108
Merge branch 'tests-fixes-1'
* tests-fixes-1:
  api: extract function to make pylint happy
  tests/vm: simplify AppVM storage test
  storage: do not use deepcopy on volume configs
  api: cleanup already started servers when some later failed
  tests: fix block devices tests when running on real system
  tests: fix some FD leaks
2017-07-14 01:05:21 +02:00
Marek Marczykowski-Górecki
63bac15b25
Merge remote-tracking branch 'qubesos/pr/130'
* qubesos/pr/130:
  qubes: have "service" features' keys separated by period
2017-07-14 01:04:42 +02:00
Marek Marczykowski-Górecki
1e45d297b3
storage: add comment about source volume lookup 2017-07-12 21:37:12 +02:00
Marek Marczykowski-Górecki
66b88cc943
api: extract function to make pylint happy 2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
2df81adab1
tests/vm: simplify AppVM storage test
Use minimal TestPool(), instead of Mock().
This allow effectively compare Volume instances (init_volume with the
same parameters return equal Volume instance).
2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
f2f89c7b0c
storage: do not use deepcopy on volume configs
There may be Pool or Volume object references, which is intentional to
keep them as is - not copy whole Pool/Volume objects.
2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
e6d5337fa7
api: cleanup already started servers when some later failed 2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
d7b968d867
tests: fix block devices tests when running on real system 2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
7aa8d74ad4
tests: fix some FD leaks
There are still much more of them...
2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
58f4369a7e
make pylint happy 2017-07-12 21:14:23 +02:00
Marek Marczykowski-Górecki
c32f0db582
vm/dispvm: convert DispVM related function to coroutines
Some functions used there (create_on_disk, remove_from_disk, kill) are
coroutines, so callers needs to be too.

Fixes QubesOS/qubes-issues#2896
2017-07-08 03:04:55 +02:00
Marek Marczykowski-Górecki
148d1cda78
vm/dispvm: fix DispVM storage definition
Specify empty 'source' field, so it gets filled with appropriate
template's images. Then also fix recursive 'source' handling - DispVM
root volume should point at TemplateVM's root volume as a source, not a
AppVM's one - which is also only a snapshot.

Fixes QubesOS/qubes-issues#2896
2017-07-08 02:58:21 +02:00
Wojtek Porczyk
b04f612374 qubes: have "service" features' keys separated by period 2017-07-05 04:16:16 +02:00
Marek Marczykowski-Górecki
dddd94b339
Merge branch 'core3-storage3' 2017-07-04 16:44:53 +02:00
Marek Marczykowski-Górecki
6db39345fb
Merge remote-tracking branch 'origin/core3-quick-fixes-20170703' 2017-07-04 16:11:18 +02:00
Marek Marczykowski-Górecki
51022cada5
Merge remote-tracking branch 'origin/policy-adminvm' 2017-07-04 15:51:30 +02:00
Marek Marczykowski-Górecki
cfbccc0a74
vm: drop old is_outdated/commit_changes
This code is unused now. Theoretically this is_outdated implementation
should be moved to FileVolume, but since we don't have VM reference
there, it isn't possible to read appropriate xenstore entry. As we're
phasing out file pool, simply ignore it for now.

QubesOS/qubes-issues#2256
2017-07-04 14:25:10 +02:00
Marek Marczykowski-Górecki
12adf8bede
storage/lvm: major cleanup, update
- remove obsolete volume types, use snap_on_start/save_on_stop directly
- handle multiple revisions
- implement is_outdated()

QubesOS/qubes-issues#2256
2017-07-04 14:25:09 +02:00
Marek Marczykowski-Górecki
c7ca4a445e
storage/kernels: support only save_on_stop=False volumes
LinuxKernel pool support only read-only volumes, so save_on_stop=True
doesn't make sense. Make it more explicit - raise NotImplementedError
otherwise.
Also, migrate old configs where snap_on_start=True, but no source was
given.

QubesOS/qubes-issues#2256
2017-07-04 14:25:08 +02:00
Marek Marczykowski-Górecki
0e554296e3
storage: drop 'internal' and 'removable' volume properties
Since dynamic volumes (qvm-block) are moved to devices API, those two
are not needed anymore.

QubesOS/qubes-issues#2256
2017-07-04 14:25:08 +02:00
Marek Marczykowski-Górecki
075d991114
config: eliminate duplicated qubes_base_dir
Remove it from system_path dict, have it in one place.
2017-07-04 14:25:07 +02:00
Marek Marczykowski-Górecki
317d140f46
storage/file: major FilePool/FileVolume cleanup and documentation
This driver isn't used in default Qubes 4.0 installation, but if we do
have it, let it follow defined API and its own documentation. And also
explicitly reject not supported operations:
 - support only revisions_to_keep<=1, but do not support revert() anyway
 (implemented version were wrong on so many levels...)
 - use 'save_on_stop'/'snap_on_start' properties directly instead of
 obsolete volume types
 - don't call sudo - qubesd is running as root
 - consistently use path, path_cow, path_source, path_source_cow

Also, add tests for BlockDevice instance returned by
FileVolume.block_device().

QubesOS/qubes-issues#2256
2017-07-04 14:25:07 +02:00
Marek Marczykowski-Górecki
1a1dd3dba2
storage: make default pool configurable
Do not always use pool named 'default'. Instead, have global
`default_pool` property to specify default storage pools.
Additionally add `default_pool_*` properties for each VM property, so
those can be set separately.

QubesOS/qubes-issues#2256
2017-07-04 14:25:06 +02:00
Marek Marczykowski-Górecki
8318038a73
tests: workaround asyncio.iscouroutinefunction(Mock)=True
It's already fixed in Python 3.5.3, but on Travis we have Python 3.5.2.
https://bugs.python.org/issue28703
2017-07-04 13:04:24 +02:00
Marek Marczykowski-Górecki
b5ad245614
vm/qubesvm: have VM lifecycle related events async
Those functions are coroutines anyway, so allow event handlers to be
too.
Some of this (`domain-create-on-disk`, `domain-remove-from-disk`) will
be useful for appmenus handling.
2017-07-04 13:04:24 +02:00
Marek Marczykowski-Górecki
d00e4538bf
devices: make attach/detach related events async
This will allow starting processes and calling RPC services in those
events. This if required for usb devices, which are attached using RPC
services.
Intentionally keep device listing events synchronous only - to
discourage putting long-running actions there.

This change also require some not-async attach method version for
loading devices from qubes.xml - have `load_persistent` for this.
2017-07-04 13:04:23 +02:00
Marek Marczykowski-Górecki
e5a9c46e3d
typos in docstrings 2017-07-04 13:04:22 +02:00
Marek Marczykowski-Górecki
ea1a04cb19
events: add support for async event handlers
See documentation for details.
2017-07-04 13:04:22 +02:00
Marek Marczykowski-Górecki
6238254f49
events: merge fire_event and fire_event_pre functions
Those functions really do very similar things - lets merge them and add
simple parameter.
2017-07-04 13:04:21 +02:00
Marek Marczykowski-Górecki
aed6de4937
events: typos in docstring 2017-07-04 12:58:39 +02:00
Marek Marczykowski-Górecki
3c4b60fa45
vm/adminvm: (re-)add vm.qdb property
Dom0 also expose some information in QubesDB - like list of block
devices.
2017-07-04 12:58:39 +02:00
Marek Marczykowski-Górecki
9d17c52a8d
tests: add missing import 2017-07-04 05:22:39 +02:00
Marek Marczykowski-Górecki
f393ce62dc
Fix docstrings formatting
Fix warnings/errors reported by Sphinx
2017-07-04 04:27:36 +02:00
Marek Marczykowski-Górecki
291a338e73
api: add missing docstring 2017-07-04 04:27:34 +02:00
Marek Marczykowski-Górecki
82c3f85042
storage: add API documentation
QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
5971873680
storage: drop functions not being part of the API
commit/recover/reset should really be handled in start/stop. Nothing
stops specific pool implementation to define such functions privately.

QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
820539e909
storage: make volume snap_on_start/save_on_stop explicit
Always define those properties, always include them in volume config.
Also simplify overriding pool based on volume type defined by those:
override pool unless snap_on_start=True.

QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
697eb05c20
storage: drop rename support
Since VM name is immutable, rename method can be dropped from storage
API.

QubesOS/qubes-issues#2868
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
dd1e05dc83
vm: drop rename related methods
Since VM name in immutable now, drop rename-related methods.

QubesOS/qubes-issues#2868
2017-07-04 04:11:13 +02:00
Marek Marczykowski-Górecki
2dbb51a38d
exc/block: fix attaching block device exposed by dom0
Don't set <backenddomain> element in libvirt XML when device is provided
by dom0.
2017-07-04 04:09:28 +02:00
Marek Marczykowski-Górecki
3721ae3e8b
vm/adminvm: setup libvirt_domain property
Since libvirt do provide object for dom0 too, return it here.
It's much easier than special-casing AdminVM everywhere. And in fact
sometimes it is actually useful (for example attaching devices from/to
dom0, adjusting memory).
2017-07-04 04:09:27 +02:00
Marek Marczykowski-Górecki
e5de8f4115
devices: better handle exceptions in device extension
Do not fail app.save() just because listing devices failed, for any
reason.
2017-07-04 04:09:27 +02:00
Marek Marczykowski-Górecki
3748eb3e2b
Merge branch 'core3-firewall2' 2017-07-04 03:38:59 +02:00
Marek Marczykowski-Górecki
f83c516082
Merge branch 'two-stage-clone' 2017-07-04 03:37:41 +02:00
Marek Marczykowski-Górecki
d0663940a7
qubes/api/admin: annotate API methods
Second attempt: this time use full words for scope, read, write,
execute.

QubesOS/qubes-issues#2871
2017-06-27 06:01:58 +02:00
Wojtek Porczyk
3e0d01cfb9
qubes/admin: Add listing of API methods 2017-06-27 06:01:58 +02:00
Marek Marczykowski-Górecki
2abdbc4628
firewall: minor simplification for old firewall.xml loading
Have `default_policy_is_accept` variable of type bool, instead of
`policy`, which is only compared to a constant value (`accept`).
Suggested by @woju
2017-06-26 18:45:59 +02:00
Marek Marczykowski-Górecki
2b963be9c7
firewall: pass untrusted values as keyword arguments
Use keyword arguments to pass untrusted arguments to make sure the
function parameter also have `untrusted_` prefix.
Suggested by @woju
2017-06-26 18:41:27 +02:00
Marek Marczykowski-Górecki
26a9974432
api/admin: split vm.volume.Clone to CloneFrom and CloneTo
The first operation returns a token, which can be passed to the second
one to actually perform clone operation. This way the caller needs have
power over both source and destination VMs (or at least appropriate
volumes), so it's easier to enforce appropriate qrexec policy.

The pending tokens are stored on Qubes() instance (as QubesAdminAPI is
not persistent). It is design choice to keep them in RAM only - those
are one time use and this way restarting qubesd is a simple way to
invalidate all of them. Otherwise we'd need some additional calls like
CloneCancel or such.

QubesOS/qubes-issues#2622
2017-06-26 13:27:48 +02:00
Marek Marczykowski-Górecki
3dcd29afea
api/admin: remove admin.vm.Clone operation
The same can be achieved with Create+volume.Clone

QubesOS/qubes-issues#2622
2017-06-26 13:10:00 +02:00
Marek Marczykowski-Górecki
fabd8119b4
storage: volume.import_volume now expect create()d volume
This is much more logical for *import*_volume function.

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
28f78ed3b8
storage/lvm: minor fixes
QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
a748b393f4
storage: move remove() to Volume
This is continuation of 0f12870 "storage: use direct object references,
not only identifiers".

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
ae600e24bf
storage: simplify pool.volumes usage
Add convenient collection wrapper for easier getting selected volume.
Storage pool implementation may still provide only volume listing
function (pool.list_volumes), or, additionally, optimized
pool.get_volume.

This means it is both possible to iterate over volumes:
```python
for volume in pool.volumes:
    ...

```

And get a single volume:
```python
volume = pool.volumes[vid]
```

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
0200fdadcb
api/admin: firewall-related methods
In the end firewall is implemented as .Get and .Set rules, with policy
statically set to 'drop'. This way allow atomic firewall updates.

Since we already have appropriate firewall format handling in
qubes.firewall module - reuse it from there, but adjust the code to be
prepared for potentially malicious input. And also mark such variables
with untrusted_ prefix.

There is also third method: .Reload - which cause firewall reload
without making any change.

QubesOS/qubes-issues#2622
Fixes QubesOS/qubes-issues#2869
2017-06-26 13:08:27 +02:00
Marek Marczykowski-Górecki
65d15e6040
api/admin: skip firewall in vm.Clone
This operation is going to be removed, so apply a quick fix for tests.

QubesOS/qubes-issues#2622
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
842efb577d
firewall: always use policy 'drop'
There is a problem with having separate default action ("policy") and
rules because it isn't possible to set both of them atomically at the
same time.
To solve this problem, always have policy 'drop' (as a safe default),
but by default have a single rule with action 'accept'

Fixes QubesOS/qubes-issues#2869
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
919841635b
qubes: validate if property value consists of ASCII only earlier
Do this for all standard property types - even if other types do
additional validation, do not expose them to non-ASCII characters.

QubesOS/qubes-issues#2622
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
3074a4064c
vm/qubesvm: forbid changing VM name
We've decided to make VM name immutable. This is especially important
for Admin API, where some parts (especially policy) are sticked to the
VM name.

Now, to rename the VM, one need to clone it under new name (thanks to
LVM, this is very quick action), then remove the old one.

Fixes QubesOS/qubes-issues#2868
2017-06-26 02:00:45 +02:00
Marek Marczykowski-Górecki
a6c16d00be
vm/qubesvm: fix handling empty kernel value
Do not fail on empty kernel value - which means "use booloader on
root.img".

Fixes 3ddc052 "vm: move kernel presence validation to event handler"
2017-06-26 01:59:39 +02:00
Marek Marczykowski-Górecki
26013122a0
Merge remote-tracking branch 'woju/devel-adminext' into core3-devel 2017-06-23 10:34:11 +02:00
Marek Marczykowski-Górecki
4208a98bd7
Merge branch 'core3-devel-20170619' 2017-06-23 02:53:17 +02:00
Marek Marczykowski-Górecki
f976f7ec6c
storage: simplify coroutine handling
Suggested by @woju
2017-06-23 02:35:49 +02:00
Marek Marczykowski-Górecki
57e293eb54
Merge branch 'core3-qmemman-fix' 2017-06-22 23:16:35 +02:00
Wojtek Porczyk
8c9ce0587b ext/admin: add explanation to PermissionDenied 2017-06-22 13:21:37 +02:00
Wojtek Porczyk
2942f8bcac qubes: admin extension
for managing tags
2017-06-21 23:12:54 +02:00
Marek Marczykowski-Górecki
c1f4c219f9
tests: adjust TC_00_QubesDaemonProtocol for reorganized api module 2017-06-21 06:59:58 +02:00
Marek Marczykowski-Górecki
588ff04f0d
qmemman: fix units on meminfo parsing
meminfo (written by VM) is expected report KiB, but qmemman internally
use bytes. Convert units.
And also move obscure unit conversion in is_meminfo_suspicious to more
logical place in sanitize_and_parse_meminfo.
2017-06-21 06:34:00 +02:00
Marek Marczykowski-Górecki
ea0cbe3a56
tests: improve tests for qrexe exit code handling
Check if exit code retrieved from dom0 is really the one expected.

Fix typo in test_065_qrexec_exit_code_vm (testvm1/testvm2), adjust for
reporing remote exit code and remove expectedFailure.

QubesOS/qubes-issues#2861
2017-06-21 05:23:35 +02:00
Marek Marczykowski-Górecki
a73dcf6016
tests: wait for session in tests requiring running GUI
Since tests expose qubesd socket, qvm-start-gui should handle starting
GUI daemons (so, GUI session inside VM). Add synchronization with it
using qubes.WaitForSession service.
2017-06-21 04:45:46 +02:00
Marek Marczykowski-Górecki
376ac4b32d
tests: fix vm.run_for_stdio in some more places
When test expect to wait for remote process, use vm.run_for_stdio.
Additionally, when the call fail, (stdout, stderr) is not assigned - use
the one attached to exception object instead.
2017-06-21 04:33:10 +02:00
Marek Marczykowski-Górecki
a0f616f14e
tests: fix checking exit code
Since run_for_stdio raise an exception for non-zero exit code, it isn't
ignored anymore. So, check if qrexec-client-vm return expected value,
instead of keep ignoring it.

QubesOS/qubes-issues#2861
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
72240c13b6
tests: fix vm_qrexec_gui/TC_10_Generic/test_000_anyvm_deny_dom0
When method (as expected) raise an exception, service output would not be
assigned. Extract it from exception object.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
a469c565f4
tests: fix QrexecPolicyContext
Flush new policy file to the disk, otherwise it will stay only in write
buffer.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
3ddc052af3
vm: move kernel presence validation to event handler
Setter is called also on qubes.xml load, so missing kernel breaks
qubes.xml parsing - for example qubesd startup to fix that property.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
4241b39b94
tests: fix tests cleanup 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
60443259d0
vm: raise CalledProcessError on failed service/command call 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
51a17ba749
tests: do not reload qubes.xml
In core3 this isn't needed anymore (and unit tests already check if
that's really true).
2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
ea5ca79133
tests: fix removing test VMs
Do it before shutting down qubesd socket - some things may require it
for VM removal/shutdown.
2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
eee6ab0c01
tests: use copy of qubes.xml, instead of empty one 2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
984a070f3e
tests: move create_*_file to SystemTestsMixin 2017-06-21 00:17:40 +02:00
Wojtek Porczyk
f56f7d13fb
tests/integ/vm_qrexec_gui: skip test_051_qrexec_simple_eof_reverse
QubesOS/qubes-issues#2851
2017-06-21 00:17:39 +02:00
Wojtek Porczyk
139f18fa1d
qubes/tests/integ/vm_qrexec_gui: some fixes 2017-06-21 00:17:39 +02:00
Wojtek Porczyk
0c0b0ea6ef
qubes/tests/integ/vm_qrexec_gui: change time.sleep to asyncio.sleep 2017-06-21 00:17:38 +02:00
Wojtek Porczyk
96a66ac6bd
qubes/api: refactor creating multiple qubesd sockets
Now there is a single function to do this, shared with tests.
2017-06-21 00:17:37 +02:00
Wojtek Porczyk
bec58fc861
qubes/tests: start qrexec policy responder for system tests 2017-06-20 13:00:20 +02:00
Wojtek Porczyk
71a4390fdb
qubes/tools/qubesd: properly unlink UNIX sockets at shutdown 2017-06-20 13:00:20 +02:00
Wojtek Porczyk
4b8e5c3704
qubes/tests/run: refuse to run tests if qubesd is running
Test suite creates some VMs and needs to pass the knowledge about them
to qrexec policy checker. This is done using Admin API, so we need to
substitute qubesd with our own API server.
2017-06-20 13:00:20 +02:00
Wojtek Porczyk
858e547525
qubes: reorganise API protocols
Now instantiating API servers is handled by common function. This is,
among other reasons, for creating ad-hoc sockets for tests.
2017-06-20 13:00:20 +02:00
Marek Marczykowski-Górecki
8196b2d5bf
admin.vm.Create: add commend about 'created-by-' tag 2017-06-20 12:47:01 +02:00
Marek Marczykowski-Górecki
c13cf44e5e
admin.vm.Create: add 'created-by-' tag 2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
083108e995
app: fix registering libvirt event handler
register_event_handlers is called early, when libvirt connection may not
be yet established - especially on empty qubes.xml. Do not skip
automatic connection logic.
2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
e4d285d479
vm/adminvm: make AdminVM sortable
One more thing gone during changing AdminVM base class.
2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
9242202db2
admin: implement admin.vm.tag.*
QubesOS/qubes-issues#2622
2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
4a1a5fc24b
exc: fix QubesNoTemplateError 2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
aadbe223c3
admin: add admin.vm.volume.Clone
QubesOS/qubes-issues#2622
2017-06-20 00:54:15 +02:00
Marek Marczykowski-Górecki
f48b1be669
storage: extract single volume clone into clone_volume
This will be useful for admin.vm.volume.Clone implementation.

QubesOS/qubes-issues#2256
2017-06-20 00:54:15 +02:00
Marek Marczykowski-Górecki
86a935e779
qubes.NotifyTools: ignore '/qubes-tools/version' completely
It isn't used for anything, so simply ignore it for good.

https://github.com/QubesOS/qubes-core-admin/pull/109#discussion_r121421409
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
55669c350c
api/misc: fix retrieving requested features from QubesDB
qdb.list return list of bytes().
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
5209bc370d
vm: init vm.storage and vm.volumes in BaseVM
This way also AdminVM will have (empty) properties there. It is much
cleaner than adding `if hasattr` or catching AttributeError everywhere.
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
160ab964bc
vm: improve sending events for tags and features
1. Send the event after setting tag/feature
2. Provide old value for feature
3. Rename 'key' kwarg to 'feature'
2017-06-14 10:44:23 +02:00
Marek Marczykowski-Górecki
8ba60cd7e9
app: fix listing VMs based on given template 2017-06-13 13:15:21 +02:00
Marek Marczykowski-Górecki
93ccb8bbda
Merge remote-tracking branch 'qubesos/pr/111'
* qubesos/pr/111:
  vm: drop 'internal' property
  qmemman: make sure to release lock
  qmemman: fix meminfo parsing for python 3
  devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
2017-06-12 23:22:55 +02:00
Marek Marczykowski-Górecki
ba4da5cbe0
Merge remote-tracking branch 'qubesos/pr/110'
* qubesos/pr/110:
  storage: use direct object references, not only identifiers
  vm: fix volume_config
  storage/lvm: prefix VM LVM volumes with 'vm-'
  storage: fix VM rename
2017-06-12 23:22:41 +02:00
Marek Marczykowski-Górecki
37245acdcf
vm: change vm.updates_available to a 'updates-available' feature 2017-06-12 12:34:23 +02:00
Marek Marczykowski-Górecki
caa03a9279
vm/qubesvm: simplify 'None' kernel handling 2017-06-12 12:26:57 +02:00
Marek Marczykowski-Górecki
ee442c754f
api/misc: integrate qubes.NotifyTools logic with qubes.FeaturesRequest
Make qubes.NotifyTools reuse logic of qubes.FeaturesRequest, then move
actual request processing to 'features-request' event handler. At the
same time implement handling 'qrexec' and 'gui' features request -
allowing to set template features when wasn't already there.
Behavior change: template is no longer allowed to change feature value
(regardless of being True or False). This means the user will always be
able to override what template have set.
2017-06-12 12:22:39 +02:00
Marek Marczykowski-Górecki
e8c303977b
vm/adminvm: fix str(AdminVM) after changing its base
BaseVM have no (useful) __str__ method.
2017-06-12 10:15:13 +02:00
Marek Marczykowski-Górecki
9d99232515
vm: drop 'internal' property
It isn't used anywhere in the code right now. And when it will be
needed, it should be a "feature" not "property".
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
206b7c02d5
qmemman: make sure to release lock
Even when handling updated meminfo or domain list something goes wrong,
make sure to release the lock - otherwise the whole qmemman will be
blocked.
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
9015414119
qmemman: fix meminfo parsing for python 3
One more place not converted to python 3.
2017-06-12 10:06:42 +02:00
Marek Marczykowski-Górecki
3564250298
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
Drop DeviceInfo.data - device extension should provide a subclass with
proper individual fields.
Drop DeviceAssignment.frontend_domain - this information is redundant -
frontend domain is defined by where DeviceAssignment is attached.
Rename DeviceCollection.devclass to bus - devclass if confusing here,
because this term is also used for DeviceInfo subclass.
2017-06-12 10:06:42 +02:00
Marek Marczykowski-Górecki
0f12870803
storage: use direct object references, not only identifiers
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object

Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.

One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.

Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
80b459b6d5
vm: fix volume_config
- kernel volume shouldn't have snap_on_start, it's read-only volume
  anyway
- root volume of AppVM should have placeholder for 'source'
- private volume of AppVM should _not_ have placeholder for 'source'
(it's ignored anyway, because snap_on_start=False)

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
fd5386cd47
storage/lvm: prefix VM LVM volumes with 'vm-'
This will allow filtering them out in udev rules - to not parse any of
it.

QubesOS/qubes-issues#2319
2017-06-09 04:54:59 +02:00
Marek Marczykowski-Górecki
fd3f19d4a6
storage: fix VM rename
When VM is renamed only volume.vid get updated, but not other attributes
calculated from it. Convert them to dynamic properties to not worry
about it.

QubesOS/qubes-issues#2256
2017-06-09 04:54:50 +02:00
Marek Marczykowski-Górecki
26f25aeecd
vm/mix/net: remove cleanup_vifs
With libvirt in place, this isn't enough - libvirt also keep VM
configuration in its memory and adjusting xenstore doesn't change that.
In fact changing xenstore behind it back make it even worse in some
situations.

QubesOS/qubes-issues#1426
2017-06-06 22:17:00 +02:00
Marek Marczykowski-Górecki
fa1da422d1
vm/adminvm: don't base AdminVM on QubesVM
dom0 isn't real VM and most properties doesn't apply to it. Lets make it
more explicit.
2017-06-06 21:57:27 +02:00
Marek Marczykowski-Górecki
67fcda2cde
tests: basic tests for AppVM
Specifically, check if root volume is updated after template switch.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
89e025ca3b
vm: make vm.get_power_state() return 'Halted' in offline_mode
Again, if libvirt or even Xen isn't running, we can safely assume VM
isn't too.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
313d7422e7
tests: remove obsolete tests, formatting 2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
04fd2ff34a
vm: adjust VM's root volume when changing its template
Re-init volume config of all 'snap_on_start' volumes at template
chanage. For this, save original volume config and re-use
config_volume_from_source function introduced in previous commit.

At the same time, forbid changing template of running AppVM or any
DispVM.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
63c23c835b
vm: reduce code duplication
Move inheriting volume from template to a helper function.

No functional change.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
28737d16ce
Migrate qubes.NotifyTools, qubes.FeaturesRequest, qubes.NotifyUpdates
Make them call into qubesd. Create separate socket for "misc" calls - VM
accessible, but not part of Admin API.
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
43fd1d76e8
tools/qubesd-query: add option to report failure with exit code 2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
05ba73bb3a
api/internal: add methods for handling host suspend 2017-06-06 03:33:29 +02:00
Marek Marczykowski-Górecki
f80fddf7ae
vm: restore code for suspending domains with PCI devices 2017-06-06 03:33:29 +02:00
Marek Marczykowski-Górecki
4c4f903fe3
vm: change kernel=None to kernel=''
vm.kernel property have type 'str'. Putting None there makes a lot of
troubles: it gets encoded as 'None' in qubes.xml and then loaded back as
'None' string, not None value. Also it isn't possible to assign None
value to str property throgh Admin API.

kernel='' is equally good to specify "no kernel from dom0".

QubesOS/qubes-issues#2622
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
d5cd58df51
templatevm: set default netvm to None for templates
Since we have qrexec-based updates proxy, we can even stronger isolate
templates from outside threats.

QubesOS/qubes-issues#1854
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
1ed0b14d93
vm: make vm.hvm=True by default
QubesOS/qubes-issues#2185
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
5d3204c730
vm: remove prepare_dvm parameter from methods and events
It was related to DispVM savefile preparation, but it is no longer
applicable in Qubes 4.0
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
018877a19c
vm: avoid starting the same VM multiple times simultaneously
While libvirt handle locking itself, there is also Qubes-specific
startup part. Especially starting qrexec-daemon and waiting until
qrexec-agent connect to it. When someone will attempt to start VM the
second time (or simply assume it's already running) - qrexec will not be
connected yet and the operation will fail. Solve the problem by wrapping
the whole vm.start() function with a lock, including a check if VM is
running and waiting for qrexec.

Also, do not throw exception if VM is already running.

This way, after a call to vm.start(), VM will be started with qrexec
connected - regardless of who really started it.
Note that, it will not solve the situation when someone check if VM is
running manually, like:

    if not vm.is_running():
        yield from vm.start()

Such code should be changed to simply:

    yield from vm.start()

Fixes QubesOS/qubes-issues#2001
Fixes QubesOS/qubes-issues#2666
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
257c8c0821
devices : make DeviceInfo sortable 2017-06-05 23:33:58 +02:00
Marek Marczykowski-Górecki
b08574acf3
api/internal: make GetSystemInfo not crash on not set default DispVM 2017-06-05 23:33:58 +02:00
Marek Marczykowski-Górecki
b5a4573290
storage: remove DomainPool
Replaced by BlockDevice extension
2017-06-05 23:33:58 +02:00
Marek Marczykowski-Górecki
9bb5054e50
ext: BlockDevices extension
Handle block devices exposed by VMs
2017-06-05 23:33:58 +02:00
Wojtek Porczyk
d9f5192cc2 last fixes (thanks, @marmarek!) 2017-06-05 14:34:06 +02:00
Wojtek Porczyk
dc793be81f Fixes from marmarek's review 2017-06-01 12:31:14 +02:00
Wojtek Porczyk
cb3c9a82b3 qubes/tests: fix qrexec policy context
Now it removes policy file if it was created while entering the context.
2017-06-01 12:31:14 +02:00
Wojtek Porczyk
bbe757d0a7 Make pylint very ♥ 2017-05-30 15:06:05 +02:00
Wojtek Porczyk
42cbd9ff68 qubes.tests asyncio, part 2
QubesOS/qubes-issues#2622
2017-05-29 19:08:38 +02:00
Wojtek Porczyk
df03800278 Fix miscellaneous warnings
Among them:
- explicitly close files (possibly using with: syntax)
- use non-deprecated methods
2017-05-29 17:34:15 +02:00
Wojtek Porczyk
b212a75ba3 qubes/vm/qubesvm: don't use .run_service() for .run()
QubesOS/qubes-issues#2622
2017-05-29 17:34:15 +02:00
Wojtek Porczyk
a9755ed76a qubes/ext/gui: remove is-fully-usable and is_guid_running 2017-05-29 17:34:15 +02:00
Wojtek Porczyk
b256af3bfb qubes.tests asyncio
QubesOS/qubes-issues#2622
2017-05-29 17:34:15 +02:00
Marek Marczykowski-Górecki
7f3dd8b3d7
storage: don't fail on import if lvm is not installed 2017-05-26 15:08:20 +02:00
Marek Marczykowski-Górecki
e54cc11a2c
vm: expose to VM only features with 'service/' prefix
And place them in /qubes-service/ QubesDB directory. This allows
extensions to easily store some data not exposed to VM, but also have
control what VM will see. And at the same time, it make it compatible
with existing services framework

QubesOS/qubes-issues#1637
2017-05-26 15:08:19 +02:00
Marek Marczykowski-Górecki
607dcbaf37
admin-api: fix handling admin.vm.property.Set with None VM value
Setting VMProperty to None VM should be encoded as '' value (according
to VMProperty._none_value). But value validation rejected this value.

QubesOS/qubes-issues#2622
2017-05-26 15:08:18 +02:00
Marek Marczykowski-Górecki
305fb5b6d0
Move qvm-template-postprocess tool to core-admin-client
Since implementing admin.vm.volume.Import, this migrated to Admin API.
2017-05-26 15:08:17 +02:00
Marek Marczykowski-Górecki
d42197e94c
make 'mgmt-permission:' event compatible with check-events script
It doesn't support str.format().
2017-05-26 15:08:17 +02:00
Marek Marczykowski-Górecki
3177fc504f
tests: basic tests for admin.vm.volume.Import 2017-05-26 15:08:16 +02:00
Marek Marczykowski-Górecki
d7af67d875
storage: minor error handling fix 2017-05-26 15:08:15 +02:00
Marek Marczykowski-Górecki
6c29c653a0
tests: improve qubes_basdir_patching
File storage pool use qubes.config.qubes_base_dir, not
qubes.config.system_path, so patch that too.
2017-05-26 15:08:15 +02:00
Marek Marczykowski-Górecki
3cacf290bb
admin: implement admin.vm.volume.Import
Implement this in two parts:
1. Permissions checks, getting a path from appropriate storage pool
2. Actual data import

The first part is done by qubesd in a standard way, but then, instead of
accepting all the data (which may be several GB), return a path to which
a shell script (in practice: `dd` command) will write the data.
Then the script call back to qubesd again to report success/failure and
qubesd response from that call is actually returned to the user.

This way we do not pass all the data through qubesd, but still can
control the process from there in a meaningful way. Note that the last
part (second call to qubesd) may perform all kind of verification (like
a signature check on the data, or so) and can also prevent VM from
starting (hooking also domain-pre-start event) from not verified image.

QubesOS/qubes-issues#2622
2017-05-26 15:08:14 +02:00
Marek Marczykowski-Górecki
46b60dbf42
storage: add Pool.import_data to the API
Allow importing not only from another volume, but also raw data. In
practice, for all currently implemented storage pools, this is the same
as Pool.export, because path returned there is read-write. But lets not
abuse this fact, some future implementation may need different methods.

QubesOS/qubes-issues#2622
QubesOS/qubes-issues#2256
2017-05-23 15:35:55 +02:00
Marek Marczykowski-Górecki
1692601fcd
typo fix 2017-05-23 15:35:21 +02:00
Marek Marczykowski-Górecki
74689dd998
qubesd: reduce verbosity
Remove debug prints, log full traceback (of handled exception) only when
debug mode enabled (--debug, introduce in this commit too).
--debug option also enables sending tracebacks to the API clients.

QubesOS/qubes-issues#853
2017-05-23 04:47:50 +02:00
Marek Marczykowski-Górecki
64b83fa95a
admin: raise QubesNoSuchPropertyError for non-existing properties
Accessing non-existing property is a common action (for example
hasattr() do try to access the property). So, introduce specific
exception, inheriting from AttributeError. It will behave very similar
to standard (non-Admin-API) property access.

This exception is reported to the Admin API user, so it will be possible
to distinguish between non-existing property and access denied. But it
isn't any significant information leak, as list of valid properties is
publicly available in the source code.

QubesOS/qubes-issues#853
2017-05-23 04:32:59 +02:00
Marek Marczykowski-Górecki
f93583e2be
devices: adjust XML serialization of device options
Use '<option name="option_name">option_value</option>' instead of
'<options option_name="option_value"/>'. It's more consistent with the
rest of qubes.xml - have one thing per element.

Also, add options deserialization test.
2017-05-22 17:11:20 +02:00
Marek Marczykowski-Górecki
29f3c9b58f
vm: don't try to define libvirt domain when just checking its state
When libvirt domain is not defined, it isn't running for sure.
This commit fixes the case when vm.is_running() appears anywhere in the
code used during libvirt xml building. In this case, it's mostly about
PCI device description for libvirt.
2017-05-22 17:11:19 +02:00
Marek Marczykowski-Górecki
6bc44b43de
pci: adjust PCIDeviceExtension device identifier syntax
':' is not allowed in device identifier, replace it with '_'.

Warning: this breaks existing qubes.xml
2017-05-22 17:11:19 +02:00
Marek Marczykowski-Górecki
fe37e0933e
Devices-related formating fixes
QubesOS/qubes-issues#
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
dde9085920
vm: fix autostart-related events handling
event signature contains 'name', not 'prop' (it's no longer positional
argument).
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
402afa1925
pci: use device attach options for disabling strict reset
Since we have now per-device options, it's more logical to use it here,
instead of features with device identifier encoded into feature name.
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
d0b8933374
devices: add assignment.device property
Make it easy to retrieve DeviceInfo object out of DeviceAssignment
object. The only missing piece of information for that is device class,
so add it. Make it optional, as it can be filled on demand when passing
the object through DeviceCollection (either by listing devices, or
attaching/detaching).

This is mostly to ease handling options in libvirt template - to get
them, you need to use `assignments()`, istead of `persistent()` or
`attached()`, but there were no _simple_ way of getting actual device
object.

This also makes DeviceCollection._device method not needed anymore.
2017-05-22 03:21:07 +02:00
Marek Marczykowski-Górecki
227010d433
pci: fix device-pre-attach event signature
Now it contains also options. PCI extension do not use it, yet.
2017-05-19 18:49:22 +02:00
Marek Marczykowski-Górecki
f9fa27dacc
property: add property.__str__ 2017-05-19 18:49:21 +02:00
Marek Marczykowski-Górecki
f7cb8f27c5
admin: add admin.vm.Remove method
QubesOS/qubes-issues#2622
2017-05-19 18:49:21 +02:00
Marek Marczykowski-Górecki
0ef010cc70
qmemman: fix list/filter handling on python3
Filter in python3 returns a generator, can be iterated only once.
This is about list of existing domains - store it as a list, otherwise
domains will "disappear" after being discovered.
2017-05-18 09:44:48 +02:00
Marek Marczykowski-Górecki
1831088c9b
api/internal: don't try to start dom0
qrexec-policy do try to start a VM before executing service. Avoid this
here, instead of qrexec-policy to keep the later clean.
2017-05-17 14:24:59 +02:00
Marek Marczykowski-Górecki
ae28d32af7
vm: make sure qmemman connection is closed even on failed VM startup
Qmemman hold global lock while serving requests, so not closing
connection will prevent any further action.
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
e5daf902b3
vm/net: fix IP address calculation
7 is not the same as 7 bits...
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
4d6d6c913a
qubesd: make sure NOTIFY_SOCKET variable is not propagated
qubesd do start other daemons - make sure they will not try to signal
systemd about it. In some cases such daemons (qubesdb-daemon) behave
differently based on this variable.
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
dc2563271c
tests: very basic tests for LinuxKernel pool 2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
0b64e3fbe0
storage: make LinuxKernel pool track vm kernel version
Do not initialize it only at qubes.xml load time, but re-read vm.kernel
property each time the path is constructed. While at it, add support for
vm.kernel set to 'None' - simply don't include modules.img (xvdd) then.
2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
595d983659
storage: make verify() asyncio aware 2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
d57aef96e5
admin.vm.Start: report libvirt errors to caller
Convert them to QubesException for that.
2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
774f4c8d49
tests: admin.vm.devices.* Admin API
QubesOS/qubes-issues#2622
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
661752bbc1
devices: list persistent devices also when not currently attached
When device extension do not return some "persistent" device as
currently attached, still return it, as it will be attached at next
domain startup. User can distinguish such devices by having
frontend_domain=None (or other VM).
Also, return a set from DeviceCollection.assignments().
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
2c0d79af4a
devices: pass attach options to device handling extension
Otherwise options would be ignored...
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
75ac611c90
admin: admin.vm.device.* methods
QubesOS/qubes-issues#2622
2017-05-16 09:05:13 +02:00
Marek Marczykowski-Górecki
f3c7fb8129
admin: add missing parts to admin.vm.Clone
Clone all the things.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
8533ec9773
devices: add clone method
Copy DeviceAssignment object
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
b4ad5f0149
firewall: add clone and comparing methods, missing docstrings 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
8de82f390e
tests: Tags() and Features() 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
680dbf432c
vm: Features fixes
clear: dict should not be mutated during interation.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
ba86d6da79
vm: implement tag manager to fire events on change
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).

Fixes QubesOS/qubes-issues#2686
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
68a426f0ba
admin: add methods for global properties
QubesOS/qubes-issues#2622
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
0160d6e2f0
admin: fix handling default template in admin.vm.Create.* methods 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
fcfb2e004a
tests: more for admin.Events method 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
3c7d2289ce
vm: stubdom_xid: use canonical method for checking VM state
Don't depend on xid==-1 for not running VM.
This breaks some tests, as xid can be a mock object (not comparable to
int).
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
e598735c3b
tests: minor formatting 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
d4353c2a30
Enable linux-stubdom by default
Also, make it possible to set default on a template for its VMs.

QubesOS/qubes-issues#2185
2017-05-16 09:01:40 +02:00
HW42
f7094bec4f
add support for linux stubdoms
(cherry picked from commit e14ac1500a05162b3c07a5f44b4f0223f43fe90c)
2017-05-15 14:43:30 +02:00
Marek Marczykowski-Górecki
625c03c504
Rename MgmtAPI to AdminAPI - part 3: Admin API
QubesOS/qubes-issues#853
2017-05-12 19:28:08 +02:00
Marek Marczykowski-Górecki
258d268a3a
Rename MgmtAPI to AdminAPI - part 2: internal API
QubesOS/qubes-issues#853
2017-05-12 19:28:08 +02:00
Marek Marczykowski-Górecki
cd489f46e1
Rename MgmtAPI to AdminAPI - part 1: classes
QubesOS/qubes-issues#853
2017-05-12 19:28:07 +02:00
Marek Marczykowski-Górecki
81162a699b
Remove unused helper scripts 2017-05-12 18:43:35 +02:00
Marek Marczykowski-Górecki
83eef56f9d
Remove tools that are moved to -client repository
QubesOS/qubes-issues#853
2017-05-12 18:43:35 +02:00
Marek Marczykowski-Górecki
bb4dc91ee8
storage/lvm: add workaround for ancient lvm2 in Travis-CI environment 2017-05-12 18:16:30 +02:00
Marek Marczykowski-Górecki
2a4af124f8
qubes/app: don't fail on saving qubes.xml when 'qubes' group isn't there 2017-05-12 18:11:27 +02:00
Marek Marczykowski-Górecki
da3eed582b
tests: rearrage LVM thin tests to run at least some in devel env
This requires creating LVM volume group, so create on based on loop dev
in /tmp.
This is rather rough, but if any of this fails, run the tests anyway -
it will simply skip LVM tests.
2017-05-12 18:10:17 +02:00
Marek Marczykowski-Górecki
b91b89a341
Make pylint ♥ 2017-05-12 17:41:38 +02:00
Marek Marczykowski-Górecki
0d84ee1c24
make pylint happy 2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
7ef020d5fe
devices: fix 'NotImplemented' usage
'__eq__' function should return 'NotImplemented' _value_, not
'NotImplementedError' exception.
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
4f0d72f99a
devices: adjust API documentation
Device ident may contain only characters allowed in qrexec argument.
This will allow using it directly in qrexec argument in Attach/Detach
methods.
This also means PCI extension will need to be updated (it uses ':' in
ident).

QubesOS/qubes-issues#853
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
ef031d6f0d
qubesvm: make vm.create_on_disk cleanup after failed attempt
Don't leave half-created VM - it will make harder to retry.
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
2a25db7eb7
qubesvm: fix stdio handling in vm.run_service_for_stdio
Do set stdin/stdout/stderr descriptors as PIPE. Otherwise would not be
redirected.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
838553baf8
qubesvm: move storage.stop() to 'domain-shutdown' event handler
QubesOS/qubes-issues#2256
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
0dfcaa63c9
Handle libvirt events in qubesd
This is required to get shutdown notification, when it wasn't initiated
by qubesd (for example 'poweroff' command inside of VM).
Libvirt event loop implementation must be registered before making
connection to libvirt, so move it to the beginning of main().
For now, only 'domain-shutdown' event is emited.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
3e067a3ef5
vm: run_service(): require user session running only when need gui access
User session may not be started at all (for example no qubes packages
installed there), so don't block it in all the cases. Also this would
prevent running 'qubes.WaitForSession' service...

In practice, default value for 'gui' argument is False, so in most cases
user session will be ignored. Which doesn't matter in most cases -
especially for services called by qubesd.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
68d5ca93e1
storage: have QubesVM() handle vm.dir_path, also on clone and rename
Keep it uniform - QubesVM() object is responsible for handling
vm.dir_path, Storage() is responsible for handling disk volumes (which
may live in that directory

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
53be79b3b7
mgmt: vm.Create* and vm.Clone
Create methods are dynamically created based on available VM classes.

QubesOS/qubes-issues#2622
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
52c3753d61
storage: support asynchronous storage pool implementations
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.

This commit does not touch any pool implementation - all of them are
still synchronous.

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
f324d30d4e
quebs: make Label() and storage.Pool() objects hashable 2017-05-12 14:27:47 +02:00
Marek Marczykowski-Górecki
ca51e8a02a
vm: remove unused metaclass 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
8c3c65f40f
vm/net: move 'netvm' value validation to setter 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
3bc8facc0b
mgmt: rework mgmt.Events to use per-instance handlers
QubesOS/qubes-issues#2622
2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
da7496794a
events: add support for per-instance handlers 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
bd1f84fcec
events: clarify docstrings about add_handler/remove_handler
https://github.com/QubesOS/qubes-core-admin/pull/100#discussion_r115745098
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
cddc507788
mgmt: factor out filtering by mgmt-permission:... events
Split actual filtering done by mgmt-permission: events into calling an
event and applying returned filters. This way filtering done in
mgmt.Events handler could reuse the same function.
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
6926725e02
Fixes for devices API
See comments in #98
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
68c3d7309a
tools: remove qubes_monitor_layout_notify tool - moved to mgmt-client 2017-05-12 14:25:30 +02:00
Marek Marczykowski-Górecki
82983cc1d8
vm: minor fix in docstring 2017-05-12 14:25:30 +02:00
Marek Marczykowski-Górecki
033d2132d3
ext/gui: remove most of it, moved to qvm-start-gui tool...
...in core-mgmt-client repository. qubesd isn't the right place to start
GUI applications, which will be even more important when GUI domain will
be something different than Dom0.

QubesOS/qubes-issues#833
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
027df9567c
mgmt: add mgmt.label.Index method
QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
278243d7dc
tools/qubesd: fix sending exception response
QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
55037e0446
tests: tools.qubesd
For now only QubesDaemonProtocol test. main function still untested

QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
0b737b3a55
mgmt: vm.features handling (mgmt.vm.feature.*)
Actual implementation and tests.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
361ecdd6ac
vm: add stubdom_xid property
Some parts (like GUI) needs to communicate with stubdomain. Lets expose
it's XID at the API level.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
acaada3fd9
vm: fix placeholder methods signatures in Features class
Some methods inherited from dict (pop and setdefault here) are covered
by placeholders raising NotImplementedError. Lets fix their signatures
(to match those of dict) to really get NotImplementedError, instead of
TypeError.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
643cef1e65
mgmt: send artificial 'connection-established' event just after connection
This allows avoid race condition between  registering event handlers and
performing some action. The important thing is the event sent after
registering event handlers in qubesd. This means state changes (like
VM start/stop) after 'connection-established' event will be included in
event stream.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
8cc0af1eda
tests: mgmt.Events
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
73df1b8ea7
tests: VM lifecycle methods tests
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
99be78646f
Add stateless_property decorator for properties exposed through mgmt API
Management API gives access only to qubes.property. And this is
actually a good thing, so instead of extending it to access also
builtins.property, add a simple decorator to define read-only, stateless
qubes.property.

QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
2aa0de3d5b
tools: remove qvm-ls tool and related integration in qubes.property
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.

The tool itself is already moved to core-mgmt-client repository.

QubesOS/qubes-issues#853
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
654e64c4a6
mgmt: implement mgmt.Events
Pass (almost) all the events. Some better filtering probably will be
needed.

QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
7e4bc4249d
tools/qubesd-query: handle events-based methods
Pass the data as it arrives, instead of buffering it all.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
40a86efd66
mgmt: add support for events-sending methods
Standard methods return only one value, after operation is completed,
but events-related methods may return multiple values during the method
execution time. Provide a callback for such cases.
Also, according to specification, avoid sending both event and non-event
values.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
bd9f1d2e7c
mgmt: add support for cancellable operations
Allow method handler to decide if operation could be cancelled. If yes,
when connection to the qubesd is terminated (and
protocol.connection_lost get called) the operation is cancelled using
standard asyncio method - in which case asyncio.CancelledError is thrown
inside method handler. This needs to be explicitly enabled, because
cancellable methods are much harder to write, to maintain consistent
system state.
Caveat: protocol.connection_lost is called only when trying to send some
data to it (and it's already terminated). Which makes this whole
mechanism useful only for events. Otherwise, when sending some data (and
possibly detecting that connection is broken), the operation is already
completed.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
ee967c1e72
events: add function to remove event handlers
Event handlers for remote mgmt API will be registered dynamically, so
allow to unregister them.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
68c8d219c9
vm/qubesvm: fix daemon startup function
'command' argument is a tuple
2017-05-12 14:25:25 +02:00
Marek Marczykowski-Górecki
6383d87be9
qmemmand: fix systemd notification 2017-05-12 14:25:25 +02:00
Marek Marczykowski-Górecki
d75fc6c28b
backup: add a little more usage info, include docs in main index 2017-04-21 19:58:32 +02:00
Marek Marczykowski-Górecki
50b812190b
Merge remote-tracking branch 'kalkin/device-assignments' into core3-devel
* kalkin/device-assignments: (21 commits)
  PCI extension cache PCIDevice objects
  Make pylint ♥
  Fix pylint warning no-else-return
  Fix pylint warning len-as-conditional
  device-list-attached event returns a dev/options tupples list
  DeviceAssignment options are now a dict
  Remove WrongAssignment exception
  Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
  Update relaxng devices option element
  Fix tests broken by the new assignment api
  Fix qubes.tests.devices
  Fix pci device integration tests
  qvm-device add support for assignments
  Update ext/pci to new api
  BaseVM add DeviceAssignment xml serialization
  Migrate DeviceCollection to new API
  Add PersistentCollection helper to qubes.devices
  Add DeviceAssignment
  qvm-device validates device parameters
  qvm-device fix handling of non block devices
  ...
2017-04-21 17:09:33 +02:00
Marek Marczykowski-Górecki
2705e223bd
Merge branch 'core3-policy' into core3-devel
* core3-policy:
  Make pylint happy
  tests: disable GTK tests on travis
  qubespolicy: make pylint happy
  qubespolicy: run GUI code inside user session and expose it as dbus object
  tests: plug rpc-window tests into main test runner
  qubespolicy: plug GUI code into qrexec-policy tool
  rpm: add rpc-window related files to package
  rpc-window: adjust for qubespolicy API
  rpc-window: use pkg_resources for glade file
  rpc-window: use 'edit-find' icon if no other is found
  rpc-window: adjust for python3
  rpc-window: code style adjustments
  Import new rpc confirmation window code
  qubesd: add second socket for in-dom0 internal calls
  policy: qrexec-policy cli tool
  tests: qubespolicy tests
  qubespolicy: initial version for core3
  vm/appvm: add dispvm_allowed property
  dispvm: don't load separate Qubes() instance when handling DispVM
2017-04-21 17:01:56 +02:00
Marek Marczykowski-Górecki
fa72679b47
Make pylint happy
New pylint throw some more warnings.
2017-04-21 16:57:35 +02:00
Bahtiar `kalkin-` Gadimov
8d60f533c3
PCI extension cache PCIDevice objects 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
79407a8717
Make pylint ♥ 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
5bc7a8f9e3
Fix pylint warning no-else-return 2017-04-15 23:49:41 +02:00
Bahtiar `kalkin-` Gadimov
eb217e0d27
Fix pylint warning len-as-conditional 2017-04-15 23:49:41 +02:00
Bahtiar `kalkin-` Gadimov
9da28c9c15
device-list-attached event returns a dev/options tupples list 2017-04-15 23:49:40 +02:00
Bahtiar `kalkin-` Gadimov
1c9636c5af
DeviceAssignment options are now a dict 2017-04-15 23:49:40 +02:00
Bahtiar `kalkin-` Gadimov
5a8cc9bdd3
Remove WrongAssignment exception
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:39 +02:00
Bahtiar `kalkin-` Gadimov
e446e7a2f4
Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:36 +02:00
Bahtiar `kalkin-` Gadimov
b1b005964f
Fix tests broken by the new assignment api 2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
827ca283f3
Fix qubes.tests.devices
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
e84114d3a9
Fix pci device integration tests
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
0f78d257d4
qvm-device add support for assignments
- Add header field ASSIGNED
- Persistent assignments are displayed with an asterisk
- Add '-p' to attach command

Fix qvm-device

Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:30 +02:00
Bahtiar `kalkin-` Gadimov
0b3aebac9f
Update ext/pci to new api
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:29 +02:00
Bahtiar `kalkin-` Gadimov
2a6266887e
BaseVM add DeviceAssignment xml serialization
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
990cfd8ab9
Migrate DeviceCollection to new API
- Use PersistentCollection as _set()
- attach/detach expect DeviceAssignment as parater
- attached(persistent=True) is now persistent()
- attached() returns all attached devices
- assigned() returns all attached device assignments

`#	modified:   templates/libvirt/xen.xml

Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
23c68c5458
Add PersistentCollection helper to qubes.devices 2017-04-15 18:05:27 +02:00
Bahtiar `kalkin-` Gadimov
211e018268
Add DeviceAssignment 2017-04-15 18:05:26 +02:00
Bahtiar `kalkin-` Gadimov
e52898972d
qvm-device validates device parameters 2017-04-15 18:05:25 +02:00
Bahtiar `kalkin-` Gadimov
6caaa6f66d
qvm-device fix handling of non block devices 2017-04-15 18:05:24 +02:00
Bahtiar `kalkin-` Gadimov
9d08e4b792
Fix bug in PCIDeviceExtension: decode buffer to string 2017-04-15 18:05:12 +02:00
Marek Marczykowski-Górecki
96db9a46d1 Fix start-related asyncio integration
cherry-picked commit 05fc3a4
2017-04-07 17:46:54 +02:00
Marek Marczykowski-Górecki
0ada6d0b64
tests: disable GTK tests on travis
The environment there is way too old.

QubesOS/qubes-issues#910
2017-04-07 17:07:31 +02:00
Marek Marczykowski-Górecki
e76ede3ad0
tests: plug rpc-window tests into main test runner
QubesOS/qubes-issues#910
2017-04-07 17:07:30 +02:00
Marek Marczykowski-Górecki
83526a28d3
qubesd: add second socket for in-dom0 internal calls
This socket (and commands) are not exposed to untrusted input, so no
need to extensive sanitization. Also, there is no need to provide a
stable API here, as those methods are used internally only.

QubesOS/qubes-issues#853
2017-04-07 17:07:28 +02:00
Marek Marczykowski-Górecki
564408eb3f
tests: qubespolicy tests
Fixes QubesOS/qubes-issues#2460
2017-04-06 15:43:17 +02:00
Marek Marczykowski-Górecki
1c836531c7
vm/appvm: add dispvm_allowed property
Speciffy whether DispVM can be created from this AppVM

Fixes QubesOS/qubes-issues#2075
2017-04-06 15:43:01 +02:00
Marek Marczykowski-Górecki
e4e92b06af
dispvm: don't load separate Qubes() instance when handling DispVM
This will not work in qubesd, where there is one and only Qubes()
instance.
2017-04-06 15:43:01 +02:00
Wojtek Porczyk
3bc4ec1e98 add mock libvirtaio module + pylint fixes 2017-04-05 17:06:45 +02:00
Wojtek Porczyk
64d358562b qubes: revert async def, use @asyncio.coroutine
Current pylint (any released version) stumbles on async def'ined
functions. Let's use @asyncio.coroutines for now.

Seems like python-3.5 is not that mature yet.

QubesOS/qubes-issues#2622
QubesOS/qubes-issues#2738
PyCQA/pylint#1126
2017-04-05 17:05:52 +02:00
Wojtek Porczyk
78693c265c qubes/vm/qubesvm: rework QubesVM.run*() methods
0) All those methods are now awaitable rather than synchronous.

1) The base method is run_service(). The method run() was rewritten
   using run_service('qubes.VMShell', input=...). There is no provision
   for running plain commands.

2) Get rid of passio*= arguments. If you'd like to get another return
   value, use another method. It's as simple as that.
   See:
      - run_service_for_stdio()
      - run_for_stdio()

   Also gone are wait= and localcmd= arguments. They are of no use
   inside qubesd.

3) The qvm-run tool and tests are left behind for now and will be fixed
   later. This is because they also need event loop, which is not
   implemented yet.

fixes QubesOS/qubes-issues#1900
QubesOS/qubes-issues#2622
2017-04-05 17:05:22 +02:00
Wojtek Porczyk
a520e8d768 qubes/mgmt: add missing events for permission for lifecycle
Thanks @marmarek for spotting this.

QubesOS/qubes-issues#2622
2017-04-05 16:36:54 +02:00
Wojtek Porczyk
c437f40284 qubes/mgmt: lifecycle and class listing
- mgmt.vmclass.List
- mgmt.vm.Start
- mgmt.vm.Shutdown
- mgmt.vm.Pause
- mgmt.vm.Unpause
- mgmt.vm.Kill

QubesOS/qubes-issues#2622
2017-04-03 21:55:01 +02:00
Wojtek Porczyk
c4ef02c377 qubes/mgmt: explicit method decorator and misc improvements
- Get rid of @not_in_api, exchange for explicit @api() decorator.
- Old @no_payload decorator becomes an argument (keyword-only).
- Factor out AbstractQubesMgmt class to be a base class for other mgmt
  backends.
- Use async def instead of @asyncio.coroutine.

QubesOS/qubes-issues#2622
2017-04-03 21:55:01 +02:00
Wojtek Porczyk
1b9479837a Merge remote-tracking branch 'marmarek/core3-mgmt-methods1' into core3-devel
Conflicts:
	qubes/mgmt.py
2017-04-01 18:33:54 +02:00
Marek Marczykowski-Górecki
3388054eae
mgmt: move property value sanitization to property definition
This also means we don't check if a VM with given name (in case of
VMProperty) exists in the system, at this stage. But this is ok, lets
not duplicate work of property setter.

QubesOS/qubes-issues#2622
2017-03-31 11:55:24 +02:00
Wojtek Porczyk
cce809c2cb qubes/vm/qubesvm: async def start
QubesOS/qubes-issues#2622
2017-03-30 23:04:55 +02:00
Marek Marczykowski-Górecki
ce3bedbf2c
vm: move validate_name to qubes/vm
This will be needed by VMProperty class in the next commit.

QubesOS/qubes-issues#2622
2017-03-28 21:04:05 +02:00
Marek Marczykowski-Górecki
ad456a3387
mgmt: minor adjustments
per @woju review

QubesOS/qubes-issues#2622
2017-03-28 20:59:55 +02:00
Wojtek Porczyk
8e3621c4e5 Move libviraio to core-libvirt, may be upstreamed
QubesOS/qubes-issues#2622
2017-03-22 15:30:55 +01:00
Marek Marczykowski-Górecki
2c115bb648
mgmt: allow functions to return None if no data is to be returned
Don't force `return ''` everywhere.

QubesOS/qubes-issues#2622
2017-03-17 12:49:28 +01:00
Marek Marczykowski-Górecki
f93674de1a
mgmt: drop ProtocolRepr
Since we've added type= argument to property.Get format, it isn't
useful anymore.

QubesOS/qubes-issues#2622
2017-03-16 21:10:06 +01:00
Marek Marczykowski-Górecki
d21f54887d
mgmt: add decorator for method calls not accepting payload
Fixes QubesOS/qubes-issues#2687
2017-03-16 20:21:11 +01:00
Marek Marczykowski-Górecki
8371ffcd0f
tests: storage and labels
QubesOS/qubes-issues#2622
2017-03-16 20:04:07 +01:00
Marek Marczykowski-Górecki
2a223811a5
tests: add missing import 2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
f1c8f3220d
tests: deal with events containing dict in kwargs
If kwargs contains dict as one of values, it isn't hashable and can't be
used as value in frozenset/tuple. Convert such values into
frozenset(dict.items()). Only one (more) level is supported, but it
should be enough.

Solution from http://stackoverflow.com/a/13264725
2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
beaa0e9348
tests/mgmt: check if argument/payload is rejected when should be
Instead of creating such tests for each method separately, use unittest
subTest functionality to handle all of them at once.
2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
944bb26369
tests/mgmt: VM property related functions 2017-03-16 20:04:05 +01:00
Marek Marczykowski-Górecki
35d1167893
qubes/vm/net: fix name of argument for property-del event handler
Since enforcing keyword arguments for event handlers, it's important now.
2017-03-16 20:04:05 +01:00
Marek Marczykowski-Górecki
32f6bc2cd9
qubes/app: fix notifying about default_netvm change
Notify every VM that is affected, not only those providing network
itself.
2017-03-16 20:04:05 +01:00
Marek Marczykowski-Górecki
2c4303efc4
Prefer qubes.exc.QubesValueError over ValueError
This provide clearer information for UI.
2017-03-16 20:04:05 +01:00
Marek Marczykowski-Górecki
010d40dc1e
mgmt: add label-related calls
QubesOS/qubes-issues#2622
2017-03-16 20:04:04 +01:00
Marek Marczykowski-Górecki
33f3fedca1
mgmt: save qubes.xml after config-modifying calls
In theory any call could modify config (through events), but lets keep
writes to qubes.xml low. In any case, qubes.xml will be eventually
written (either at next config-modifying call, or daemon exit).
2017-03-16 20:04:04 +01:00
Marek Marczykowski-Górecki
868dbeac3e
mgmt: implement mgmt.vm.property.Set
Sanitization of input value is tricky here, and also very important at
the same time. If property define value type (and it's something more
specific than 'str'), use that. Otherwise allow only printable ASCII
characters, and let appropriate event and setter handle value.
At this point I've reviewed all QubesVM properties in this category and
added appropriate setters where needed.

QubesOS/qubes-issues#2622
2017-03-16 20:04:03 +01:00
Marek Marczykowski-Górecki
da51e6f032
vm/qubesvm: add validator for default_user property
Don't allow characters potentially interfering with qrexec. To be on the
safe side, allow only alphanumeric characters + very few selected
punctuations.
2017-03-16 20:04:03 +01:00
Marek Marczykowski-Górecki
0f8fab088e
vm/qubesvm: remove pool_name property
It isn't used anywhere - in core3 each storage volume have pool assigned
- which may be different for each volume.
2017-03-16 20:04:03 +01:00
Marek Marczykowski-Górecki
2d2672ec58
vm/qubesvm: convert firewall_conf into dumb, read-only property
Don't allow anything else than firewall.xml.
2017-03-16 20:04:03 +01:00
Marek Marczykowski-Górecki
123feced36
vm/qubesvm: forbid '/' in kernel property
It would give VM access to some files outside of
/var/lib/qubes/vm-kernels.
2017-03-16 20:04:02 +01:00
Marek Marczykowski-Górecki
a036e2a8a0
vm/qubesvm: improve name property setter
Split it into two functions: validate_name - context-less verification,
and actual _setter_name which perform additional verification in
context of actual VM.
Switch to qubes.exc.* exceptions where appropriate.
2017-03-16 20:04:02 +01:00
Marek Marczykowski-Górecki
dbf2066dfd
mgmt: encode property type in property.Get
This also require having property.type public.

QubesOS/qubes-issues#2622
2017-03-16 20:04:02 +01:00
Marek Marczykowski-Górecki
3b36e92b6d
vm/qubesvm: fix few more keyword arguments for events 2017-03-16 20:04:02 +01:00
Marek Marczykowski-Górecki
772293d0b5
vm/qubesvm: define 'updateable' as qubes.property
This will allow property being accessed through management API
2017-03-16 20:04:01 +01:00
Marek Marczykowski-Górecki
f7eabf8eb0
tools/qubesd: do not close connection before sending response
eof_received callback should return True, if connection should not be
automatically closed just after returning from it.
2017-03-16 20:04:01 +01:00
Marek Marczykowski-Górecki
c41585e2f5
Initialize dom0 label
It is required property. Additionally, define icon_path to None,
otherwise it tries to access dom0.dir_path, which isn't set.
2017-03-16 20:04:01 +01:00
Marek Marczykowski-Górecki
10a07c8726
mgmt: allow vm.List call to a particular VM
This allow getting info about a single VM.

QubesOS/qubes-issues#853
2017-03-16 20:04:01 +01:00
Marek Marczykowski-Górecki
fb7bd6823a
mgmt: implement storage-related methods
QubesOS/qubes-issues#2622
2017-03-16 20:04:01 +01:00
Wojtek Porczyk
80807fb872 qubes/libvirtaio: document and prepare for upstream
QubesOS/qubes-issues#2622
2017-03-16 15:21:16 +01:00
Wojtek Porczyk
a5c59a5075 qubes/mgmt: Drop custom repr
QubesOS/qubes-issues#2622
2017-03-13 22:00:15 +01:00
Wojtek Porczyk
93153da893 Add documentation for mgmt
QubesOS/qubes-issues#2622
2017-03-13 21:51:52 +01:00
Wojtek Porczyk
417cb6e912 qubes/vm/mix/net: fix event arguments
Arguments to events are now keyword-only and require exact names.
2017-03-06 17:20:57 +01:00
Wojtek Porczyk
4a247b1b1b Merge remote-tracking branch 'origin/pull/90/head' into core3-devel 2017-03-02 13:19:57 +01:00
Marek Marczykowski-Górecki
f7d73893d7
qubes/storage: py3k related fixes 2017-03-01 21:50:06 +01:00
Wojtek Porczyk
c2a0d34ade pylint: do not interpret asyncio.ensure_future compat hack 2017-03-01 18:30:49 +01:00
Wojtek Porczyk
865ab10a0c qubesd+mgmt: convert mgmt functions to coroutines
QubesOS/qubes-issues#2622
2017-03-01 17:31:37 +01:00
Marek Marczykowski-Górecki
3e0f225938
qubes: allow 'property' object comparing with str
This will allow checking if a given name is valid property name, using
simple `name in vm.property_list()`.

QubesOS/qubes-issues#853
2017-02-27 21:57:56 +01:00
Marek Marczykowski-Górecki
2169075807
qubesd: fix response message header
Type is not 16 bit big-endian. Encode it as 8bit code and \x00 as
delimiter explicitly.

QubesOS/qubes-issues#853
2017-02-27 21:43:14 +01:00
Marek Marczykowski-Górecki
6ab7032b11
qubes/mgmt: encode VM name without quotes
That's how it is in the specification.

QubesOS/qubes-issues#853
2017-02-27 21:42:30 +01:00
Marek Marczykowski-Górecki
f4616fc366
qubesd: make qubesd socket qubes-group owned
QubesOS/qubes-issues#853
2017-02-27 21:42:06 +01:00
Marek Marczykowski-Górecki
c454973596
qubes/mgmt: use keyword arguments in events
QubesOS/qubes-issues#853
2017-02-27 20:56:16 +01:00
Marek Marczykowski-Górecki
751415434c
backup: make hmac verification more defensive
Check HMAC file size, read it as binary or with 'ascii' encoding only.
2017-02-27 02:37:52 +01:00
Marek Marczykowski-Górecki
a6c7da6061
tests: be even more defensive on cleaning up VMs
Don't fail even if qubes-test.xml do not load at all because of syntax
error - for example empty file.
2017-02-27 02:37:52 +01:00
Marek Marczykowski-Górecki
45709b510a
backup: minor fixes after bringing back scrypt support 2017-02-27 02:37:51 +01:00
Marek Marczykowski-Górecki
1363251438
Revert "Revert "backup: use 'scrypt' tool for backup encryption and integrity protection""
This reverts commit 0f1672dc63.

Bring it back. Lets not revert the whole feature just because required
package exists only in qubes-builder, not in some online repository.
Also, this revert didn't go as planned - there was a reference to a
'passphrase' local variable, but it wasn't assigned any value.

Cc: @woju
2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
13fc810363
tests: some more fixes for core3 API 2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
3ecc0a9bcb
tests: improve devices API unit test
Check fired events - inspired by qvm-device test.
2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
7f2ca33774
tests: fix importing template in non-default pool 2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
3726c7d9c3
python: decode xrandr output earlier, don't use regexp on bytes 2017-02-27 02:37:49 +01:00
Marek Marczykowski-Górecki
5e43d26abd
qubes: unify property ordering
We already have property ordering defined in property_list(), lets move
it to proper place: property.__lt__.
2017-02-27 02:37:49 +01:00
Marek Marczykowski-Górecki
9ace4e66f1
tests: more py3k related fixes 2017-02-27 02:37:49 +01:00
Marek Marczykowski-Górecki
33416f2549
qmemman: update for py3k
This just make the code compatible with py3k, but nothing more.
Converting to asyncio is probably the next step.
2017-02-27 02:37:49 +01:00
Marek Marczykowski-Górecki
2c3e112951
backup: one more py3k related fix 2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
e52d8fb051
qubes: allow passing name of class to app.add_new_vm
This will allow more flexible API usage, especially when using mgmt API
- we need to use VM type as string there.
We don't lose any flexibility here - VM class names needs to be uniquely
identified by a string (used in qubes.xml) anyway.
2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
cae68f64ca
tests: just one more test in vm_qrexec_gui to core3 API 2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
3f29345d32
tests/storage: read-only volume should not have save_on_stop=True 2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
5ee05e06e5
qubes/core2migration: update locking API
Sync with 0141e1a "qubes/app: Allow keeping lock after load"
2017-02-27 02:37:47 +01:00
Marek Marczykowski-Górecki
570cbe5225
qubes: py3k related fixes 2017-02-27 02:37:45 +01:00
Marek Marczykowski-Górecki
9c5c70fe25
qubes/backup: py3k related fixes
- str/bytes usage
- functools.reduce
- dict.items instead of dict.iteritems etc
2017-02-27 02:35:50 +01:00
Marek Marczykowski-Górecki
f2a1687879
typos in comments 2017-02-27 02:35:49 +01:00
Marek Marczykowski-Górecki
9cad353939
tests: py3k related fixes - bytes/str
Adjust usage of bytes vs str type.
2017-02-27 02:35:49 +01:00
Marek Marczykowski-Górecki
d68499f17f
qubes: add property ordering 2017-02-27 02:35:49 +01:00
Wojtek Porczyk
be53db4db9 qubes/events: they accept only keyword arguments
Positional arguments are hereby deprecated, with immediate effect.

QubesOS/qubes-issues#2622
2017-02-21 14:46:42 +01:00
Wojtek Porczyk
48f10a79c9 qubes/tools/qubesd: add response header
QubesOS/qubes-issues#2622
2017-02-21 14:46:42 +01:00
Wojtek Porczyk
25d81b8ab6 Merge remote-tracking branch 'origin/pull/88/head' into core3-devel 2017-02-15 12:17:41 +01:00
Marek Marczykowski-Górecki
e50b17a6b3
tools/qvm-features: make pylint happy
reduce number of return statements.
2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
a317e81d7e
qubes/ext/gui: adjust shm.id path
It's moved to /var/run/qubes and now is built based on $DISPLAY.
2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
bd9300b38e
tests: copy pool configuration into qubes-test.xml
If template choosen for the tests is installed in non-default storage
pool, this pool also needs to be imported into qubes-test.xml.
2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
0c43329188
tools/qvm-features: fix handling empty list of features 2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
98edc9779c
tools/qvm-features: fix domain argument handling
It's args.domains[0], not args.vm.
2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
c3fc4062d8
tests: add basic test for qvm-features 2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
bcab92ee64
qubes/vm: make sure to close qmemman socket after failed startup
If qmemman socket isn't closed, it will block other VM startups.
2017-02-14 23:59:07 +01:00
Marek Marczykowski-Górecki
01aedb7f18
storage: fix handling snap_on_start=True file volumes
Use the right cow image and apply the second layer to provide read-write
access. The correct setup is:
 - base image + base cow -> read-only snapshot (base changes "cached"
   until committed)
 - read-only snapshot + VM cow -> read-write snapshot (changes discarded
   after VM shutdown)

This way, even VM without Qubes-specific startup scripts will can
benefit from Template VMs, while VMs with Qubes-specific startup scripts
may still see original root.img content (for possible signature
verification, when storage domain got implemented).

QubesOS/qubes-issues#2256
2017-02-14 23:59:07 +01:00
Marek Marczykowski-Górecki
48f78dfbc8
tests: check if snap_on_start=True volumes are not persistent
Content should be reset back to base volume at each VM startup.
Disposable VMs depend on this behaviour.

QubesOS/qubes-issues#2256
2017-02-14 23:59:07 +01:00
Wojtek Porczyk
8e678c0172 qubes/mgmt: mgmt.vm.property.Reset
QubesOS/qubes-issues#2622
2017-02-14 11:37:17 +01:00
Wojtek Porczyk
ada0437f52 qubes/mgmt: mgmt.vm.property.Help
QubesOS/qubes-issues#2622
2017-02-13 21:28:27 +01:00
Wojtek Porczyk
e8a5bc9b36 qubesd: improve exception handling
QubesOS/qubes-issues#2622
2017-02-10 23:25:45 +01:00
Wojtek Porczyk
249d8c08e2 qubes/tools/qubesd-query: low-level interrogation tool 2017-02-10 23:25:45 +01:00
Wojtek Porczyk
02639b8d02 qubes/mgmt: mgmt.vm.property.List
QubesOS/qubes-issues#2622
2017-02-10 22:55:53 +01:00
Wojtek Porczyk
c12fc744a2 qubes/mgmt: move mgmt api to separate module
QubesOS/qubes-issues#2622
2017-02-09 23:29:05 +01:00
Wojtek Porczyk
5d455ac3c4 misc pylint fixes related to qubesd 2017-02-08 15:37:39 +01:00
Wojtek Porczyk
0be3b1fbb1 qubes/tools/qubesd: initial version 2017-02-07 17:07:53 +01:00
Wojtek Porczyk
1be75d9c83 misc python3 fixes 2017-02-07 17:07:52 +01:00
Wojtek Porczyk
d74567d65f qubes: port core to python3
fixes QubesOS/qubes-issues#2074
2017-01-20 16:42:51 +01:00
Wojtek Porczyk
0f1672dc63 Revert "backup: use 'scrypt' tool for backup encryption and integrity protection"
This reverts commit 418d749680.

Package `scrypt` is currently not installable (not present in any repo).

Cc: @marmarek
2016-12-05 18:36:13 +01:00
Wojtek Porczyk
25912f5787 qubes/tools: add qvm-tags
QubesOS/qubes-issues#865
2016-12-02 14:17:09 +01:00
Wojtek Porczyk
5f436360f7 qubes/app: Fix create_empty_store signature
QubesOS/qubes-issues#1729
2016-11-30 18:34:11 +01:00
Wojtek Porczyk
68ad60c1b3 Merge remote-tracking branch 'origin/master' into core3-devel
Conflicts:
	core/qubes.py
	doc/Makefile
	doc/manpages/qvm-prefs.rst
	doc/qvm-tools/qvm-add-appvm.rst
	doc/qvm-tools/qvm-backup-restore.rst
	doc/qvm-tools/qvm-backup.rst
	doc/qvm-tools/qvm-block.rst
	doc/qvm-tools/qvm-clone.rst
	doc/qvm-tools/qvm-firewall.rst
	doc/qvm-tools/qvm-ls.rst
	doc/qvm-tools/qvm-pci.rst
	doc/qvm-tools/qvm-run.rst
	doc/qvm-tools/qvm-shutdown.rst
	doc/qvm-tools/qvm-start.rst
	doc/qvm-tools/qvm-template-commit.rst
	qvm-tools/qvm-ls
	qvm-tools/qvm-prefs
	qvm-tools/qvm-remove
	tests/__init__.py
	vm-config/xen-vm-template.xml
2016-11-30 03:07:39 +01:00
Marek Marczykowski-Górecki
8f443547fb
qubes/vm: disconnect from old QubesDB when going to start new instance
QubesDB daemon no longer remove socket created by new instance, so one
part of VM restart race condition is solved. The only remaining part is
to ensure that we really connect to the new instance, instead of talking
to the old one (soon to be terminated).

Fixes QubesOS/qubes-issues#1694
2016-11-26 04:09:00 +01:00
Marek Marczykowski-Górecki
3b209515c2
qubes/vm/dispvm: don't crash when DispVM is already killed
This is regression of QubesOS/qubes-issues#1660

Fixes QubesOS/qubes-issues#1660
2016-11-26 04:09:00 +01:00
Marek Marczykowski-Górecki
6ff836dfa4
qubes/log: add FD_CLOEXEC to log files
Don't leak log file descriptors. At least 'lvm' complains.

QubesOS/qubes-issues#2412
2016-11-26 04:08:59 +01:00
Marek Marczykowski-Górecki
210cb65d1c
qubes/tools: drop requirement of qubes-prefs --force-root
None of properties set there do any "dangerous thing" for filesystem
permissions (at least for now), so do not require it. This is mostly to
keep compatibility with %post rpm scripts (kernel-qubes-vm at least).

QubesOS/qubes-issues#2412
2016-11-26 04:08:59 +01:00
Marek Marczykowski-Górecki
a318d5cea9
Don't fail on DBus connection error or opening log
Especially in offline mode - like during installation, tests etc.

QubesOS/qubes-issues#2412
2016-11-26 04:08:59 +01:00
Marek Marczykowski-Górecki
02a0713665
qubes/tools: better handle qvm-template-postprocess called as root
This tool by design is called as root, so try to:
 - switch to normal user if possible
 - fix file permissions afterwards - if not

QubesOS/qubes-issues#2412
2016-11-26 04:08:58 +01:00
Marek Marczykowski-Górecki
28d4feb0d0
qubes: fix network-related functions
- cleanup_vifs crash when non-networked VM is running
- type error in get_vms_connected_to (store VM objects, not qid)
2016-11-26 04:08:58 +01:00
Marek Marczykowski-Górecki
e85b0663f6
tools: fire 'template-postinstall' event for extensions
Allow extensions to finish template setup. This include retrieving
appmenus, settings defaults etc.

QubesOS/qubes-issues#2412
2016-11-26 04:08:58 +01:00
Marek Marczykowski-Górecki
0041063b8e
qubes/config: set default labels
There was a comment '# Set later', but actually values were never set.
This break adding just installed template (qvm-template-postprocess).

QubesOS/qubes-issues#2412
2016-11-26 04:08:57 +01:00
Marek Marczykowski-Górecki
c4e85a81fb
qubes/app: automatically enable offline mode when running in chroot
Do not spray --offline-mode over every installer-related script.

QubesOS/qubes-issues#2412
2016-11-26 04:08:50 +01:00
Marek Marczykowski-Górecki
d8a1216daf
Fix qubes-create in offline mode
QubesOS/qubes-issues#2412
2016-11-26 04:08:09 +01:00
Marek Marczykowski-Górecki
5e15db4176
qubes/tools: accept qvm-start --no-guid
Lets keep compatibility with older scripts.

QubesOS/qubes-issues#2412
2016-11-26 04:08:08 +01:00
Marek Marczykowski-Górecki
964955758c
qubes/app: create 'default' storage pool as LVM when present
When system is installed with LVM thin pool, it should be used by
default. But lets keep file-based on for /var/lib/qubes for some corner
cases, migration etc.

QubesOS/qubes-issues#2412
2016-11-26 04:08:08 +01:00
Marek Marczykowski-Górecki
badc58837a
Add qvm-template-postprocess tool
This is intended to call to finish template installation/removal.
Template RPM package is basically container for root.img, nothing more.
Other parts needs to be generated after root.img extraction. Previously
it was open coded in rpm post-install script, but lets keep it as qvm
tool to ease supporting multiple version in template builder

QubesOS/qubes-issues#2412
2016-11-26 04:08:08 +01:00
Marek Marczykowski-Górecki
1418555346
qubes/vm: don't fail on removing VM without files
VM files may be already removed. Don't fail on this while removing a
VM, it's probably the reason why domain is being removed.

qvm-remove tool have its own guard for this, but it isn't enough - if
rmtree(dir_path) fails, storage.remove() would not be called, so
non-file storages would not be cleaned up.

This is also needed to correctly handle template reinstallation - where
VM directory is moved away to call create_on_disk again.

QubesOS/qubes-issues#2412
2016-11-26 04:08:07 +01:00
Marek Marczykowski-Górecki
cc440c62f6
qubes/tools: accept properties with '-'
'-' is invalid character in python identifier, so all the properties
have '_'. But in previous versions qvm-* tools accepted names with '-',
so lets not break this.

QubesOS/qubes-issues#2412
2016-11-26 04:08:07 +01:00
Marek Marczykowski-Górecki
880566a387
qubes/tools: do not reject --set/--get in *-prefs tools
Those options are no longer needed, but lets not reject them to preserve
compatibility with older scripts

QubesOS/qubes-issues#2412
2016-11-26 04:08:07 +01:00
Marek Marczykowski-Górecki
91727389c4
qubes/log: ensure logs are group writable
/var/log/qubes directory have setgid set, so all the files will be owned
by qubes group (that's ok), but there is no enforcement of creating it
group writable, which undermine group ownership (logs created by root
would not be writable by normal user)

QubesOS/qubes-issues#2412
2016-11-26 04:08:07 +01:00
Marek Marczykowski-Górecki
80c0093c77
qubes/tools/qubes-create: reject overriding existing qubes.xml
If someone really want do to this, need to manually remove the file
first.

QubesOS/qubes-issues#2412
2016-11-26 04:08:06 +01:00
Marek Marczykowski-Górecki
c08766e157
qubes/features: rename 'services/ntpd' to 'service/ntpd'
It makes much more sense to use singular form here - ntpd is a single
service.
2016-11-26 04:08:06 +01:00
Wojtek Porczyk
2b0ad51b18 Merge remote-tracking branch 'origin/pull/68/head' into core3-devel 2016-11-15 17:41:47 +01:00
Wojtek Porczyk
37bfd0d2a3 Merge remote-tracking branch 'marmarek/core3-fake-ip' into core3-devel 2016-11-15 17:40:30 +01:00
Wojtek Porczyk
a4d50409df Merge remote-tracking branch 'marmarek/core3-storage-fixes' into core3-devel 2016-11-15 17:36:53 +01:00
Wojtek Porczyk
fd953f4f27 Merge remote-tracking branch 'marmarek/core3-backup2' into core3-devel 2016-11-15 17:34:12 +01:00
Bahtiar `kalkin-` Gadimov
5db67fca8d
Fix init property swap in DomainVolumes 2016-11-07 23:26:53 +01:00
Marek Marczykowski-Górecki
b011cef8af
tests/storage: add tests for basic volumes properties
Things like if read-only volume is really read-only, volatile is
volatile etc.

QubesOS/qubes-issues#2256
2016-11-04 14:18:56 +01:00
Marek Marczykowski-Górecki
b59463e8e8
qvm-block: fix listing non-internal volumes
In case of LVM (at least), "internal" flag is initialized only when
listing volume attached to given VM, but not when listing them from the
pool. This looks like a limitation (bug?) of pool driver, it looks like
much nicer fix is to handle the flag in qvm-block tool (which list VMs
volumes anyway), than in LVM storage pool driver (which would need to
keep second copy of volumes list - just like file driver).

QubesOS/qubes-issues#2256
2016-11-04 14:18:56 +01:00
Marek Marczykowski-Górecki
1a7f2892d1
storage/lvm: fix logic regarding snapshots, start, stop etc
There are mutiple cases when snapshots are inconsistently created, for
example:
 - "-back" snapshot created from the "new" data, instead of old one
 - "-snap" created even when volume.snap_on_start=False
 - probably more

Fix this by following volume.snap_on_start and volume.save_on_stop
directly, instead of using abstraction of old volume types.

QubesOS/qubes-issues#2256
2016-11-04 14:18:56 +01:00
Marek Marczykowski-Górecki
ab9d7fbb76
storage: improve/fix handling extra volumes
Just calling pool.init_volume isn't enough - a lot of code depends on
additional data loaded into vm.storage object. Provide a convenient
wrapper for this.

At the same time, fix loading extra volumes from qubes.xml - don't fail
on volume not mentioned in initial vm.volume_config.

QubesOS/qubes-issues#2256
2016-11-04 14:18:56 +01:00
Marek Marczykowski-Górecki
4323651afb
storage/lvm: remove duplicated _reset function
There were two: _reset and _reset_volume. Neither of them was working,
but the later was closer. Remove the other one.

QubesOS/qubes-issues#2256
2016-11-04 14:18:55 +01:00
Marek Marczykowski-Górecki
37dbf29bc1
storage/lvm: don't fail on removing already removed volumes
This may happen when removing not fully created VM.

QubesOS/qubes-issues#2256
2016-11-04 14:18:55 +01:00
Marek Marczykowski-Górecki
400e92b25a
storage/lvm: misc fixes
- add missing lvm remove call when commiting changes
- delay creating volatile image until domain startup (it will be created
  then anyway)
- reset cache only when really changed anything
- attach VM to the volume (snapshot) created for its runtime - to not
  expose changes (for example in root volume) to child VMs until
  shutdown

QubesOS/qubes-issues#2412
QubesOS/qubes-issues#2256
2016-11-04 14:18:55 +01:00
Marek Marczykowski-Górecki
0471453773
storage/lvm: call lvm directly, don't use qubes-lvm wrapper
The wrapper doesn't do anything else than translating command
parameters, but it's load time is significant (because of python imports
mostly). Since we can't use python lvm API from non-root user anyway,
lets drop the wrapper and call `lvm` directly (or through sudo when
necessary).

This makes VM startup much faster - storage preparation is down from
over 10s to about 3s.

QubesOS/qubes-issues#2256
2016-11-04 14:18:55 +01:00
Marek Marczykowski-Górecki
9197bde76e
storage/lvm: use dd for importing volumes
...instead of manual copy in python. DD is much faster and when used
with `conv=sparse` it will correctly preserve sparse image.

QubesOS/qubes-issues#2256
2016-11-04 14:18:54 +01:00
Marek Marczykowski-Górecki
38fc504ca0
qubes/vm/net: set mapped IP info before attaching network
Set parameters for possibly hiding domain's real IP before attaching
network to it, otherwise we'll have race condition with vif-route-qubes
script.

QubesOS/qubes-issues#1143
2016-11-01 00:37:43 +01:00
Marek Marczykowski-Górecki
b4fa8cdce3
qubes/vm/net: use domain's "visible IP" for a gateway address
This is the IP known to the domain itself and downstream domains. It may
be a different one than seen be its upstream domain.

Related to QubesOS/qubes-issues#1143`
2016-11-01 00:30:11 +01:00
Marek Marczykowski-Górecki
ec81b3046f
tests: add missing app.save() before starting a domain
Otherwise domain will be unknown to other processes (like qrexec
services).
2016-11-01 00:28:37 +01:00
Marek Marczykowski-Górecki
d999d91049
tests: few more tests for fake/custom IP
QubesOS/qubes-issues#1143
QubesOS/qubes-issues#1477
2016-10-31 03:39:46 +01:00
Marek Marczykowski-Górecki
ea33fef9cc
tests: drop dispvm_netvm tests
This property no longer exists in core3.
2016-10-31 03:10:12 +01:00
Marek Marczykowski-Górecki
5072acc8f2
tests: custom VM IP
QubesOS/qubes-issues#1477
2016-10-31 03:09:45 +01:00
Marek Marczykowski-Górecki
b8145595a9
qubes/vm/net: allow setting custom IP
Fixes QubesOS/qubes-issues#1477
2016-10-31 03:04:13 +01:00
Marek Marczykowski-Górecki
4585f2b503
tests: add tests for fake IP feature
QubesOS/qubes-issues#1143
2016-10-31 02:17:21 +01:00
Marek Marczykowski-Górecki
2c6c476410
qubes/vm/net: add feature of hiding real IP from the VM
This helps hiding VM IP for anonymous VMs (Whonix) even when some
application leak it. VM will know only some fake IP, which should be set
to something as common as possible.
The feature is mostly implemented at (Proxy)VM side using NAT in
separate network namespace. Core here is only passing arguments to it.
It is designed the way that multiple VMs can use the same IP and still
do not interfere with each other. Even more: it is possible to address
each of them (using their "native" IP), even when multiple of them share
the same "fake" IP.

Original approach (marmarek/old-qubes-core-admin#2) used network script
arguments by appending them to script name, but libxl in Xen >= 4.6
fixed that side effect and it isn't possible anymore. So use QubesDB
instead.

From user POV, this adds 3 "features":
 - net/fake-ip - IP address visible in the VM
 - net/fake-gateway - default gateway in the VM
 - net/fake-netmask - network mask
The feature is enabled if net/fake-ip is set (to some IP address) and is
different than VM native IP. All of those "features" can be set on
template, to affect all of VMs.
Firewall rules etc in (Proxy)VM should still be applied to VM "native"
IP.

Fixes QubesOS/qubes-issues#1143
2016-10-31 02:06:01 +01:00
Marek Marczykowski-Górecki
b91714b204
qubes/features: handle recursive templates
Have features.check_with_template() check the template recursively.
The longest path (currently) is: DispVM -> AppVM -> TemplateVM.
2016-10-31 02:04:27 +01:00
Wojtek Porczyk
0141e1ac73 qubes/app: Allow keeping lock after load
QubesOS/qubes-issues#1729
2016-10-28 15:43:43 +02:00
Marek Marczykowski-Górecki
8cf19e3c92
tests/backupcompatibility: verify restored VM properties 2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
36bd834c01
core2migration: try to set properties to "default" when possible
Core3 keep information whether property have default value for all the
properties (not only few like netvm or kernel). Try to use this feature
as much as possible.
2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
64ac7f6e8d
tests/backup: check non-ASCII passphrase
QubesOS/qubes-issues#2398
2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
af182c4fd1
backup: fixup restore options just before restoring VMs
When user included/excluded some VMs for restoration, it may be
neceesarry to fix dependencies between them (for example when default
template is no longer going to be restored).
Also fix handling conflicting names.
2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
043d20c05d
backup: fix handling non-ascii characters in backup passphrase
Fixes QubesOS/qubes-issues#2398
2016-10-28 11:53:33 +02:00
Marek Marczykowski-Górecki
fc00dd211e
tests/backup: test backup with non-ASCII passphrase 2016-10-28 11:53:33 +02:00
Marek Marczykowski-Górecki
49e718cf57
backup: mark 'encryption' option as deprecated - all backups are encrypted
QubesOS/qubes-issues#971
2016-10-28 11:53:33 +02:00
Marek Marczykowski-Górecki
51b66208f3
backup: verify if archive chunks are not reordered
Now, when file name is also integrity protected (prefixed to the
passphrase), we can make sure that input files are given in the same
order. And are parts of the same VM.

QubesOS/qubes-issues#971
2016-10-28 11:53:33 +02:00
Marek Marczykowski-Górecki
4ad15c082b
backup: add 'backup_id' to integrity protection
This prevent switching parts of backup of the same VM between different
backups made by the same user (or actually: with the same passphrase).

QubesOS/qubes-issues#971
2016-10-28 11:53:32 +02:00
Marek Marczykowski-Górecki
418d749680
backup: use 'scrypt' tool for backup encryption and integrity protection
`openssl dgst` and `openssl enc` used previously poorly handle key
stretching - in case of `openssl enc` encryption key is derived using
single MD5 iteration, without even any salt. This hardly prevent
brute force or even rainbow tables attacks. To make things worse, the
same key is used for encryption and integrity protection which ease
brute force even further.
All this is still about brute force attacks, so when using long, high
entropy passphrase, it should be still relatively safe. But lets do
better.
According to discussion in QubesOS/qubes-issues#971, scrypt algorithm is
a good choice for key stretching (it isn't the best of all existing, but
a good one and widely adopted). At the same time, lets switch away from
`openssl` tool, as it is very limited and apparently not designed for
production use. Use `scrypt` tool, which is very simple and does exactly
what we need - encrypt the data and integrity protect it. Its archive
format have own (simple) header with data required by the `scrypt`
algorithm, including salt. Internally data is encrypted with AES256-CTR
and integrity protected with HMAC-SHA256. For details see:
https://github.com/tarsnap/scrypt/blob/master/FORMAT

This means change of backup format. Mainly:

1. HMAC is stored in scrypt header, so don't use separate file for it.
Instead have data in files with `.enc` extension.
2. For compatibility leave `backup-header` and `backup-header.hmac`. But
`backup-header.hmac` is really scrypt-encrypted version of `backup-header`.
3. For each file, prepend its identifier to the passphrase, to
authenticate filename itself too. Having this we can guard against
reordering archive files within a single backup and across backups. This
identifier is built as:

        backup ID (from backup-header)!filename!

For backup-header itself, there is no backup ID (just 'backup-header!').

Fixes QubesOS/qubes-issues#971
2016-10-28 11:53:32 +02:00
Marek Marczykowski-Górecki
d7c355eadb
backup: make wait_backup_feedback/handle_streams less ugly
Have a generic function `handle_streams`, instead of
`wait_backup_feedback` with open coded process names and manual
iteration over them.

No functional change, besides minor logging change.
2016-10-28 11:53:32 +02:00
Marek Marczykowski-Górecki
6ee200236c
tests/backup: verify migration into LVM thin pool 2016-10-28 11:53:31 +02:00
Marek Marczykowski-Górecki
673fe4423a
tests: handle LVM thin pool 2016-10-28 11:53:31 +02:00
Marek Marczykowski-Górecki
fbecd08a58
tests/backup: exclude some VMs during restore 2016-10-28 11:53:31 +02:00
Wojtek Porczyk
3553b2e1d4 Make pylint happy 2016-10-25 17:27:02 +02:00
Wojtek Porczyk
8edbf0e406 qubes: Document all the events
fixes QubesOS/qubes-issues#1811
2016-10-25 17:11:38 +02:00
Wojtek Porczyk
5e62d3f7cb qubes/tests: substitute_entry_points
New context manager for temporary overriding entry point groups.

fixes QubesOS/qubes-issues#2111
2016-10-24 15:16:39 +02:00
Wojtek Porczyk
4c73c1b93a More green paint 2016-10-19 16:09:58 +02:00
Marek Marczykowski-Górecki
5babb68031
tests/backupcompatibility: verify if all files got restored
There is still no verification of disk images content, nor VM
properties...
2016-10-19 01:54:44 +02:00
Wojtek Porczyk
8097da7cab Paint the project green for testbench launch 2016-10-18 19:07:20 +02:00
Wojtek Porczyk
c81346ba51 qubes/test/run: Print tracebacks of expected fails
fixes QubesOS/qubes-issues#2376
2016-10-14 17:20:14 +02:00
Wojtek Porczyk
526f2c3751 Merge remote-tracking branch 'marmarek/core3-backup' into core3-devel 2016-10-14 15:29:37 +02:00
Wojtek Porczyk
e06829ab2c Make pylint happy 2016-10-11 13:42:37 +02:00
Wojtek Porczyk
c6c0a545e6 Merge remote-tracking branch 'origin/pull/58/head' into core3-devel 2016-10-11 11:37:15 +02:00
Marek Marczykowski-Górecki
33fecd90c1
qubes/backup: misc fixes
Fix restoring ProxyVM and NetVM from core2. Use correct VM class.
2016-10-05 01:58:11 +02:00
Marek Marczykowski-Górecki
339c47480e
qubes/backup: include LVM volumes content in backup
Use just introduced tar writer to archive content of LVM volumes (or
more generally: block devices). Place them as 'private.img' and
'root.img' files in the backup - just like in old format. This require
support for replacing file name in tar header - another thing trivially
supported with tar writer.
2016-10-05 01:55:30 +02:00
Marek Marczykowski-Górecki
36eb7f923f
qubes/tarwriter: add simple sparse-tar writer module
tar can't write archive with _contents_ of block device. We need this to
backup LVM-based disk images. To avoid dumping image to a file first,
create a simple tar archiver just for this purpose.

Python is not the fastest possible technology, it's 3 times slower than
equivalent written in C. But it's much easier to read, much less
error-prone, and still process 1GB image under 1s (CPU time, leaving
along actual disk reads). So, it's acceptable.
2016-10-05 01:54:41 +02:00
Marek Marczykowski-Górecki
278a5340dc
qubes/backup: fix relative path calculation
os.path.relpath strip trailing '/.' from the path, but it is important
to distinguish whole-directory archive (which is tar of '.').
2016-10-04 21:38:59 +02:00
Wojtek Porczyk
72bc0506b9 Code documentation: fix 🎫 reference to query github
Previously it queried long obsolete Trac at wiki.qubes-os.org.
2016-10-04 15:31:06 +02:00
Wojtek Porczyk
d7d926edaa Fix manpage generation 2016-10-04 11:32:04 +02:00
Wojtek Porczyk
9dc37c1ee7 Add possibility to override libvirt config
This is the equivalent of "custom config" from R3.x.

fixes QubesOS/qubes-issues#1798
2016-10-04 11:31:31 +02:00
Marek Marczykowski-Górecki
ab69fdd7f4
qubes/backup: reduce code duplication
Move inner tar process cleanup to a separate function
2016-10-03 13:43:36 +02:00
WetwareLabs
cedd822735 Fix sending monitor layout info when xrandr has one output disconnected
Signed-off-by: WetwareLabs <marcus@wetwa.re>
2016-09-29 14:13:38 +02:00
Marek Marczykowski-Górecki
c4632d6be8
tests/backup: test idea 2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
f2d79b9379
tests/backup: use round volume size
When handling LVM volumes, size must be multiply of 4MB.
2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
e938aa61ab
tests: cleanup test LVM volumes
Handle the case when vm.remove_from_disk does not cleanup all the
things.
2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
20590bff57
backup: adjust LVM volume size when restoring its content.
Old backup metadata (old qubes.xml) does not contain info about
individual volume sizes. So, extract it from tar header (using verbose
output during restore) and resize volume accordingly.
Without this, restoring volumes larger than default would be impossible.
2016-09-29 02:08:29 +02:00
Marek Marczykowski-Górecki
0a35bd06aa
backup: support relocating files to different storage pool
To ease all this, rework restore workflow: first create QubesVM objects,
and all their files (as for fresh VM), then override them with data
from backup - possibly redirecting some files to new location. This
allows generic code to create LVM volumes and then only restore its
content.
2016-09-29 02:08:29 +02:00
Marek Marczykowski-Górecki
4d45dd5549
tests/backup: check backup+restore of LVM based VM
The test fails for now...
2016-09-29 01:59:55 +02:00
Marek Marczykowski-Górecki
226695534b
tests/backup: handle non-default pool in BackupTestsMixin 2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
ae42308f5f
storage: improve handling volume export
1. Add a helper function on vm.storage. This is equivalent of:

    vm.storage.get_pool(vm.volumes[name]).export(vm.volumes[name])

2. Make sure the path returned by `export` on LVM volume is accessible.
2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
9395e8fc33
storage: set only 'default' pool when creating VM on custom one
Do not replace 'linux-kernel' pool for example.
2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
e1d9de1cc2
tests/backup: minor fix for python3 2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
6d5959b31d
tests/backup: use proper logging instead of print 2016-09-29 01:59:53 +02:00
Marek Marczykowski-Górecki
016c3d8e88
tests/backup: check restored disk images 2016-09-29 01:59:53 +02:00
Bahtiar `kalkin-` Gadimov
8d9b6f19fd
Add catch-all '*' to event handlers 2016-09-28 21:28:43 +02:00
Bahtiar `kalkin-` Gadimov
ef56620b6e
Make pylint happy ♥ qubes/core2migration.py 2016-09-28 21:28:33 +02:00
Marek Marczykowski-Górecki
e499b529ad
tests: move BackupTestMixin to qubes.tests.int.backup
This is much more logical place, don't pollute main qubes.tests module.
2016-09-25 16:31:31 +02:00
Marek Marczykowski-Górecki
533804ebdc
Make pylint happy ♥ 2016-09-21 16:39:06 +02:00
Marek Marczykowski-Górecki
96a4bb650b
qubes/tools: qvm-backup and qvm-backup-restore tools
Fixes QubesOS/qubes-issues#1213
Fixes QubesOS/qubes-issues#1214
2016-09-21 16:02:50 +02:00
Marek Marczykowski-Górecki
7af3f4b19a
Make pylint happy 2016-09-19 20:36:32 +02:00
Marek Marczykowski-Górecki
d4f29bb8b7
qubes: add qvm-firewall tool
Fixes QubesOS/qubes-issues#1815
2016-09-19 20:36:32 +02:00
Marek Marczykowski-Górecki
f8dd7e6cb7
qubes/firewall: make xml parameter to Rule optional
QubesOS/qubes-issues#1815
2016-09-19 20:36:32 +02:00
Marek Marczykowski-Górecki
cdc97730cd
qubes/firewall: apply only IPv4 rules
Currently dom0 do not assign IPv6 addresses for VMs, so there is no
sense in IPv6 firewall yet.

QubesOS/qubes-issues#1815
2016-09-19 20:36:32 +02:00
Marek Marczykowski-Górecki
202042bd8d
tests: update for new firewall API
QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
d5b3d971ee
qubes/ext/r3compat: update firewall handling for new API
QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
5123f466eb
qubes/firewall: allow listing only IPv4/IPv6 rules
This will allow setting only IPv4-related rules to IPv4 address, and the
same for IPv6

QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
e01f7b97d9
qubes/vm: plug in new firewall code, create QubesDB entries
QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
1da75a676f
qubes/firewall: new firewall interface
First part - handling firewall.xml and rules formatting.
Specification on https://qubes-os.org/doc/vm-interface/

TODO (for dom0):
 - plug into QubesVM object
 - expose rules in QubesDB (including reloading)
 - drop old functions (vm.get_firewall_conf etc)

QubesOS/qubes-issues#1815
2016-09-19 20:36:30 +02:00
Marek Marczykowski-Górecki
1af1784c69
Merge remote-tracking branch 'qubesos/pr/54' into core3-devel
* qubesos/pr/54:
  Fix file storage resize
  Fix lvm reset_cache size cache
2016-09-19 20:36:20 +02:00
Marek Marczykowski-Górecki
0ea7afd245
tests: adjust LVM tests for non-default VG name 2016-09-19 20:35:25 +02:00
Marek Marczykowski-Górecki
8b408a7a70
qubes/vm: minor formating 2016-09-13 02:15:25 +02:00
Marek Marczykowski-Górecki
8ca08c7790
qubes/ext/pci: fix handling dom0 before starting first VM
Before starting fist VM, backend/pci xenstore directory does not exists.
Do not crash on it

QubesOS/qubes-issues#2257
2016-09-13 02:15:25 +02:00
Marek Marczykowski-Górecki
2956f62bbc
qmemman: fix meminfo handling
This fixes "a2d9b15 qmemman: support simple VM meminfo format"
2016-09-13 02:15:25 +02:00
Marek Marczykowski-Górecki
9ae3dc2ebf
tests/qvm-device: tolerate different 'column' tool versions
Newer one do output column even if empty, but older one (in Travis-CI
env) do not. Ignore trailing spaces to work with both version.
2016-09-13 02:01:22 +02:00
Marek Marczykowski-Górecki
141128e768
qubes/tools: fix 'column' input formating
Include '\n' at the last line too. Otherwise the tool (depending on
version) will complain about truncated line ('line too long').
2016-09-13 01:22:24 +02:00
Marek Marczykowski-Górecki
bb78eb1ce6
qubes: fix resetting already default property
`getattr(instance, self.__name__)` always succeed if attribute has
default value. This results in deleting attribute even when it isn't
set.
2016-09-08 04:17:58 +02:00
Marek Marczykowski-Górecki
a2d9b15413
qmemman: support simple VM meminfo format
Instead of excerpt from /proc/meminfo, use just one integer. This make
qmemman handling much easier and ease implementation for non-Linux OSes
(where /proc/meminfo doesn't exist).

For now keep also support for old format.

Fixes QubesOS/qubes-issues#1312
2016-09-08 04:17:48 +02:00
Marek Marczykowski-Górecki
aa0674e8bb
qubes/vm: make VM QubesDB interface as much compatible as possible
All the base keys can be kept easily the same, so do it.

QubesOS/qubes-issues#1812
2016-09-08 04:17:48 +02:00
Marek Marczykowski-Górecki
d5a41e838f
Revert "qubes/vm/qubesvm: remove prefixes from qubesdb keys"
There is no point in changing *public API* for just a change without any
better reason. It turned out most of those settings will be the same in
Qubes 4.0, so keep names the same.

This reverts commit 2d6ad3b60c.

QubesOS/qubes-issues#1812
2016-09-08 04:17:48 +02:00
Marek Marczykowski-Górecki
f6bc61997a
qubes/vm: fix handling dynamic netvm change
Correctly detach network before attaching new one.
2016-09-08 04:17:48 +02:00
Marek Marczykowski-Górecki
c534b68665
qubes/vm: start VM daemons as normal user
This is migration of core2 commits:

commit d0ba43f253
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date:   Mon Jun 6 02:21:08 2016 +0200

    core: start guid as normal user even when VM started by root

    Another attempt to avoid permissions-related problems...

    QubesOS/qubes-issues#1768

commit 89d002a031
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date:   Mon Jun 6 02:19:51 2016 +0200

    core: use runuser instead of sudo for switching root->user

    There are problems with using sudo in early system startup
    (systemd-logind not running yet, pam_systemd timeouts). Since we don't
    need full session here, runuser is good enough (even better: faster).

commit 2265fd3d52
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date:   Sat Jun 4 17:42:24 2016 +0200

    core: start qubesdb as normal user, even when VM is started by root

    On VM start, old qubesdb-daemon is terminated (if still running). In
    practice it happen only at VM startart (shutdown and quickly start
    again). But in that case, if the VM was started by root, such operation
    would fail.
    So when VM is started by root, make sure that qubesdb-daemon will be
    running as normal user (the first user in group 'qubes' - there should
    be only one).

    Fixes QubesOS/qubes-issues#1745
2016-09-08 04:17:47 +02:00
Marek Marczykowski-Górecki
19d9edc291
qubes/ext/gui: adjust guid parameters when running on KDE5
Commit from core2:

    commit 94d52a13e7

    core: adjust guid parameters when running on KDE5

    On KDE5 native decoration plugin is used and requires special properties
    set (instead of `_QUBES_VMNAME` etc).
    Special care needs to be taken when detecting environment, because
    environment variables aren't good enough - this script may be running
    with cleared environment (through sudo, or from systemd). So check
    properties of X11 root window.

    QubesOS/qubes-issues#1784
2016-09-08 04:17:47 +02:00
Bahtiar `kalkin-` Gadimov
d63d93b0ab
Fix file storage resize 2016-09-05 16:56:57 +02:00
Bahtiar `kalkin-` Gadimov
d72448615d
Fix lvm reset_cache size cache 2016-09-04 23:25:39 +02:00
Marek Marczykowski-Górecki
2285789b49
Merge remote-tracking branch 'qubesos/pr/54' into core3-devel
* qubesos/pr/54:
  Volume add size property wrapper
2016-09-04 22:38:06 +02:00
Bahtiar `kalkin-` Gadimov
b863152f73
Volume add size property wrapper 2016-09-04 21:55:56 +02:00
Marek Marczykowski-Górecki
16db68b053
Merge branch 'core3-devices' into core3-devel
* core3-devices:
  Fix core2migration and tests for new devices API
  tests: more qubes.devices tests
  qubes/ext/pci: implement pci-no-strict-reset/BDF feature
  qubes/tools: allow calling qvm-device as qvm-devclass (like qvm-pci)
  qubes: make pylint happy
  qubes/tools: add qvm-device tool (and tests)
  tests: load qubes.tests.tools.qvm_ls
  tests: PCI devices tests
  tests: add context manager to catch stdout
  qubes/ext/pci: move PCI devices handling to an extension
  qubes/devices: use more detailed exceptions than just KeyError
  qubes/devices: allow non-persistent attach
  qubes/storage: misc fixes for VM-exposed block devices handling
  qubes: new devices API

Fixes QubesOS/qubes-issues#2257
2016-09-04 21:32:01 +02:00
Marek Marczykowski-Górecki
ae72e294cd
Merge remote-tracking branch 'qubesos/pr/52' into core3-devel
* qubesos/pr/52:
  qvm-block extend Fix help message
  Fix a few typo bugs in qubes.storage.lvm
  qvm-block extends volumes to NEW_SIZE
  Make pylint happy ♥ qubes.storage.lvm
  Fix ThinVolume _size access
  qvm-block extend use qubes.utils.parse_size
  Make pylint happy ♥ qubes_lvm
  Fix lvm size/usage
  Add ThinPool.resize()
  Document qubes-block extend command in the manpage
  Fix file storage resize
  qvm-block add extend sub command
  qvm-block refactor attach/detach sub_parser init
2016-09-04 21:21:43 +02:00
Marek Marczykowski-Górecki
dc3dd1f5e9
Merge remote-tracking branch 'qubesos/pr/53' into core3-devel
* qubesos/pr/53:
  Fix qvm-block lists internal volumes on non default pools
2016-09-04 21:11:27 +02:00
Marek Marczykowski-Górecki
6cb624daf3
Merge remote-tracking branch 'qubesos/pr/51' into core3-devel
* qubesos/pr/51:
  Make pylint happy ♥ qubes.storage.lvm
  Fix lvm AppVM startup from lvm
  qubes.storage.lvm.qubes_lvm use debug not info
  Fix qvm-create lvm based AppVM
  qvm-block & qvm-pool add better Exception handling
  Remove unexpected property
  qvm-start --debug don't catch exceptions
2016-09-04 21:02:19 +02:00
Bahtiar `kalkin-` Gadimov
5f586a99d4
qvm-block extend Fix help message 2016-09-04 21:01:16 +02:00
Marek Marczykowski-Górecki
5cf15ef3da
tests: improve DispVM error message 2016-09-04 21:00:02 +02:00
Marek Marczykowski-Górecki
c15dc0a392
tools/qvm_block: add missing column header
QubesOS/qubes-issues#2256
2016-09-04 20:59:37 +02:00
Marek Marczykowski-Górecki
c2617663e5
qubes/vm/standalonevm: volatile image should be writable
QubesOS/qubes-issues#2256
2016-09-04 20:58:11 +02:00
Marek Marczykowski-Górecki
0af5fdd267
qubes/storage/file: clone file images as normal user
No need to be root here.

QubesOS/qubes-issues#2256
2016-09-04 20:56:02 +02:00
Marek Marczykowski-Górecki
11191ea694
Fix core2migration and tests for new devices API
QubesOS/qubes-issues#2257
2016-09-04 20:55:07 +02:00
Marek Marczykowski-Górecki
e536bcca30
tests: more qubes.devices tests
QubesOS/qubes-issues#2257
2016-09-03 20:41:06 +02:00
Marek Marczykowski-Górecki
93e88e0c22
qubes/ext/pci: implement pci-no-strict-reset/BDF feature
Instead of old per-VM flag 'pci_strictreset', now implement this as
per-device flag using features. To not fail on particular device
assignment set 'pci-no-strict-reset/DEVICE-BDF' to True. For
example 'pci-no-strict-reset/00:1b.0'.

QubesOS/qubes-issues#2257
2016-09-03 20:41:06 +02:00
Marek Marczykowski-Górecki
4d327fbc95
qubes/tools: allow calling qvm-device as qvm-devclass (like qvm-pci)
Tool can be symlinked to `qvm-class` for particular device class (for
example `qvm-pci`) - then device class can be omitted.

QubesOS/qubes-issues#2257
2016-09-03 20:41:05 +02:00
Marek Marczykowski-Górecki
5c7f589330
qubes: make pylint happy 2016-09-03 20:41:05 +02:00
Marek Marczykowski-Górecki
e8d011b83f
qubes/tools: add qvm-device tool (and tests)
Add a tool to manipulate various devices.

QubesOS/qubes-issues#2257
2016-09-03 20:41:05 +02:00
Marek Marczykowski-Górecki
6aae6863b0
tests: load qubes.tests.tools.qvm_ls
The test module was implemented, but not loaded
2016-09-03 20:41:04 +02:00
Marek Marczykowski-Górecki
fbb086aed5
tests: PCI devices tests
Just basic things for now.

QubesOS/qubes-issues#2257
2016-09-03 20:41:04 +02:00
Marek Marczykowski-Górecki
4bfb523974
tests: add context manager to catch stdout
This will avoid code duplication in tools tests.
2016-09-03 20:41:03 +02:00
Marek Marczykowski-Górecki
aa67a4512e
qubes/ext/pci: move PCI devices handling to an extension
Implement required event handlers according to documentation in
qubes.devices.

A modification of qubes.devices.DeviceInfo is needed to allow dynamic,
read-only properties.

QubesOS/qubes-issues#2257
2016-09-03 20:41:03 +02:00
Marek Marczykowski-Górecki
e1de82ea53
qubes/devices: use more detailed exceptions than just KeyError
Especially inherit from QubesException, so tools will treat this as
properly handled error.

QubesOS/qubes-issues#2257
2016-09-03 20:41:03 +02:00
Marek Marczykowski-Górecki
e5d6c4e078
qubes/devices: allow non-persistent attach
QubesOS/qubes-issues#2257
2016-09-03 20:41:02 +02:00
Marek Marczykowski-Górecki
70d3f58024
qubes/storage: misc fixes for VM-exposed block devices handling
Add 'backenddomain' element when source (not target) domain is not dom0.
Fix XML elemenet name. Actually set volume.domain when listing
VM-exposed devices.

QubesOS/qubes-issues#2256
2016-09-03 20:41:02 +02:00
Marek Marczykowski-Górecki
d7a3c0d319
qubes: new devices API
Allow device plugin to list attached and available devices. Enforce
at API level every device being exposed by some domain.

This commit only changes devices API, but not update existing users
(pci) yet.

QubesOS/qubes-issues#2257
2016-09-03 20:40:39 +02:00
Bahtiar `kalkin-` Gadimov
e4433f9a65
Make pylint happy ♥ qubes.storage.lvm 2016-09-02 20:16:03 +02:00
Bahtiar `kalkin-` Gadimov
46eb4a72dd
Fix a few typo bugs in qubes.storage.lvm 2016-09-02 19:57:35 +02:00
Bahtiar `kalkin-` Gadimov
dcfc47fefe
qvm-block extends volumes to NEW_SIZE 2016-09-02 19:57:13 +02:00
Bahtiar `kalkin-` Gadimov
be48d48e56
Make pylint happy ♥ qubes.storage.lvm 2016-09-02 19:46:11 +02:00
Bahtiar `kalkin-` Gadimov
8974b2cbc0
Fix ThinVolume _size access 2016-09-02 19:45:31 +02:00
Bahtiar `kalkin-` Gadimov
1cf701ff60
qvm-block extend use qubes.utils.parse_size 2016-09-02 19:21:04 +02:00
Bahtiar `kalkin-` Gadimov
818ed20248
Make pylint happy ♥ qubes_lvm 2016-09-02 19:17:00 +02:00
Bahtiar `kalkin-` Gadimov
bbcf16410c
Fix qvm-block lists internal volumes on non default pools 2016-08-29 17:39:49 +02:00
Bahtiar `kalkin-` Gadimov
1ca3c00797
Fix lvm size/usage
- Uses a size_cache, because it's faster than doing a call for each volume
2016-08-29 00:15:47 +02:00
Bahtiar `kalkin-` Gadimov
0561dfce60
Add ThinPool.resize() 2016-08-29 00:13:50 +02:00
Bahtiar `kalkin-` Gadimov
e7caa14a85
Fix file storage resize 2016-08-29 00:09:55 +02:00
Bahtiar `kalkin-` Gadimov
4733e836ce
qvm-block add extend sub command 2016-08-29 00:09:55 +02:00
Bahtiar `kalkin-` Gadimov
4c65a9c69a
qvm-block refactor attach/detach sub_parser init 2016-08-29 00:09:53 +02:00
Bahtiar `kalkin-` Gadimov
355ea0c2a8
Fix lvm AppVM startup from lvm 2016-08-29 00:05:19 +02:00
Bahtiar `kalkin-` Gadimov
24bbc971b0
qubes.storage.lvm.qubes_lvm use debug not info 2016-08-29 00:05:19 +02:00
Bahtiar `kalkin-` Gadimov
e0bf4bdb8e
Fix qvm-create lvm based AppVM 2016-08-29 00:05:18 +02:00
Bahtiar `kalkin-` Gadimov
c0f1a6b4e8
qvm-block & qvm-pool add better Exception handling 2016-08-29 00:05:18 +02:00
Bahtiar `kalkin-` Gadimov
b712d0f6f3
Remove unexpected property 2016-08-29 00:05:18 +02:00
Bahtiar `kalkin-` Gadimov
8037182dcd
qvm-start --debug don't catch exceptions 2016-08-29 00:05:17 +02:00
Marek Marczykowski-Górecki
3e30dc17cb
qubes/vm: remove special case for 'meminfo-writer' feature
It should be either in a plugin, or handled entirely from inside of VM.
But surely not in main QubesVM class.

Fixes QubesOS/qubes-issues#2101
2016-08-18 11:59:45 +02:00
Marek Marczykowski-Górecki
0293c1c7ef
qubes/vm: move misc XML tags loading to separate method 2016-08-18 11:57:11 +02:00
Marek Marczykowski-Górecki
a5e575618c
qubes-lvm: drop sudo from lvm calls
The script is already running as root

QubesOS/qubes-issues#2256
2016-08-18 11:52:26 +02:00
Marek Marczykowski-Górecki
6bdc82b3a6
qubes-lvm: do not import 'qubes' module
This script should run as fast as possible, so avoid importing large
module. In fact the only used thing was argparse wrapper, so switch to
the standard one and drop aliases.

QubesOS/qubes-issues#2256
2016-08-18 11:52:26 +02:00
Marek Marczykowski-Górecki
de5d420331
qubes-lvm: drop unused lvm_image_changed function
It tries to parse (untrusted) volume content, so remove it to not use it
accidentally.

QubesOS/qubes-issues#2256
2016-08-18 11:43:01 +02:00
Marek Marczykowski-Górecki
06f46243ec
qubes/storage: add resize to Pool API
It was already implemented by FilePool and used by Storage object, but
wasn't included in base class.
2016-08-18 11:42:00 +02:00
Marek Marczykowski-Górecki
42d6fddcd3
qubes/storage: support on-line resize
Call `qubes.ResizeDisk` qrexec service after resizing volume, to adjust
filesystem size there.
2016-08-18 11:40:38 +02:00
Marek Marczykowski-Górecki
6e383c703d
qubes/storage: allow passing just volume name to storage.resize
It makes much more sense to call:

    vm.storage.resize('private', 10000000000)

instead of:

    vm.storage.resize(vm.volumes['private'], 10000000000)
2016-08-17 22:42:15 +02:00
Marek Marczykowski-Górecki
d8286b6dab
tests: port vm_qrexec_gui to core3 API 2016-08-17 22:41:57 +02:00
Marek Marczykowski-Górecki
e19df4c5b6
tests: port dispvm tests to core3 API
Some tests do not apply, as there is no savefile and attributes
propagation is much simpler. Dropped tests:
 - test_000_firewall_propagation
 - test_001_firewall_propagation
 - test_000_prepare_dvm

QubesOS/qubes-issues#2253
2016-08-17 04:15:35 +02:00
Marek Marczykowski-Górecki
46791e05f4
qubes/dispvm: do not clone 'template' property from base VM
template should point at the base VM itself.
2016-08-17 02:14:19 +02:00
Marek Marczykowski-Górecki
53b2b30c0a
qubes: fix some error messages 2016-08-17 02:13:59 +02:00
Marek Marczykowski-Górecki
3a2a87839f
qubes/vm: fix TemplateVM.commit_changes
Storage method was renamed to just `commit`, as it applies not only to
templates now.
2016-08-17 01:47:09 +02:00
Marek Marczykowski-Górecki
d9e23d6535
qubes/core2migration: make pylint happy
It doesn't see the class that is just being defined. But since it is in
lambda, that's ok.
2016-08-17 01:46:13 +02:00
Marek Marczykowski-Górecki
6b546ca1e6
tests/backupcompatibility: no errors are expected on restoring R1 backup
Fallback to default kernel is enabled by default.
2016-08-17 01:31:33 +02:00
Marek Marczykowski-Górecki
690727fef8
tests/backup: remove VMs in reversed order
One place had missing `reversed()`.
2016-08-17 01:30:34 +02:00
Marek Marczykowski-Górecki
bbbe045b62
qubes/backup: better handle restoring kernel=None case
It makes sense for HVM domains, so allow such configuration.
2016-08-17 01:18:11 +02:00
Marek Marczykowski-Górecki
5d8ecd60de
qubes: minor fixes in handling defaults
- fix assigning 'template' property - do not do it if VM already have it
set
- cap default maxmem at 4000, as we clamp it to 10*memory anyway (and
  default memory is 400)
2016-08-17 00:50:38 +02:00
Marek Marczykowski-Górecki
dd93650ea7
tests/backup: fix handling different VM types
Not all VM types have 'template' property.
2016-08-17 00:49:53 +02:00
Marek Marczykowski-Górecki
460f0849a7
qubes/features: handle recursive templates in check_with_template
This is the case for DispVMs: DispVM -> AppVM -> TemplateVM
2016-08-17 00:47:32 +02:00
Marek Marczykowski-Górecki
1a215e47ab
qubes: add 'default_dispvm' property - both Qubes and QubesVM
QubesOS/qubes-issues#2253
2016-08-17 00:46:43 +02:00
Marek Marczykowski-Górecki
ab6f961931
qubes/dispvm: misc fixes, make it actually working
- DispVM is no longer a special case for storage
- Add missing 'rw=True' for volatile volume
- Handle storage initialization (copy&paste from AppVM)
- Clone properties from DispVM template

QubesOS/qubes-issues#2253
2016-08-17 00:42:17 +02:00
Marek Marczykowski-Górecki
2f977ff4f5
backup: open backup collection in offline mode
Otherwise it may try to manipulate live libvirt objects with the same
names (and probably different UUIDs, which would fail).
2016-08-16 00:00:14 +02:00
Marek Marczykowski-Górecki
eb637147fb
core2migration: adjust for recent storage API changes
QubesVM.storage.vmdir uses QubesVM.dir_path, so can't be used directly
as default value for dir_path. Use `super()`.
2016-08-15 23:47:28 +02:00
Marek Marczykowski-Górecki
660c726315
tests: add qvm-check 2016-08-09 05:14:13 +02:00
Marek Marczykowski-Górecki
da6faf83bf
qubes/tools: allow unpause multiple domains at once
There is no reason to not allow it.
2016-08-09 05:14:13 +02:00
Marek Marczykowski-Górecki
a1e2888e96
qubes/tools: fix messages in qvm-check 2016-08-09 05:14:13 +02:00
Marek Marczykowski-Górecki
d4e534bc05
qubes/vm: fix pause handling
1. There is no such thing as libvirt_domain.pause().
2. libvirt_domain.state() returns [state, reason]
2016-08-09 05:14:12 +02:00
Marek Marczykowski-Górecki
d2393f91e8
qubes/tools: adjust qvm-check for core3 API 2016-08-09 05:14:12 +02:00
Marek Marczykowski-Górecki
4d5a314e77
qubes/tools: add hidden --offline-mode options
It is useful on some cases to prevent talking to hypervisor.

Warning - it may have sense only when action do not access any runtime
VM status. For example running the domain will fail, but changing its
properties should work.
2016-08-09 05:14:12 +02:00
Marek Marczykowski-Górecki
96d032919a
tests: minor fix to revert changes tests, mark as expected failure
While the test should be ok now, the functionality itself doesn't work (at
least on file volumes).
2016-08-09 05:14:12 +02:00
Marek Marczykowski-Górecki
ac68ffe39f
tests: correctly compare DeviceManager objects
Usefull for assertEqual(vm1.devices, vm2.devices) - clone tests, backup
tests etc.
2016-08-09 05:14:11 +02:00
Marek Marczykowski-Górecki
f17f80ee9d
qubes/tools: add qvm-template-commit
This tool is called from /etc/xen/scripts/block-origin script at VM
shutdown.
2016-08-09 05:14:11 +02:00
Marek Marczykowski-Górecki
6490767278
qubes/storage: update vm.storage.commit
Better name it 'commit', as it isn't only for template. And also fix to
call `pool.commit`
2016-08-09 05:14:11 +02:00
Marek Marczykowski-Górecki
ec7a84ea9d
minor: formatting
QubesOS/qubes-issues#
2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
af7b4087f0
tests: port few more tests to core3 API 2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
faa341ffa2
tests/backup: adjust for core3 API properties to verify after restore 2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
7d1cbd6934
tests: handle per-template tests in BackupTestsMixin 2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
fba6eac07f
qubes/tests: do not undefine libvirt domain twice
Do not access vm.libvirt_domain after it being already removed - this
will redefine it again in libvirt, just to undefine it in a moment.
On the other hand, few lines below there is fallback libvirt cleanup, in
case of proper one not working.
2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
6d45b97357
tests: workaround 'int' name being shadowed by module name
qubes.tests.int shadows 'int' type in qubes.tests module.

QubesOS/qubes-issues#2241
2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
3f6c96ee39
qubes/vm: fix handling autostart set/reset
1. Service symlink must have ".service" extension
2. Disable service on property reset to default
2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
5b4e30524a
qubes/vm: remove duplicated check for duplicate name on rename 2016-08-09 03:02:04 +02:00
Marek Marczykowski-Górecki
73d61bb378
qubes/events: fix event handlers ordering
Since "qubes: fix event framework", handlers from extensions looks the
same as from the VM class itself, so it isn't possible to order them
correctly. Specification says:

    For each class first are called bound handlers (specified in class
    definition), then handlers from extensions. Aside from above,
    remaining order is undefined.

So, restore this property, which is later correctly used to order
handlers.
2016-08-09 02:58:14 +02:00
Marek Marczykowski-Górecki
5177f5e12b
qubes/core2migration: fix vm.template handling
VM constructor can't be given just template QID, it must be TemplateVM
object.
2016-08-09 02:57:23 +02:00
Marek Marczykowski-Górecki
067cfb7cd6
Send approximate physical screen dimensions to the VM
When properly set, applications will have a chance to automatically
detect HiDPI and act accordingly. This is the case for Fedora 23
template and GNOME apps (maybe even all built on top of GTK).

But for privacy reasons, don't provide real values, only some
approximate one. Give enough information to distinguish DPI above 150,
200 and 300. This is some compromise between privacy and HiDPI support.

QubesOS/qubes-issues#1951

This commit is migrated from gui-daemon repository
(dec462795d14a336bf27cc46948bbd592c307401).
2016-08-08 04:03:00 +02:00
Marek Marczykowski-Górecki
6bd127beb3
qubes/core2migration: fix handling templates 2016-08-08 04:00:37 +02:00
Marek Marczykowski-Górecki
5ab860a18c
qubes: don't fail VM removal if already undefined in libvirt
This may happen when it was manually undefined for some reason. Or
during tests cleanup (tearDown).
2016-08-08 00:15:46 +02:00
Marek Marczykowski-Górecki
53562799cd
Fix typo in libvirt auto reconnection 2016-08-08 00:13:26 +02:00
Marek Marczykowski-Górecki
4e022382a5
Merge remote-tracking branch 'origin/master' into core3-devel 2016-08-08 00:11:46 +02:00
Marek Marczykowski-Górecki
1604aee8d4
Merge remote-tracking branch 'qubesos/pr/42' into core3-devel
* qubesos/pr/42:
  Make pylint happy ♥
2016-08-04 17:46:16 +02:00
Bahtiar `kalkin-` Gadimov
4477ea9678
Make pylint happy ♥ 2016-08-04 17:38:39 +02:00
Wojtek Porczyk
288ff85731 Merge remote-tracking branch 'origin/pull/48/head' into core3-devel 2016-08-03 18:25:02 +02:00
Bahtiar `kalkin-` Gadimov
ea34c0ed56
Fix volume_exists in qubes_lvm 2016-08-03 02:57:46 +02:00
Bahtiar `kalkin-` Gadimov
02c8fc999c
ThinPool.import() skips not saveable volumes 2016-08-03 02:57:46 +02:00
Bahtiar `kalkin-` Gadimov
9190fdeb79
Fix lvm pylint disable=import-error 2016-08-03 02:57:45 +02:00
Bahtiar `kalkin-` Gadimov
064629b63a
Fix qubes.lvm _reset 2016-08-03 02:57:44 +02:00
Bahtiar `kalkin-` Gadimov
9d30bb132a
Fix umask 2016-08-03 02:57:44 +02:00
Bahtiar `kalkin-` Gadimov
824c359319
Fix ThinPool revision handling 2016-08-03 02:57:43 +02:00
Bahtiar `kalkin-` Gadimov
f3072e7dd0
qubes-create adds default lvm pool if it exists 2016-08-03 02:57:42 +02:00
Bahtiar `kalkin-` Gadimov
3ae6530cd8
Add lvm thin pool storage implementation 2016-08-03 02:57:42 +02:00
Bahtiar `kalkin-` Gadimov
7d11a7afbd
Add lvm tests 2016-08-03 02:57:41 +02:00
Bahtiar `kalkin-` Gadimov
f90d86fe13
Add qubes.utils.random_string() 2016-08-03 02:57:40 +02:00
Bahtiar `kalkin-` Gadimov
c559ffdeab
VMCollection reword vm already exists exception 2016-08-03 02:57:39 +02:00
Bahtiar `kalkin-` Gadimov
0dd1875fb6
Add vm.shutdown(wait) 2016-08-01 15:09:28 +02:00
Bahtiar `kalkin-` Gadimov
8972254e22
Add qvm-check 2016-07-21 23:40:24 +02:00