Commit Graph

1161 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
9d17c52a8d
tests: add missing import 2017-07-04 05:22:39 +02:00
Marek Marczykowski-Górecki
f393ce62dc
Fix docstrings formatting
Fix warnings/errors reported by Sphinx
2017-07-04 04:27:36 +02:00
Marek Marczykowski-Górecki
291a338e73
api: add missing docstring 2017-07-04 04:27:34 +02:00
Marek Marczykowski-Górecki
82c3f85042
storage: add API documentation
QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
5971873680
storage: drop functions not being part of the API
commit/recover/reset should really be handled in start/stop. Nothing
stops specific pool implementation to define such functions privately.

QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
820539e909
storage: make volume snap_on_start/save_on_stop explicit
Always define those properties, always include them in volume config.
Also simplify overriding pool based on volume type defined by those:
override pool unless snap_on_start=True.

QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
697eb05c20
storage: drop rename support
Since VM name is immutable, rename method can be dropped from storage
API.

QubesOS/qubes-issues#2868
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
dd1e05dc83
vm: drop rename related methods
Since VM name in immutable now, drop rename-related methods.

QubesOS/qubes-issues#2868
2017-07-04 04:11:13 +02:00
Marek Marczykowski-Górecki
2dbb51a38d
exc/block: fix attaching block device exposed by dom0
Don't set <backenddomain> element in libvirt XML when device is provided
by dom0.
2017-07-04 04:09:28 +02:00
Marek Marczykowski-Górecki
3721ae3e8b
vm/adminvm: setup libvirt_domain property
Since libvirt do provide object for dom0 too, return it here.
It's much easier than special-casing AdminVM everywhere. And in fact
sometimes it is actually useful (for example attaching devices from/to
dom0, adjusting memory).
2017-07-04 04:09:27 +02:00
Marek Marczykowski-Górecki
e5de8f4115
devices: better handle exceptions in device extension
Do not fail app.save() just because listing devices failed, for any
reason.
2017-07-04 04:09:27 +02:00
Marek Marczykowski-Górecki
3748eb3e2b
Merge branch 'core3-firewall2' 2017-07-04 03:38:59 +02:00
Marek Marczykowski-Górecki
f83c516082
Merge branch 'two-stage-clone' 2017-07-04 03:37:41 +02:00
Marek Marczykowski-Górecki
d0663940a7
qubes/api/admin: annotate API methods
Second attempt: this time use full words for scope, read, write,
execute.

QubesOS/qubes-issues#2871
2017-06-27 06:01:58 +02:00
Wojtek Porczyk
3e0d01cfb9
qubes/admin: Add listing of API methods 2017-06-27 06:01:58 +02:00
Marek Marczykowski-Górecki
2abdbc4628
firewall: minor simplification for old firewall.xml loading
Have `default_policy_is_accept` variable of type bool, instead of
`policy`, which is only compared to a constant value (`accept`).
Suggested by @woju
2017-06-26 18:45:59 +02:00
Marek Marczykowski-Górecki
2b963be9c7
firewall: pass untrusted values as keyword arguments
Use keyword arguments to pass untrusted arguments to make sure the
function parameter also have `untrusted_` prefix.
Suggested by @woju
2017-06-26 18:41:27 +02:00
Marek Marczykowski-Górecki
26a9974432
api/admin: split vm.volume.Clone to CloneFrom and CloneTo
The first operation returns a token, which can be passed to the second
one to actually perform clone operation. This way the caller needs have
power over both source and destination VMs (or at least appropriate
volumes), so it's easier to enforce appropriate qrexec policy.

The pending tokens are stored on Qubes() instance (as QubesAdminAPI is
not persistent). It is design choice to keep them in RAM only - those
are one time use and this way restarting qubesd is a simple way to
invalidate all of them. Otherwise we'd need some additional calls like
CloneCancel or such.

QubesOS/qubes-issues#2622
2017-06-26 13:27:48 +02:00
Marek Marczykowski-Górecki
3dcd29afea
api/admin: remove admin.vm.Clone operation
The same can be achieved with Create+volume.Clone

QubesOS/qubes-issues#2622
2017-06-26 13:10:00 +02:00
Marek Marczykowski-Górecki
fabd8119b4
storage: volume.import_volume now expect create()d volume
This is much more logical for *import*_volume function.

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
28f78ed3b8
storage/lvm: minor fixes
QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
a748b393f4
storage: move remove() to Volume
This is continuation of 0f12870 "storage: use direct object references,
not only identifiers".

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
ae600e24bf
storage: simplify pool.volumes usage
Add convenient collection wrapper for easier getting selected volume.
Storage pool implementation may still provide only volume listing
function (pool.list_volumes), or, additionally, optimized
pool.get_volume.

This means it is both possible to iterate over volumes:
```python
for volume in pool.volumes:
    ...

```

And get a single volume:
```python
volume = pool.volumes[vid]
```

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
0200fdadcb
api/admin: firewall-related methods
In the end firewall is implemented as .Get and .Set rules, with policy
statically set to 'drop'. This way allow atomic firewall updates.

Since we already have appropriate firewall format handling in
qubes.firewall module - reuse it from there, but adjust the code to be
prepared for potentially malicious input. And also mark such variables
with untrusted_ prefix.

There is also third method: .Reload - which cause firewall reload
without making any change.

QubesOS/qubes-issues#2622
Fixes QubesOS/qubes-issues#2869
2017-06-26 13:08:27 +02:00
Marek Marczykowski-Górecki
65d15e6040
api/admin: skip firewall in vm.Clone
This operation is going to be removed, so apply a quick fix for tests.

QubesOS/qubes-issues#2622
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
842efb577d
firewall: always use policy 'drop'
There is a problem with having separate default action ("policy") and
rules because it isn't possible to set both of them atomically at the
same time.
To solve this problem, always have policy 'drop' (as a safe default),
but by default have a single rule with action 'accept'

Fixes QubesOS/qubes-issues#2869
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
919841635b
qubes: validate if property value consists of ASCII only earlier
Do this for all standard property types - even if other types do
additional validation, do not expose them to non-ASCII characters.

QubesOS/qubes-issues#2622
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
3074a4064c
vm/qubesvm: forbid changing VM name
We've decided to make VM name immutable. This is especially important
for Admin API, where some parts (especially policy) are sticked to the
VM name.

Now, to rename the VM, one need to clone it under new name (thanks to
LVM, this is very quick action), then remove the old one.

Fixes QubesOS/qubes-issues#2868
2017-06-26 02:00:45 +02:00
Marek Marczykowski-Górecki
a6c16d00be
vm/qubesvm: fix handling empty kernel value
Do not fail on empty kernel value - which means "use booloader on
root.img".

Fixes 3ddc052 "vm: move kernel presence validation to event handler"
2017-06-26 01:59:39 +02:00
Marek Marczykowski-Górecki
26013122a0
Merge remote-tracking branch 'woju/devel-adminext' into core3-devel 2017-06-23 10:34:11 +02:00
Marek Marczykowski-Górecki
4208a98bd7
Merge branch 'core3-devel-20170619' 2017-06-23 02:53:17 +02:00
Marek Marczykowski-Górecki
f976f7ec6c
storage: simplify coroutine handling
Suggested by @woju
2017-06-23 02:35:49 +02:00
Marek Marczykowski-Górecki
57e293eb54
Merge branch 'core3-qmemman-fix' 2017-06-22 23:16:35 +02:00
Wojtek Porczyk
8c9ce0587b ext/admin: add explanation to PermissionDenied 2017-06-22 13:21:37 +02:00
Wojtek Porczyk
2942f8bcac qubes: admin extension
for managing tags
2017-06-21 23:12:54 +02:00
Marek Marczykowski-Górecki
c1f4c219f9
tests: adjust TC_00_QubesDaemonProtocol for reorganized api module 2017-06-21 06:59:58 +02:00
Marek Marczykowski-Górecki
588ff04f0d
qmemman: fix units on meminfo parsing
meminfo (written by VM) is expected report KiB, but qmemman internally
use bytes. Convert units.
And also move obscure unit conversion in is_meminfo_suspicious to more
logical place in sanitize_and_parse_meminfo.
2017-06-21 06:34:00 +02:00
Marek Marczykowski-Górecki
ea0cbe3a56
tests: improve tests for qrexe exit code handling
Check if exit code retrieved from dom0 is really the one expected.

Fix typo in test_065_qrexec_exit_code_vm (testvm1/testvm2), adjust for
reporing remote exit code and remove expectedFailure.

QubesOS/qubes-issues#2861
2017-06-21 05:23:35 +02:00
Marek Marczykowski-Górecki
a73dcf6016
tests: wait for session in tests requiring running GUI
Since tests expose qubesd socket, qvm-start-gui should handle starting
GUI daemons (so, GUI session inside VM). Add synchronization with it
using qubes.WaitForSession service.
2017-06-21 04:45:46 +02:00
Marek Marczykowski-Górecki
376ac4b32d
tests: fix vm.run_for_stdio in some more places
When test expect to wait for remote process, use vm.run_for_stdio.
Additionally, when the call fail, (stdout, stderr) is not assigned - use
the one attached to exception object instead.
2017-06-21 04:33:10 +02:00
Marek Marczykowski-Górecki
a0f616f14e
tests: fix checking exit code
Since run_for_stdio raise an exception for non-zero exit code, it isn't
ignored anymore. So, check if qrexec-client-vm return expected value,
instead of keep ignoring it.

QubesOS/qubes-issues#2861
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
72240c13b6
tests: fix vm_qrexec_gui/TC_10_Generic/test_000_anyvm_deny_dom0
When method (as expected) raise an exception, service output would not be
assigned. Extract it from exception object.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
a469c565f4
tests: fix QrexecPolicyContext
Flush new policy file to the disk, otherwise it will stay only in write
buffer.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
3ddc052af3
vm: move kernel presence validation to event handler
Setter is called also on qubes.xml load, so missing kernel breaks
qubes.xml parsing - for example qubesd startup to fix that property.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
4241b39b94
tests: fix tests cleanup 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
60443259d0
vm: raise CalledProcessError on failed service/command call 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
51a17ba749
tests: do not reload qubes.xml
In core3 this isn't needed anymore (and unit tests already check if
that's really true).
2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
ea5ca79133
tests: fix removing test VMs
Do it before shutting down qubesd socket - some things may require it
for VM removal/shutdown.
2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
eee6ab0c01
tests: use copy of qubes.xml, instead of empty one 2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
984a070f3e
tests: move create_*_file to SystemTestsMixin 2017-06-21 00:17:40 +02:00
Wojtek Porczyk
f56f7d13fb
tests/integ/vm_qrexec_gui: skip test_051_qrexec_simple_eof_reverse
QubesOS/qubes-issues#2851
2017-06-21 00:17:39 +02:00
Wojtek Porczyk
139f18fa1d
qubes/tests/integ/vm_qrexec_gui: some fixes 2017-06-21 00:17:39 +02:00
Wojtek Porczyk
0c0b0ea6ef
qubes/tests/integ/vm_qrexec_gui: change time.sleep to asyncio.sleep 2017-06-21 00:17:38 +02:00
Wojtek Porczyk
96a66ac6bd
qubes/api: refactor creating multiple qubesd sockets
Now there is a single function to do this, shared with tests.
2017-06-21 00:17:37 +02:00
Wojtek Porczyk
bec58fc861
qubes/tests: start qrexec policy responder for system tests 2017-06-20 13:00:20 +02:00
Wojtek Porczyk
71a4390fdb
qubes/tools/qubesd: properly unlink UNIX sockets at shutdown 2017-06-20 13:00:20 +02:00
Wojtek Porczyk
4b8e5c3704
qubes/tests/run: refuse to run tests if qubesd is running
Test suite creates some VMs and needs to pass the knowledge about them
to qrexec policy checker. This is done using Admin API, so we need to
substitute qubesd with our own API server.
2017-06-20 13:00:20 +02:00
Wojtek Porczyk
858e547525
qubes: reorganise API protocols
Now instantiating API servers is handled by common function. This is,
among other reasons, for creating ad-hoc sockets for tests.
2017-06-20 13:00:20 +02:00
Marek Marczykowski-Górecki
8196b2d5bf
admin.vm.Create: add commend about 'created-by-' tag 2017-06-20 12:47:01 +02:00
Marek Marczykowski-Górecki
c13cf44e5e
admin.vm.Create: add 'created-by-' tag 2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
083108e995
app: fix registering libvirt event handler
register_event_handlers is called early, when libvirt connection may not
be yet established - especially on empty qubes.xml. Do not skip
automatic connection logic.
2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
e4d285d479
vm/adminvm: make AdminVM sortable
One more thing gone during changing AdminVM base class.
2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
9242202db2
admin: implement admin.vm.tag.*
QubesOS/qubes-issues#2622
2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
4a1a5fc24b
exc: fix QubesNoTemplateError 2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
aadbe223c3
admin: add admin.vm.volume.Clone
QubesOS/qubes-issues#2622
2017-06-20 00:54:15 +02:00
Marek Marczykowski-Górecki
f48b1be669
storage: extract single volume clone into clone_volume
This will be useful for admin.vm.volume.Clone implementation.

QubesOS/qubes-issues#2256
2017-06-20 00:54:15 +02:00
Marek Marczykowski-Górecki
86a935e779
qubes.NotifyTools: ignore '/qubes-tools/version' completely
It isn't used for anything, so simply ignore it for good.

https://github.com/QubesOS/qubes-core-admin/pull/109#discussion_r121421409
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
55669c350c
api/misc: fix retrieving requested features from QubesDB
qdb.list return list of bytes().
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
5209bc370d
vm: init vm.storage and vm.volumes in BaseVM
This way also AdminVM will have (empty) properties there. It is much
cleaner than adding `if hasattr` or catching AttributeError everywhere.
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
160ab964bc
vm: improve sending events for tags and features
1. Send the event after setting tag/feature
2. Provide old value for feature
3. Rename 'key' kwarg to 'feature'
2017-06-14 10:44:23 +02:00
Marek Marczykowski-Górecki
8ba60cd7e9
app: fix listing VMs based on given template 2017-06-13 13:15:21 +02:00
Marek Marczykowski-Górecki
93ccb8bbda
Merge remote-tracking branch 'qubesos/pr/111'
* qubesos/pr/111:
  vm: drop 'internal' property
  qmemman: make sure to release lock
  qmemman: fix meminfo parsing for python 3
  devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
2017-06-12 23:22:55 +02:00
Marek Marczykowski-Górecki
ba4da5cbe0
Merge remote-tracking branch 'qubesos/pr/110'
* qubesos/pr/110:
  storage: use direct object references, not only identifiers
  vm: fix volume_config
  storage/lvm: prefix VM LVM volumes with 'vm-'
  storage: fix VM rename
2017-06-12 23:22:41 +02:00
Marek Marczykowski-Górecki
37245acdcf
vm: change vm.updates_available to a 'updates-available' feature 2017-06-12 12:34:23 +02:00
Marek Marczykowski-Górecki
caa03a9279
vm/qubesvm: simplify 'None' kernel handling 2017-06-12 12:26:57 +02:00
Marek Marczykowski-Górecki
ee442c754f
api/misc: integrate qubes.NotifyTools logic with qubes.FeaturesRequest
Make qubes.NotifyTools reuse logic of qubes.FeaturesRequest, then move
actual request processing to 'features-request' event handler. At the
same time implement handling 'qrexec' and 'gui' features request -
allowing to set template features when wasn't already there.
Behavior change: template is no longer allowed to change feature value
(regardless of being True or False). This means the user will always be
able to override what template have set.
2017-06-12 12:22:39 +02:00
Marek Marczykowski-Górecki
e8c303977b
vm/adminvm: fix str(AdminVM) after changing its base
BaseVM have no (useful) __str__ method.
2017-06-12 10:15:13 +02:00
Marek Marczykowski-Górecki
9d99232515
vm: drop 'internal' property
It isn't used anywhere in the code right now. And when it will be
needed, it should be a "feature" not "property".
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
206b7c02d5
qmemman: make sure to release lock
Even when handling updated meminfo or domain list something goes wrong,
make sure to release the lock - otherwise the whole qmemman will be
blocked.
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
9015414119
qmemman: fix meminfo parsing for python 3
One more place not converted to python 3.
2017-06-12 10:06:42 +02:00
Marek Marczykowski-Górecki
3564250298
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
Drop DeviceInfo.data - device extension should provide a subclass with
proper individual fields.
Drop DeviceAssignment.frontend_domain - this information is redundant -
frontend domain is defined by where DeviceAssignment is attached.
Rename DeviceCollection.devclass to bus - devclass if confusing here,
because this term is also used for DeviceInfo subclass.
2017-06-12 10:06:42 +02:00
Marek Marczykowski-Górecki
0f12870803
storage: use direct object references, not only identifiers
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object

Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.

One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.

Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
80b459b6d5
vm: fix volume_config
- kernel volume shouldn't have snap_on_start, it's read-only volume
  anyway
- root volume of AppVM should have placeholder for 'source'
- private volume of AppVM should _not_ have placeholder for 'source'
(it's ignored anyway, because snap_on_start=False)

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
fd5386cd47
storage/lvm: prefix VM LVM volumes with 'vm-'
This will allow filtering them out in udev rules - to not parse any of
it.

QubesOS/qubes-issues#2319
2017-06-09 04:54:59 +02:00
Marek Marczykowski-Górecki
fd3f19d4a6
storage: fix VM rename
When VM is renamed only volume.vid get updated, but not other attributes
calculated from it. Convert them to dynamic properties to not worry
about it.

QubesOS/qubes-issues#2256
2017-06-09 04:54:50 +02:00
Marek Marczykowski-Górecki
26f25aeecd
vm/mix/net: remove cleanup_vifs
With libvirt in place, this isn't enough - libvirt also keep VM
configuration in its memory and adjusting xenstore doesn't change that.
In fact changing xenstore behind it back make it even worse in some
situations.

QubesOS/qubes-issues#1426
2017-06-06 22:17:00 +02:00
Marek Marczykowski-Górecki
fa1da422d1
vm/adminvm: don't base AdminVM on QubesVM
dom0 isn't real VM and most properties doesn't apply to it. Lets make it
more explicit.
2017-06-06 21:57:27 +02:00
Marek Marczykowski-Górecki
67fcda2cde
tests: basic tests for AppVM
Specifically, check if root volume is updated after template switch.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
89e025ca3b
vm: make vm.get_power_state() return 'Halted' in offline_mode
Again, if libvirt or even Xen isn't running, we can safely assume VM
isn't too.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
313d7422e7
tests: remove obsolete tests, formatting 2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
04fd2ff34a
vm: adjust VM's root volume when changing its template
Re-init volume config of all 'snap_on_start' volumes at template
chanage. For this, save original volume config and re-use
config_volume_from_source function introduced in previous commit.

At the same time, forbid changing template of running AppVM or any
DispVM.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
63c23c835b
vm: reduce code duplication
Move inheriting volume from template to a helper function.

No functional change.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
28737d16ce
Migrate qubes.NotifyTools, qubes.FeaturesRequest, qubes.NotifyUpdates
Make them call into qubesd. Create separate socket for "misc" calls - VM
accessible, but not part of Admin API.
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
43fd1d76e8
tools/qubesd-query: add option to report failure with exit code 2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
05ba73bb3a
api/internal: add methods for handling host suspend 2017-06-06 03:33:29 +02:00
Marek Marczykowski-Górecki
f80fddf7ae
vm: restore code for suspending domains with PCI devices 2017-06-06 03:33:29 +02:00
Marek Marczykowski-Górecki
4c4f903fe3
vm: change kernel=None to kernel=''
vm.kernel property have type 'str'. Putting None there makes a lot of
troubles: it gets encoded as 'None' in qubes.xml and then loaded back as
'None' string, not None value. Also it isn't possible to assign None
value to str property throgh Admin API.

kernel='' is equally good to specify "no kernel from dom0".

QubesOS/qubes-issues#2622
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
d5cd58df51
templatevm: set default netvm to None for templates
Since we have qrexec-based updates proxy, we can even stronger isolate
templates from outside threats.

QubesOS/qubes-issues#1854
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
1ed0b14d93
vm: make vm.hvm=True by default
QubesOS/qubes-issues#2185
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
5d3204c730
vm: remove prepare_dvm parameter from methods and events
It was related to DispVM savefile preparation, but it is no longer
applicable in Qubes 4.0
2017-06-05 23:38:30 +02:00