Commit Graph

504 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
b5e07238a0
vm: always log startup fail 2017-11-07 03:10:40 +01:00
HW42
b907303a39 qubes/vm: Improve stopped event handling
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.

This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.

Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.

Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
2017-10-21 05:57:57 +02:00
Marek Marczykowski-Górecki
4500c4dcab
vm/adminvm: add is_halted() method
It is used in some places, for example attaching devices.

Fixes QubesOS/qubes-issues#3054
2017-10-21 04:05:17 +02:00
Marek Marczykowski-Górecki
9a9452ae3a
vm/net: fix handling netvm reset to default
If there was some netvm set, unset it first (same as with ordinary set).
Otherwise it will try to attach new netvm without detaching the old one
first.
2017-10-21 04:05:16 +02:00
Marek Marczykowski-Górecki
2164a8d7b8
Change license to LGPL v2.1+
See this thread for reasoning and acceptance from contributors:
https://groups.google.com/d/topic/qubes-devel/G7KzrfU0lWY/discussion
"Changing qubes-core-admin license to LGPL v2.1+"
2017-10-12 00:11:50 +02:00
HW42
67c06eb428 Don't allow attached PCI devices and virt_mode = 'pvh' 2017-10-11 21:10:43 +02:00
Marek Marczykowski-Górecki
08640f4779
Merge remote-tracking branch 'qubesos/pr/154'
* qubesos/pr/154:
  Add basic PVHv2 support
2017-10-09 00:08:35 +02:00
Marek Marczykowski-Górecki
bce7b64d4a
Fix calling qubes.SuspendPre/qubes.SuspendPost services
Those should be called as root, not default user.

Thanks @jpouellet for debugging this.
Fixes QubesOS/qubes-issues#3151
Fixes QubesOS/qubes-issues#3142
2017-10-07 02:36:02 +02:00
HW42
c583cf5190 Add basic PVHv2 support 2017-10-02 22:23:27 +02:00
Marek Marczykowski-Górecki
12b7e22d27
vm: do not start QubesDB watch instance multiple times
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
2017-09-29 12:05:23 +02:00
Marek Marczykowski-Górecki
b12fa13f06
vm: report storage.stop() errors to log
Catch exception there and log it. Otherwise asyncio complains about not
retrieved exception. There is no one else to handle this exception,
because shutdown event is triggered from libvirt, not any Admin API.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
97e3dced92
vm: move comment
Place comment describing self.app near self.app definition.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
a90dea34de
vm: add vm.klass property
Allow to get domain class as a property, not using admin.vm.List call.
This makes it unnecessary to call admin.vm.List on the client side to
construct wrapper object.
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
9f88fa7f0c
Move QubesVM.{name,qid,uuid,label} to BaseVM
Reduce strange code in BaseVM (accessing non-existing self.name) and
code duplication.
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
dc0e1a5481
vm: do not allow deleting template property from AppVM and DispVM
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.

Fixes QubesOS/qubes-issues#3115
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
efe600537e
vm/qubesvm: emit event on failed startup
If VM startup failed before starting anything (even in paused state),
there will be no further event, not even domain-shutdown. This makes it
hard for event-listening applications (like domains tray) to account
domain state. Fix this by emiting domain-start-failed event in every
case of failed startup after emiting domain-pre-start.

Related QubesOS/qubes-issues#3100
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
8c847faacc
vm/qubesvm: remove duplicated qmemman_client.close() 2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
e38e227503
vm/dispvm: cleanup DispVM also on failed startup
If dispvm.auto_cleanup is set, cleanup it also after failed startup
(like not enough memory).

Fixes QubesOS/qubes-issues#3045
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
7c6b04677f
vm/dispvm: fix error message
Fixes QubesOS/qubes-issues#3114
2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
e0cadfdd67
Merge remote-tracking branch 'qubesos/pr/150'
* qubesos/pr/150:
  qubes/tests: moar fixes
  test-packages: add missing libvirt classes
  qubes/tests: do not deadlock on .drain()
  qubes/vm: put name= first in __repr__
  tests: fix some memory leaks
  tests: complain about memory leaks
  tests: use one event loop and one libvirtaio impl
2017-09-22 20:54:11 +02:00
Wojtek Porczyk
71eefbea14 qubes/vm: put name= first in __repr__
This is to have it in case the repr is truncated.
2017-09-19 17:02:19 +02:00
Wojtek Porczyk
bd123a3812 tests: fix some memory leaks
This is work in progress. There are still more leaks.
2017-09-19 17:01:29 +02:00
Marek Marczykowski-Górecki
99bd193688
Rename 'dispvm_allowed' to 'template_for_dispvms'
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.

Fixes QubesOS/qubes-issues#3047
2017-09-05 02:07:26 +02:00
Marek Marczykowski-Górecki
c247ddff72
dispvm: fix setting up new DispVM
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).

Fixes QubesOS/qubes-issues#3057
2017-09-05 02:07:26 +02:00
Wojtek Porczyk
029ec9cb38 fix some more fd leaks 2017-08-31 20:30:24 +02:00
Wojtek Porczyk
de8ff20976 qubes: Allow for explicit closing of objects 2017-08-31 20:30:23 +02:00
Marek Marczykowski-Górecki
9f7915d998
Forbid creating VM with name 'none' or 'default'
Those were special names in Qubes 3.x, don't allow such VMs to avoid
(potentially fatal) confusion.

Fixes QubesOS/qubes-issues#3002
2017-08-14 02:24:32 +02:00
Marek Marczykowski-Górecki
8488d5dc72
vm/adminvm: add default_dispvm property to AdminVM
This is useful to select default DispVM template for VMs started
directly by the user. This makes sense as long as AdminVM == GUIVM.

QubesOS/qubes-issues#2974
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
691a6f4d8c
vm/dispvm: add auto_cleanup property, unify creating new DispVM
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.

New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).

Tests included.

QubesOS/qubes-issues#2974
2017-08-06 20:54:09 +02:00
Marek Marczykowski-Górecki
36f1a3abaf
Merge branch 'services'
* services:
  tests: check clockvm-related handlers
  doc: include list of extensions
  qubesvm: fix docstring
  ext/services: move exporting 'service.*' features to extensions
  app: update handling features/service os ClockVM
2017-07-29 05:09:32 +02:00
Marek Marczykowski-Górecki
76640df091
Merge branch 'tests-storage'
* tests-storage:
  tests: register libvirt events
  tests: even more agressive cleanup in tearDown
  app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
  api: keep track of established connections
  tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
  storage: fix Storage.clone and Storage.clone_volume
  tests: more tests fixes
  firewall: raise ValueError on invalid hostname in dsthost=
  qmemman: don't load qubes.xml
  tests: fix AdminVM test
  tests: create temporary files in /tmp
  tests: remove renaming test - it isn't supported anymore
  tests: various fixes for storage tests
  tests: fix removing LVM volumes
  tests: fix asyncio usage in some tests
  tests: minor fixes to api/admin tests
  storage/file: create -cow.img only when needed
  storage: move volume_config['source'] filling to one place
  app: do not create 'default' storage pool
  app: add missing setters for default_pool* global properties
2017-07-29 05:01:42 +02:00
Marek Marczykowski-Górecki
5321e11003
qubesvm: fix docstring 2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
1a6728cb12
ext/services: move exporting 'service.*' features to extensions
This means core code will not publish any features by default.
2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
c5667791e8
storage: move volume_config['source'] filling to one place
Don't set 'source' volume in various places (each VM class constructor
etc), do it as part of volume initialization. And when it needs to be
re-calculated, call storage.init_volume again.

This code was duplicated, and as usual in such a case, those copies
were different - one have set 'size', the other one not.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
3a21e1f1b3
app: do not create 'default' storage pool
Since we have app.default_pool* properties, create appropriately named
pool and let those properties choose the right pool. This also means we
don't need to specify pool name in default volume config anymore

QubesOS/qubes-issues#2256
2017-07-25 06:20:42 +02:00
Marek Marczykowski-Górecki
b7f0cf7d82
vm: add API for watching changes in QubesDB
Provide an API for use QubesDB.watch() inside of qubesd.

Fixes QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
d5b94d1cbd
vm: optimize imports
Remove unused, sort others.
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
1759bca00f
Rename vm.qdb to vm.untrusted_qdb
QubesDB can be freely modified by a VM, so one should take care when
reading any data retrieved from it.

Fixes QubesOS/qubes-issues#2934
2017-07-24 13:01:55 +02:00
Marek Marczykowski-Górecki
801d6acf5c
Rename 'net/*' features to 'net.*'
Same reasoning as with 'service/*' - '/' is not allowed in qrexec (so
Admin API) call argument.

Related to QubesOS/qubes-issues#2906
2017-07-18 03:50:02 +02:00
Marek Marczykowski-Górecki
4037bf9abc
vm/qubesvm: do not assign all host vCPUs to a qube
Fixes QubesOS/qubes-issues#2900
2017-07-17 12:33:52 +02:00
Marek Marczykowski-Górecki
11fac7be56
vm/qubesvm: minor fix 2017-07-17 12:29:25 +02:00
Marek Marczykowski-Górecki
9ba84ecdad
Add code and test for migrating vm.hvm into vm.virt_mode
This will allow to load old qubes.xml - otherwise previous change render
the system unusable (until manual qubes.xml edit).

QubesOS/qubes-issues#2912
2017-07-17 12:27:17 +02:00
Marek Marczykowski-Górecki
21940bef90
Change vm.hvm into vm.virt_mode
Fixes QubesOS/qubes-issues#2912
2017-07-17 12:26:10 +02:00
Marek Marczykowski-Górecki
ab7ed776f9
Merge branch 'dispvm-fix'
* dispvm-fix:
  storage: add comment about source volume lookup
  vm/dispvm: convert DispVM related function to coroutines
  vm/dispvm: fix DispVM storage definition
2017-07-14 01:13:20 +02:00
Marek Marczykowski-Górecki
a42e500108
Merge branch 'tests-fixes-1'
* tests-fixes-1:
  api: extract function to make pylint happy
  tests/vm: simplify AppVM storage test
  storage: do not use deepcopy on volume configs
  api: cleanup already started servers when some later failed
  tests: fix block devices tests when running on real system
  tests: fix some FD leaks
2017-07-14 01:05:21 +02:00
Marek Marczykowski-Górecki
f2f89c7b0c
storage: do not use deepcopy on volume configs
There may be Pool or Volume object references, which is intentional to
keep them as is - not copy whole Pool/Volume objects.
2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
c32f0db582
vm/dispvm: convert DispVM related function to coroutines
Some functions used there (create_on_disk, remove_from_disk, kill) are
coroutines, so callers needs to be too.

Fixes QubesOS/qubes-issues#2896
2017-07-08 03:04:55 +02:00
Marek Marczykowski-Górecki
148d1cda78
vm/dispvm: fix DispVM storage definition
Specify empty 'source' field, so it gets filled with appropriate
template's images. Then also fix recursive 'source' handling - DispVM
root volume should point at TemplateVM's root volume as a source, not a
AppVM's one - which is also only a snapshot.

Fixes QubesOS/qubes-issues#2896
2017-07-08 02:58:21 +02:00
Wojtek Porczyk
b04f612374 qubes: have "service" features' keys separated by period 2017-07-05 04:16:16 +02:00
Marek Marczykowski-Górecki
dddd94b339
Merge branch 'core3-storage3' 2017-07-04 16:44:53 +02:00
Marek Marczykowski-Górecki
6db39345fb
Merge remote-tracking branch 'origin/core3-quick-fixes-20170703' 2017-07-04 16:11:18 +02:00
Marek Marczykowski-Górecki
51022cada5
Merge remote-tracking branch 'origin/policy-adminvm' 2017-07-04 15:51:30 +02:00
Marek Marczykowski-Górecki
cfbccc0a74
vm: drop old is_outdated/commit_changes
This code is unused now. Theoretically this is_outdated implementation
should be moved to FileVolume, but since we don't have VM reference
there, it isn't possible to read appropriate xenstore entry. As we're
phasing out file pool, simply ignore it for now.

QubesOS/qubes-issues#2256
2017-07-04 14:25:10 +02:00
Marek Marczykowski-Górecki
0e554296e3
storage: drop 'internal' and 'removable' volume properties
Since dynamic volumes (qvm-block) are moved to devices API, those two
are not needed anymore.

QubesOS/qubes-issues#2256
2017-07-04 14:25:08 +02:00
Marek Marczykowski-Górecki
075d991114
config: eliminate duplicated qubes_base_dir
Remove it from system_path dict, have it in one place.
2017-07-04 14:25:07 +02:00
Marek Marczykowski-Górecki
b5ad245614
vm/qubesvm: have VM lifecycle related events async
Those functions are coroutines anyway, so allow event handlers to be
too.
Some of this (`domain-create-on-disk`, `domain-remove-from-disk`) will
be useful for appmenus handling.
2017-07-04 13:04:24 +02:00
Marek Marczykowski-Górecki
d00e4538bf
devices: make attach/detach related events async
This will allow starting processes and calling RPC services in those
events. This if required for usb devices, which are attached using RPC
services.
Intentionally keep device listing events synchronous only - to
discourage putting long-running actions there.

This change also require some not-async attach method version for
loading devices from qubes.xml - have `load_persistent` for this.
2017-07-04 13:04:23 +02:00
Marek Marczykowski-Górecki
6238254f49
events: merge fire_event and fire_event_pre functions
Those functions really do very similar things - lets merge them and add
simple parameter.
2017-07-04 13:04:21 +02:00
Marek Marczykowski-Górecki
3c4b60fa45
vm/adminvm: (re-)add vm.qdb property
Dom0 also expose some information in QubesDB - like list of block
devices.
2017-07-04 12:58:39 +02:00
Marek Marczykowski-Górecki
f393ce62dc
Fix docstrings formatting
Fix warnings/errors reported by Sphinx
2017-07-04 04:27:36 +02:00
Marek Marczykowski-Górecki
820539e909
storage: make volume snap_on_start/save_on_stop explicit
Always define those properties, always include them in volume config.
Also simplify overriding pool based on volume type defined by those:
override pool unless snap_on_start=True.

QubesOS/qubes-issues#2256
2017-07-04 04:11:14 +02:00
Marek Marczykowski-Górecki
dd1e05dc83
vm: drop rename related methods
Since VM name in immutable now, drop rename-related methods.

QubesOS/qubes-issues#2868
2017-07-04 04:11:13 +02:00
Marek Marczykowski-Górecki
3721ae3e8b
vm/adminvm: setup libvirt_domain property
Since libvirt do provide object for dom0 too, return it here.
It's much easier than special-casing AdminVM everywhere. And in fact
sometimes it is actually useful (for example attaching devices from/to
dom0, adjusting memory).
2017-07-04 04:09:27 +02:00
Marek Marczykowski-Górecki
3074a4064c
vm/qubesvm: forbid changing VM name
We've decided to make VM name immutable. This is especially important
for Admin API, where some parts (especially policy) are sticked to the
VM name.

Now, to rename the VM, one need to clone it under new name (thanks to
LVM, this is very quick action), then remove the old one.

Fixes QubesOS/qubes-issues#2868
2017-06-26 02:00:45 +02:00
Marek Marczykowski-Górecki
a6c16d00be
vm/qubesvm: fix handling empty kernel value
Do not fail on empty kernel value - which means "use booloader on
root.img".

Fixes 3ddc052 "vm: move kernel presence validation to event handler"
2017-06-26 01:59:39 +02:00
Marek Marczykowski-Górecki
4208a98bd7
Merge branch 'core3-devel-20170619' 2017-06-23 02:53:17 +02:00
Marek Marczykowski-Górecki
3ddc052af3
vm: move kernel presence validation to event handler
Setter is called also on qubes.xml load, so missing kernel breaks
qubes.xml parsing - for example qubesd startup to fix that property.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
60443259d0
vm: raise CalledProcessError on failed service/command call 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
e4d285d479
vm/adminvm: make AdminVM sortable
One more thing gone during changing AdminVM base class.
2017-06-20 12:47:00 +02:00
Marek Marczykowski-Górecki
9242202db2
admin: implement admin.vm.tag.*
QubesOS/qubes-issues#2622
2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
5209bc370d
vm: init vm.storage and vm.volumes in BaseVM
This way also AdminVM will have (empty) properties there. It is much
cleaner than adding `if hasattr` or catching AttributeError everywhere.
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
160ab964bc
vm: improve sending events for tags and features
1. Send the event after setting tag/feature
2. Provide old value for feature
3. Rename 'key' kwarg to 'feature'
2017-06-14 10:44:23 +02:00
Marek Marczykowski-Górecki
93ccb8bbda
Merge remote-tracking branch 'qubesos/pr/111'
* qubesos/pr/111:
  vm: drop 'internal' property
  qmemman: make sure to release lock
  qmemman: fix meminfo parsing for python 3
  devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
2017-06-12 23:22:55 +02:00
Marek Marczykowski-Górecki
ba4da5cbe0
Merge remote-tracking branch 'qubesos/pr/110'
* qubesos/pr/110:
  storage: use direct object references, not only identifiers
  vm: fix volume_config
  storage/lvm: prefix VM LVM volumes with 'vm-'
  storage: fix VM rename
2017-06-12 23:22:41 +02:00
Marek Marczykowski-Górecki
37245acdcf
vm: change vm.updates_available to a 'updates-available' feature 2017-06-12 12:34:23 +02:00
Marek Marczykowski-Górecki
caa03a9279
vm/qubesvm: simplify 'None' kernel handling 2017-06-12 12:26:57 +02:00
Marek Marczykowski-Górecki
e8c303977b
vm/adminvm: fix str(AdminVM) after changing its base
BaseVM have no (useful) __str__ method.
2017-06-12 10:15:13 +02:00
Marek Marczykowski-Górecki
9d99232515
vm: drop 'internal' property
It isn't used anywhere in the code right now. And when it will be
needed, it should be a "feature" not "property".
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
0f12870803
storage: use direct object references, not only identifiers
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object

Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.

One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.

Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
80b459b6d5
vm: fix volume_config
- kernel volume shouldn't have snap_on_start, it's read-only volume
  anyway
- root volume of AppVM should have placeholder for 'source'
- private volume of AppVM should _not_ have placeholder for 'source'
(it's ignored anyway, because snap_on_start=False)

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
26f25aeecd
vm/mix/net: remove cleanup_vifs
With libvirt in place, this isn't enough - libvirt also keep VM
configuration in its memory and adjusting xenstore doesn't change that.
In fact changing xenstore behind it back make it even worse in some
situations.

QubesOS/qubes-issues#1426
2017-06-06 22:17:00 +02:00
Marek Marczykowski-Górecki
fa1da422d1
vm/adminvm: don't base AdminVM on QubesVM
dom0 isn't real VM and most properties doesn't apply to it. Lets make it
more explicit.
2017-06-06 21:57:27 +02:00
Marek Marczykowski-Górecki
89e025ca3b
vm: make vm.get_power_state() return 'Halted' in offline_mode
Again, if libvirt or even Xen isn't running, we can safely assume VM
isn't too.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
04fd2ff34a
vm: adjust VM's root volume when changing its template
Re-init volume config of all 'snap_on_start' volumes at template
chanage. For this, save original volume config and re-use
config_volume_from_source function introduced in previous commit.

At the same time, forbid changing template of running AppVM or any
DispVM.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
63c23c835b
vm: reduce code duplication
Move inheriting volume from template to a helper function.

No functional change.

QubesOS/qubes-issues#2256
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
28737d16ce
Migrate qubes.NotifyTools, qubes.FeaturesRequest, qubes.NotifyUpdates
Make them call into qubesd. Create separate socket for "misc" calls - VM
accessible, but not part of Admin API.
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
f80fddf7ae
vm: restore code for suspending domains with PCI devices 2017-06-06 03:33:29 +02:00
Marek Marczykowski-Górecki
4c4f903fe3
vm: change kernel=None to kernel=''
vm.kernel property have type 'str'. Putting None there makes a lot of
troubles: it gets encoded as 'None' in qubes.xml and then loaded back as
'None' string, not None value. Also it isn't possible to assign None
value to str property throgh Admin API.

kernel='' is equally good to specify "no kernel from dom0".

QubesOS/qubes-issues#2622
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
d5cd58df51
templatevm: set default netvm to None for templates
Since we have qrexec-based updates proxy, we can even stronger isolate
templates from outside threats.

QubesOS/qubes-issues#1854
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
1ed0b14d93
vm: make vm.hvm=True by default
QubesOS/qubes-issues#2185
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
5d3204c730
vm: remove prepare_dvm parameter from methods and events
It was related to DispVM savefile preparation, but it is no longer
applicable in Qubes 4.0
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
018877a19c
vm: avoid starting the same VM multiple times simultaneously
While libvirt handle locking itself, there is also Qubes-specific
startup part. Especially starting qrexec-daemon and waiting until
qrexec-agent connect to it. When someone will attempt to start VM the
second time (or simply assume it's already running) - qrexec will not be
connected yet and the operation will fail. Solve the problem by wrapping
the whole vm.start() function with a lock, including a check if VM is
running and waiting for qrexec.

Also, do not throw exception if VM is already running.

This way, after a call to vm.start(), VM will be started with qrexec
connected - regardless of who really started it.
Note that, it will not solve the situation when someone check if VM is
running manually, like:

    if not vm.is_running():
        yield from vm.start()

Such code should be changed to simply:

    yield from vm.start()

Fixes QubesOS/qubes-issues#2001
Fixes QubesOS/qubes-issues#2666
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
b5a4573290
storage: remove DomainPool
Replaced by BlockDevice extension
2017-06-05 23:33:58 +02:00
Wojtek Porczyk
bbe757d0a7 Make pylint very ♥ 2017-05-30 15:06:05 +02:00
Wojtek Porczyk
df03800278 Fix miscellaneous warnings
Among them:
- explicitly close files (possibly using with: syntax)
- use non-deprecated methods
2017-05-29 17:34:15 +02:00
Wojtek Porczyk
b212a75ba3 qubes/vm/qubesvm: don't use .run_service() for .run()
QubesOS/qubes-issues#2622
2017-05-29 17:34:15 +02:00
Wojtek Porczyk
a9755ed76a qubes/ext/gui: remove is-fully-usable and is_guid_running 2017-05-29 17:34:15 +02:00
Marek Marczykowski-Górecki
e54cc11a2c
vm: expose to VM only features with 'service/' prefix
And place them in /qubes-service/ QubesDB directory. This allows
extensions to easily store some data not exposed to VM, but also have
control what VM will see. And at the same time, it make it compatible
with existing services framework

QubesOS/qubes-issues#1637
2017-05-26 15:08:19 +02:00
Marek Marczykowski-Górecki
607dcbaf37
admin-api: fix handling admin.vm.property.Set with None VM value
Setting VMProperty to None VM should be encoded as '' value (according
to VMProperty._none_value). But value validation rejected this value.

QubesOS/qubes-issues#2622
2017-05-26 15:08:18 +02:00
Marek Marczykowski-Górecki
1692601fcd
typo fix 2017-05-23 15:35:21 +02:00
Marek Marczykowski-Górecki
f93583e2be
devices: adjust XML serialization of device options
Use '<option name="option_name">option_value</option>' instead of
'<options option_name="option_value"/>'. It's more consistent with the
rest of qubes.xml - have one thing per element.

Also, add options deserialization test.
2017-05-22 17:11:20 +02:00
Marek Marczykowski-Górecki
29f3c9b58f
vm: don't try to define libvirt domain when just checking its state
When libvirt domain is not defined, it isn't running for sure.
This commit fixes the case when vm.is_running() appears anywhere in the
code used during libvirt xml building. In this case, it's mostly about
PCI device description for libvirt.
2017-05-22 17:11:19 +02:00
Marek Marczykowski-Górecki
dde9085920
vm: fix autostart-related events handling
event signature contains 'name', not 'prop' (it's no longer positional
argument).
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
ae28d32af7
vm: make sure qmemman connection is closed even on failed VM startup
Qmemman hold global lock while serving requests, so not closing
connection will prevent any further action.
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
e5daf902b3
vm/net: fix IP address calculation
7 is not the same as 7 bits...
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
0b64e3fbe0
storage: make LinuxKernel pool track vm kernel version
Do not initialize it only at qubes.xml load time, but re-read vm.kernel
property each time the path is constructed. While at it, add support for
vm.kernel set to 'None' - simply don't include modules.img (xvdd) then.
2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
595d983659
storage: make verify() asyncio aware 2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
680dbf432c
vm: Features fixes
clear: dict should not be mutated during interation.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
ba86d6da79
vm: implement tag manager to fire events on change
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).

Fixes QubesOS/qubes-issues#2686
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
3c7d2289ce
vm: stubdom_xid: use canonical method for checking VM state
Don't depend on xid==-1 for not running VM.
This breaks some tests, as xid can be a mock object (not comparable to
int).
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
d4353c2a30
Enable linux-stubdom by default
Also, make it possible to set default on a template for its VMs.

QubesOS/qubes-issues#2185
2017-05-16 09:01:40 +02:00
HW42
f7094bec4f
add support for linux stubdoms
(cherry picked from commit e14ac1500a05162b3c07a5f44b4f0223f43fe90c)
2017-05-15 14:43:30 +02:00
Marek Marczykowski-Górecki
b91b89a341
Make pylint ♥ 2017-05-12 17:41:38 +02:00
Marek Marczykowski-Górecki
0d84ee1c24
make pylint happy 2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
ef031d6f0d
qubesvm: make vm.create_on_disk cleanup after failed attempt
Don't leave half-created VM - it will make harder to retry.
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
2a25db7eb7
qubesvm: fix stdio handling in vm.run_service_for_stdio
Do set stdin/stdout/stderr descriptors as PIPE. Otherwise would not be
redirected.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
838553baf8
qubesvm: move storage.stop() to 'domain-shutdown' event handler
QubesOS/qubes-issues#2256
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
0dfcaa63c9
Handle libvirt events in qubesd
This is required to get shutdown notification, when it wasn't initiated
by qubesd (for example 'poweroff' command inside of VM).
Libvirt event loop implementation must be registered before making
connection to libvirt, so move it to the beginning of main().
For now, only 'domain-shutdown' event is emited.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
3e067a3ef5
vm: run_service(): require user session running only when need gui access
User session may not be started at all (for example no qubes packages
installed there), so don't block it in all the cases. Also this would
prevent running 'qubes.WaitForSession' service...

In practice, default value for 'gui' argument is False, so in most cases
user session will be ignored. Which doesn't matter in most cases -
especially for services called by qubesd.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
68d5ca93e1
storage: have QubesVM() handle vm.dir_path, also on clone and rename
Keep it uniform - QubesVM() object is responsible for handling
vm.dir_path, Storage() is responsible for handling disk volumes (which
may live in that directory

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
52c3753d61
storage: support asynchronous storage pool implementations
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.

This commit does not touch any pool implementation - all of them are
still synchronous.

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
ca51e8a02a
vm: remove unused metaclass 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
8c3c65f40f
vm/net: move 'netvm' value validation to setter 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
82983cc1d8
vm: minor fix in docstring 2017-05-12 14:25:30 +02:00
Marek Marczykowski-Górecki
361ecdd6ac
vm: add stubdom_xid property
Some parts (like GUI) needs to communicate with stubdomain. Lets expose
it's XID at the API level.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
acaada3fd9
vm: fix placeholder methods signatures in Features class
Some methods inherited from dict (pop and setdefault here) are covered
by placeholders raising NotImplementedError. Lets fix their signatures
(to match those of dict) to really get NotImplementedError, instead of
TypeError.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
99be78646f
Add stateless_property decorator for properties exposed through mgmt API
Management API gives access only to qubes.property. And this is
actually a good thing, so instead of extending it to access also
builtins.property, add a simple decorator to define read-only, stateless
qubes.property.

QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
2aa0de3d5b
tools: remove qvm-ls tool and related integration in qubes.property
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.

The tool itself is already moved to core-mgmt-client repository.

QubesOS/qubes-issues#853
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
68c8d219c9
vm/qubesvm: fix daemon startup function
'command' argument is a tuple
2017-05-12 14:25:25 +02:00
Marek Marczykowski-Górecki
50b812190b
Merge remote-tracking branch 'kalkin/device-assignments' into core3-devel
* kalkin/device-assignments: (21 commits)
  PCI extension cache PCIDevice objects
  Make pylint ♥
  Fix pylint warning no-else-return
  Fix pylint warning len-as-conditional
  device-list-attached event returns a dev/options tupples list
  DeviceAssignment options are now a dict
  Remove WrongAssignment exception
  Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
  Update relaxng devices option element
  Fix tests broken by the new assignment api
  Fix qubes.tests.devices
  Fix pci device integration tests
  qvm-device add support for assignments
  Update ext/pci to new api
  BaseVM add DeviceAssignment xml serialization
  Migrate DeviceCollection to new API
  Add PersistentCollection helper to qubes.devices
  Add DeviceAssignment
  qvm-device validates device parameters
  qvm-device fix handling of non block devices
  ...
2017-04-21 17:09:33 +02:00
Marek Marczykowski-Górecki
2705e223bd
Merge branch 'core3-policy' into core3-devel
* core3-policy:
  Make pylint happy
  tests: disable GTK tests on travis
  qubespolicy: make pylint happy
  qubespolicy: run GUI code inside user session and expose it as dbus object
  tests: plug rpc-window tests into main test runner
  qubespolicy: plug GUI code into qrexec-policy tool
  rpm: add rpc-window related files to package
  rpc-window: adjust for qubespolicy API
  rpc-window: use pkg_resources for glade file
  rpc-window: use 'edit-find' icon if no other is found
  rpc-window: adjust for python3
  rpc-window: code style adjustments
  Import new rpc confirmation window code
  qubesd: add second socket for in-dom0 internal calls
  policy: qrexec-policy cli tool
  tests: qubespolicy tests
  qubespolicy: initial version for core3
  vm/appvm: add dispvm_allowed property
  dispvm: don't load separate Qubes() instance when handling DispVM
2017-04-21 17:01:56 +02:00
Marek Marczykowski-Górecki
fa72679b47
Make pylint happy
New pylint throw some more warnings.
2017-04-21 16:57:35 +02:00
Bahtiar `kalkin-` Gadimov
79407a8717
Make pylint ♥ 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
5bc7a8f9e3
Fix pylint warning no-else-return 2017-04-15 23:49:41 +02:00
Bahtiar `kalkin-` Gadimov
e446e7a2f4
Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:36 +02:00
Bahtiar `kalkin-` Gadimov
2a6266887e
BaseVM add DeviceAssignment xml serialization
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
990cfd8ab9
Migrate DeviceCollection to new API
- Use PersistentCollection as _set()
- attach/detach expect DeviceAssignment as parater
- attached(persistent=True) is now persistent()
- attached() returns all attached devices
- assigned() returns all attached device assignments

`#	modified:   templates/libvirt/xen.xml

Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
23c68c5458
Add PersistentCollection helper to qubes.devices 2017-04-15 18:05:27 +02:00
Marek Marczykowski-Górecki
96db9a46d1 Fix start-related asyncio integration
cherry-picked commit 05fc3a4
2017-04-07 17:46:54 +02:00
Marek Marczykowski-Górecki
1c836531c7
vm/appvm: add dispvm_allowed property
Speciffy whether DispVM can be created from this AppVM

Fixes QubesOS/qubes-issues#2075
2017-04-06 15:43:01 +02:00
Marek Marczykowski-Górecki
e4e92b06af
dispvm: don't load separate Qubes() instance when handling DispVM
This will not work in qubesd, where there is one and only Qubes()
instance.
2017-04-06 15:43:01 +02:00
Wojtek Porczyk
3bc4ec1e98 add mock libvirtaio module + pylint fixes 2017-04-05 17:06:45 +02:00
Wojtek Porczyk
64d358562b qubes: revert async def, use @asyncio.coroutine
Current pylint (any released version) stumbles on async def'ined
functions. Let's use @asyncio.coroutines for now.

Seems like python-3.5 is not that mature yet.

QubesOS/qubes-issues#2622
QubesOS/qubes-issues#2738
PyCQA/pylint#1126
2017-04-05 17:05:52 +02:00
Wojtek Porczyk
78693c265c qubes/vm/qubesvm: rework QubesVM.run*() methods
0) All those methods are now awaitable rather than synchronous.

1) The base method is run_service(). The method run() was rewritten
   using run_service('qubes.VMShell', input=...). There is no provision
   for running plain commands.

2) Get rid of passio*= arguments. If you'd like to get another return
   value, use another method. It's as simple as that.
   See:
      - run_service_for_stdio()
      - run_for_stdio()

   Also gone are wait= and localcmd= arguments. They are of no use
   inside qubesd.

3) The qvm-run tool and tests are left behind for now and will be fixed
   later. This is because they also need event loop, which is not
   implemented yet.

fixes QubesOS/qubes-issues#1900
QubesOS/qubes-issues#2622
2017-04-05 17:05:22 +02:00
Wojtek Porczyk
c437f40284 qubes/mgmt: lifecycle and class listing
- mgmt.vmclass.List
- mgmt.vm.Start
- mgmt.vm.Shutdown
- mgmt.vm.Pause
- mgmt.vm.Unpause
- mgmt.vm.Kill

QubesOS/qubes-issues#2622
2017-04-03 21:55:01 +02:00
Wojtek Porczyk
1b9479837a Merge remote-tracking branch 'marmarek/core3-mgmt-methods1' into core3-devel
Conflicts:
	qubes/mgmt.py
2017-04-01 18:33:54 +02:00
Marek Marczykowski-Górecki
3388054eae
mgmt: move property value sanitization to property definition
This also means we don't check if a VM with given name (in case of
VMProperty) exists in the system, at this stage. But this is ok, lets
not duplicate work of property setter.

QubesOS/qubes-issues#2622
2017-03-31 11:55:24 +02:00
Wojtek Porczyk
cce809c2cb qubes/vm/qubesvm: async def start
QubesOS/qubes-issues#2622
2017-03-30 23:04:55 +02:00
Marek Marczykowski-Górecki
ce3bedbf2c
vm: move validate_name to qubes/vm
This will be needed by VMProperty class in the next commit.

QubesOS/qubes-issues#2622
2017-03-28 21:04:05 +02:00
Marek Marczykowski-Górecki
35d1167893
qubes/vm/net: fix name of argument for property-del event handler
Since enforcing keyword arguments for event handlers, it's important now.
2017-03-16 20:04:05 +01:00