Commit Graph

1029 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
402afa1925
pci: use device attach options for disabling strict reset
Since we have now per-device options, it's more logical to use it here,
instead of features with device identifier encoded into feature name.
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
d0b8933374
devices: add assignment.device property
Make it easy to retrieve DeviceInfo object out of DeviceAssignment
object. The only missing piece of information for that is device class,
so add it. Make it optional, as it can be filled on demand when passing
the object through DeviceCollection (either by listing devices, or
attaching/detaching).

This is mostly to ease handling options in libvirt template - to get
them, you need to use `assignments()`, istead of `persistent()` or
`attached()`, but there were no _simple_ way of getting actual device
object.

This also makes DeviceCollection._device method not needed anymore.
2017-05-22 03:21:07 +02:00
Marek Marczykowski-Górecki
227010d433
pci: fix device-pre-attach event signature
Now it contains also options. PCI extension do not use it, yet.
2017-05-19 18:49:22 +02:00
Marek Marczykowski-Górecki
f9fa27dacc
property: add property.__str__ 2017-05-19 18:49:21 +02:00
Marek Marczykowski-Górecki
f7cb8f27c5
admin: add admin.vm.Remove method
QubesOS/qubes-issues#2622
2017-05-19 18:49:21 +02:00
Marek Marczykowski-Górecki
0ef010cc70
qmemman: fix list/filter handling on python3
Filter in python3 returns a generator, can be iterated only once.
This is about list of existing domains - store it as a list, otherwise
domains will "disappear" after being discovered.
2017-05-18 09:44:48 +02:00
Marek Marczykowski-Górecki
1831088c9b
api/internal: don't try to start dom0
qrexec-policy do try to start a VM before executing service. Avoid this
here, instead of qrexec-policy to keep the later clean.
2017-05-17 14:24:59 +02:00
Marek Marczykowski-Górecki
ae28d32af7
vm: make sure qmemman connection is closed even on failed VM startup
Qmemman hold global lock while serving requests, so not closing
connection will prevent any further action.
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
e5daf902b3
vm/net: fix IP address calculation
7 is not the same as 7 bits...
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
4d6d6c913a
qubesd: make sure NOTIFY_SOCKET variable is not propagated
qubesd do start other daemons - make sure they will not try to signal
systemd about it. In some cases such daemons (qubesdb-daemon) behave
differently based on this variable.
2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
dc2563271c
tests: very basic tests for LinuxKernel pool 2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
0b64e3fbe0
storage: make LinuxKernel pool track vm kernel version
Do not initialize it only at qubes.xml load time, but re-read vm.kernel
property each time the path is constructed. While at it, add support for
vm.kernel set to 'None' - simply don't include modules.img (xvdd) then.
2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
595d983659
storage: make verify() asyncio aware 2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
d57aef96e5
admin.vm.Start: report libvirt errors to caller
Convert them to QubesException for that.
2017-05-17 11:17:21 +02:00
Marek Marczykowski-Górecki
774f4c8d49
tests: admin.vm.devices.* Admin API
QubesOS/qubes-issues#2622
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
661752bbc1
devices: list persistent devices also when not currently attached
When device extension do not return some "persistent" device as
currently attached, still return it, as it will be attached at next
domain startup. User can distinguish such devices by having
frontend_domain=None (or other VM).
Also, return a set from DeviceCollection.assignments().
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
2c0d79af4a
devices: pass attach options to device handling extension
Otherwise options would be ignored...
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
75ac611c90
admin: admin.vm.device.* methods
QubesOS/qubes-issues#2622
2017-05-16 09:05:13 +02:00
Marek Marczykowski-Górecki
f3c7fb8129
admin: add missing parts to admin.vm.Clone
Clone all the things.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
8533ec9773
devices: add clone method
Copy DeviceAssignment object
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
b4ad5f0149
firewall: add clone and comparing methods, missing docstrings 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
8de82f390e
tests: Tags() and Features() 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
680dbf432c
vm: Features fixes
clear: dict should not be mutated during interation.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
ba86d6da79
vm: implement tag manager to fire events on change
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).

Fixes QubesOS/qubes-issues#2686
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
68a426f0ba
admin: add methods for global properties
QubesOS/qubes-issues#2622
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
0160d6e2f0
admin: fix handling default template in admin.vm.Create.* methods 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
fcfb2e004a
tests: more for admin.Events method 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
3c7d2289ce
vm: stubdom_xid: use canonical method for checking VM state
Don't depend on xid==-1 for not running VM.
This breaks some tests, as xid can be a mock object (not comparable to
int).
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
e598735c3b
tests: minor formatting 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
d4353c2a30
Enable linux-stubdom by default
Also, make it possible to set default on a template for its VMs.

QubesOS/qubes-issues#2185
2017-05-16 09:01:40 +02:00
HW42
f7094bec4f
add support for linux stubdoms
(cherry picked from commit e14ac1500a05162b3c07a5f44b4f0223f43fe90c)
2017-05-15 14:43:30 +02:00
Marek Marczykowski-Górecki
625c03c504
Rename MgmtAPI to AdminAPI - part 3: Admin API
QubesOS/qubes-issues#853
2017-05-12 19:28:08 +02:00
Marek Marczykowski-Górecki
258d268a3a
Rename MgmtAPI to AdminAPI - part 2: internal API
QubesOS/qubes-issues#853
2017-05-12 19:28:08 +02:00
Marek Marczykowski-Górecki
cd489f46e1
Rename MgmtAPI to AdminAPI - part 1: classes
QubesOS/qubes-issues#853
2017-05-12 19:28:07 +02:00
Marek Marczykowski-Górecki
81162a699b
Remove unused helper scripts 2017-05-12 18:43:35 +02:00
Marek Marczykowski-Górecki
83eef56f9d
Remove tools that are moved to -client repository
QubesOS/qubes-issues#853
2017-05-12 18:43:35 +02:00
Marek Marczykowski-Górecki
bb4dc91ee8
storage/lvm: add workaround for ancient lvm2 in Travis-CI environment 2017-05-12 18:16:30 +02:00
Marek Marczykowski-Górecki
2a4af124f8
qubes/app: don't fail on saving qubes.xml when 'qubes' group isn't there 2017-05-12 18:11:27 +02:00
Marek Marczykowski-Górecki
da3eed582b
tests: rearrage LVM thin tests to run at least some in devel env
This requires creating LVM volume group, so create on based on loop dev
in /tmp.
This is rather rough, but if any of this fails, run the tests anyway -
it will simply skip LVM tests.
2017-05-12 18:10:17 +02:00
Marek Marczykowski-Górecki
b91b89a341
Make pylint ♥ 2017-05-12 17:41:38 +02:00
Marek Marczykowski-Górecki
0d84ee1c24
make pylint happy 2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
7ef020d5fe
devices: fix 'NotImplemented' usage
'__eq__' function should return 'NotImplemented' _value_, not
'NotImplementedError' exception.
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
4f0d72f99a
devices: adjust API documentation
Device ident may contain only characters allowed in qrexec argument.
This will allow using it directly in qrexec argument in Attach/Detach
methods.
This also means PCI extension will need to be updated (it uses ':' in
ident).

QubesOS/qubes-issues#853
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
ef031d6f0d
qubesvm: make vm.create_on_disk cleanup after failed attempt
Don't leave half-created VM - it will make harder to retry.
2017-05-12 17:04:20 +02:00
Marek Marczykowski-Górecki
2a25db7eb7
qubesvm: fix stdio handling in vm.run_service_for_stdio
Do set stdin/stdout/stderr descriptors as PIPE. Otherwise would not be
redirected.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
838553baf8
qubesvm: move storage.stop() to 'domain-shutdown' event handler
QubesOS/qubes-issues#2256
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
0dfcaa63c9
Handle libvirt events in qubesd
This is required to get shutdown notification, when it wasn't initiated
by qubesd (for example 'poweroff' command inside of VM).
Libvirt event loop implementation must be registered before making
connection to libvirt, so move it to the beginning of main().
For now, only 'domain-shutdown' event is emited.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
3e067a3ef5
vm: run_service(): require user session running only when need gui access
User session may not be started at all (for example no qubes packages
installed there), so don't block it in all the cases. Also this would
prevent running 'qubes.WaitForSession' service...

In practice, default value for 'gui' argument is False, so in most cases
user session will be ignored. Which doesn't matter in most cases -
especially for services called by qubesd.
2017-05-12 17:04:19 +02:00
Marek Marczykowski-Górecki
68d5ca93e1
storage: have QubesVM() handle vm.dir_path, also on clone and rename
Keep it uniform - QubesVM() object is responsible for handling
vm.dir_path, Storage() is responsible for handling disk volumes (which
may live in that directory

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
53be79b3b7
mgmt: vm.Create* and vm.Clone
Create methods are dynamically created based on available VM classes.

QubesOS/qubes-issues#2622
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
52c3753d61
storage: support asynchronous storage pool implementations
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.

This commit does not touch any pool implementation - all of them are
still synchronous.

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
f324d30d4e
quebs: make Label() and storage.Pool() objects hashable 2017-05-12 14:27:47 +02:00
Marek Marczykowski-Górecki
ca51e8a02a
vm: remove unused metaclass 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
8c3c65f40f
vm/net: move 'netvm' value validation to setter 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
3bc8facc0b
mgmt: rework mgmt.Events to use per-instance handlers
QubesOS/qubes-issues#2622
2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
da7496794a
events: add support for per-instance handlers 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
bd1f84fcec
events: clarify docstrings about add_handler/remove_handler
https://github.com/QubesOS/qubes-core-admin/pull/100#discussion_r115745098
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
cddc507788
mgmt: factor out filtering by mgmt-permission:... events
Split actual filtering done by mgmt-permission: events into calling an
event and applying returned filters. This way filtering done in
mgmt.Events handler could reuse the same function.
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
6926725e02
Fixes for devices API
See comments in #98
2017-05-12 14:25:31 +02:00
Marek Marczykowski-Górecki
68c3d7309a
tools: remove qubes_monitor_layout_notify tool - moved to mgmt-client 2017-05-12 14:25:30 +02:00
Marek Marczykowski-Górecki
82983cc1d8
vm: minor fix in docstring 2017-05-12 14:25:30 +02:00
Marek Marczykowski-Górecki
033d2132d3
ext/gui: remove most of it, moved to qvm-start-gui tool...
...in core-mgmt-client repository. qubesd isn't the right place to start
GUI applications, which will be even more important when GUI domain will
be something different than Dom0.

QubesOS/qubes-issues#833
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
027df9567c
mgmt: add mgmt.label.Index method
QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
278243d7dc
tools/qubesd: fix sending exception response
QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
55037e0446
tests: tools.qubesd
For now only QubesDaemonProtocol test. main function still untested

QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
0b737b3a55
mgmt: vm.features handling (mgmt.vm.feature.*)
Actual implementation and tests.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
361ecdd6ac
vm: add stubdom_xid property
Some parts (like GUI) needs to communicate with stubdomain. Lets expose
it's XID at the API level.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
acaada3fd9
vm: fix placeholder methods signatures in Features class
Some methods inherited from dict (pop and setdefault here) are covered
by placeholders raising NotImplementedError. Lets fix their signatures
(to match those of dict) to really get NotImplementedError, instead of
TypeError.
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
643cef1e65
mgmt: send artificial 'connection-established' event just after connection
This allows avoid race condition between  registering event handlers and
performing some action. The important thing is the event sent after
registering event handlers in qubesd. This means state changes (like
VM start/stop) after 'connection-established' event will be included in
event stream.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
8cc0af1eda
tests: mgmt.Events
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
73df1b8ea7
tests: VM lifecycle methods tests
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
99be78646f
Add stateless_property decorator for properties exposed through mgmt API
Management API gives access only to qubes.property. And this is
actually a good thing, so instead of extending it to access also
builtins.property, add a simple decorator to define read-only, stateless
qubes.property.

QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
2aa0de3d5b
tools: remove qvm-ls tool and related integration in qubes.property
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.

The tool itself is already moved to core-mgmt-client repository.

QubesOS/qubes-issues#853
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
654e64c4a6
mgmt: implement mgmt.Events
Pass (almost) all the events. Some better filtering probably will be
needed.

QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
7e4bc4249d
tools/qubesd-query: handle events-based methods
Pass the data as it arrives, instead of buffering it all.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
40a86efd66
mgmt: add support for events-sending methods
Standard methods return only one value, after operation is completed,
but events-related methods may return multiple values during the method
execution time. Provide a callback for such cases.
Also, according to specification, avoid sending both event and non-event
values.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
bd9f1d2e7c
mgmt: add support for cancellable operations
Allow method handler to decide if operation could be cancelled. If yes,
when connection to the qubesd is terminated (and
protocol.connection_lost get called) the operation is cancelled using
standard asyncio method - in which case asyncio.CancelledError is thrown
inside method handler. This needs to be explicitly enabled, because
cancellable methods are much harder to write, to maintain consistent
system state.
Caveat: protocol.connection_lost is called only when trying to send some
data to it (and it's already terminated). Which makes this whole
mechanism useful only for events. Otherwise, when sending some data (and
possibly detecting that connection is broken), the operation is already
completed.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
ee967c1e72
events: add function to remove event handlers
Event handlers for remote mgmt API will be registered dynamically, so
allow to unregister them.

QubesOS/qubes-issues#2622
2017-05-12 14:25:26 +02:00
Marek Marczykowski-Górecki
68c8d219c9
vm/qubesvm: fix daemon startup function
'command' argument is a tuple
2017-05-12 14:25:25 +02:00
Marek Marczykowski-Górecki
6383d87be9
qmemmand: fix systemd notification 2017-05-12 14:25:25 +02:00
Marek Marczykowski-Górecki
d75fc6c28b
backup: add a little more usage info, include docs in main index 2017-04-21 19:58:32 +02:00
Marek Marczykowski-Górecki
50b812190b
Merge remote-tracking branch 'kalkin/device-assignments' into core3-devel
* kalkin/device-assignments: (21 commits)
  PCI extension cache PCIDevice objects
  Make pylint ♥
  Fix pylint warning no-else-return
  Fix pylint warning len-as-conditional
  device-list-attached event returns a dev/options tupples list
  DeviceAssignment options are now a dict
  Remove WrongAssignment exception
  Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
  Update relaxng devices option element
  Fix tests broken by the new assignment api
  Fix qubes.tests.devices
  Fix pci device integration tests
  qvm-device add support for assignments
  Update ext/pci to new api
  BaseVM add DeviceAssignment xml serialization
  Migrate DeviceCollection to new API
  Add PersistentCollection helper to qubes.devices
  Add DeviceAssignment
  qvm-device validates device parameters
  qvm-device fix handling of non block devices
  ...
2017-04-21 17:09:33 +02:00
Marek Marczykowski-Górecki
2705e223bd
Merge branch 'core3-policy' into core3-devel
* core3-policy:
  Make pylint happy
  tests: disable GTK tests on travis
  qubespolicy: make pylint happy
  qubespolicy: run GUI code inside user session and expose it as dbus object
  tests: plug rpc-window tests into main test runner
  qubespolicy: plug GUI code into qrexec-policy tool
  rpm: add rpc-window related files to package
  rpc-window: adjust for qubespolicy API
  rpc-window: use pkg_resources for glade file
  rpc-window: use 'edit-find' icon if no other is found
  rpc-window: adjust for python3
  rpc-window: code style adjustments
  Import new rpc confirmation window code
  qubesd: add second socket for in-dom0 internal calls
  policy: qrexec-policy cli tool
  tests: qubespolicy tests
  qubespolicy: initial version for core3
  vm/appvm: add dispvm_allowed property
  dispvm: don't load separate Qubes() instance when handling DispVM
2017-04-21 17:01:56 +02:00
Marek Marczykowski-Górecki
fa72679b47
Make pylint happy
New pylint throw some more warnings.
2017-04-21 16:57:35 +02:00
Bahtiar `kalkin-` Gadimov
8d60f533c3
PCI extension cache PCIDevice objects 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
79407a8717
Make pylint ♥ 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
5bc7a8f9e3
Fix pylint warning no-else-return 2017-04-15 23:49:41 +02:00
Bahtiar `kalkin-` Gadimov
eb217e0d27
Fix pylint warning len-as-conditional 2017-04-15 23:49:41 +02:00
Bahtiar `kalkin-` Gadimov
9da28c9c15
device-list-attached event returns a dev/options tupples list 2017-04-15 23:49:40 +02:00
Bahtiar `kalkin-` Gadimov
1c9636c5af
DeviceAssignment options are now a dict 2017-04-15 23:49:40 +02:00
Bahtiar `kalkin-` Gadimov
5a8cc9bdd3
Remove WrongAssignment exception
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:39 +02:00
Bahtiar `kalkin-` Gadimov
e446e7a2f4
Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:36 +02:00
Bahtiar `kalkin-` Gadimov
b1b005964f
Fix tests broken by the new assignment api 2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
827ca283f3
Fix qubes.tests.devices
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
e84114d3a9
Fix pci device integration tests
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
0f78d257d4
qvm-device add support for assignments
- Add header field ASSIGNED
- Persistent assignments are displayed with an asterisk
- Add '-p' to attach command

Fix qvm-device

Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:30 +02:00
Bahtiar `kalkin-` Gadimov
0b3aebac9f
Update ext/pci to new api
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:29 +02:00
Bahtiar `kalkin-` Gadimov
2a6266887e
BaseVM add DeviceAssignment xml serialization
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
990cfd8ab9
Migrate DeviceCollection to new API
- Use PersistentCollection as _set()
- attach/detach expect DeviceAssignment as parater
- attached(persistent=True) is now persistent()
- attached() returns all attached devices
- assigned() returns all attached device assignments

`#	modified:   templates/libvirt/xen.xml

Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 18:05:28 +02:00
Bahtiar `kalkin-` Gadimov
23c68c5458
Add PersistentCollection helper to qubes.devices 2017-04-15 18:05:27 +02:00