* tests-fixes-1:
api: extract function to make pylint happy
tests/vm: simplify AppVM storage test
storage: do not use deepcopy on volume configs
api: cleanup already started servers when some later failed
tests: fix block devices tests when running on real system
tests: fix some FD leaks
Specify empty 'source' field, so it gets filled with appropriate
template's images. Then also fix recursive 'source' handling - DispVM
root volume should point at TemplateVM's root volume as a source, not a
AppVM's one - which is also only a snapshot.
FixesQubesOS/qubes-issues#2896
This code is unused now. Theoretically this is_outdated implementation
should be moved to FileVolume, but since we don't have VM reference
there, it isn't possible to read appropriate xenstore entry. As we're
phasing out file pool, simply ignore it for now.
QubesOS/qubes-issues#2256
Those functions are coroutines anyway, so allow event handlers to be
too.
Some of this (`domain-create-on-disk`, `domain-remove-from-disk`) will
be useful for appmenus handling.
This will allow starting processes and calling RPC services in those
events. This if required for usb devices, which are attached using RPC
services.
Intentionally keep device listing events synchronous only - to
discourage putting long-running actions there.
This change also require some not-async attach method version for
loading devices from qubes.xml - have `load_persistent` for this.
Always define those properties, always include them in volume config.
Also simplify overriding pool based on volume type defined by those:
override pool unless snap_on_start=True.
QubesOS/qubes-issues#2256
Since libvirt do provide object for dom0 too, return it here.
It's much easier than special-casing AdminVM everywhere. And in fact
sometimes it is actually useful (for example attaching devices from/to
dom0, adjusting memory).
We've decided to make VM name immutable. This is especially important
for Admin API, where some parts (especially policy) are sticked to the
VM name.
Now, to rename the VM, one need to clone it under new name (thanks to
LVM, this is very quick action), then remove the old one.
FixesQubesOS/qubes-issues#2868
* qubesos/pr/111:
vm: drop 'internal' property
qmemman: make sure to release lock
qmemman: fix meminfo parsing for python 3
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
* qubesos/pr/110:
storage: use direct object references, not only identifiers
vm: fix volume_config
storage/lvm: prefix VM LVM volumes with 'vm-'
storage: fix VM rename
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object
Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.
One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.
Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.
QubesOS/qubes-issues#2256
- kernel volume shouldn't have snap_on_start, it's read-only volume
anyway
- root volume of AppVM should have placeholder for 'source'
- private volume of AppVM should _not_ have placeholder for 'source'
(it's ignored anyway, because snap_on_start=False)
QubesOS/qubes-issues#2256
With libvirt in place, this isn't enough - libvirt also keep VM
configuration in its memory and adjusting xenstore doesn't change that.
In fact changing xenstore behind it back make it even worse in some
situations.
QubesOS/qubes-issues#1426
Re-init volume config of all 'snap_on_start' volumes at template
chanage. For this, save original volume config and re-use
config_volume_from_source function introduced in previous commit.
At the same time, forbid changing template of running AppVM or any
DispVM.
QubesOS/qubes-issues#2256
vm.kernel property have type 'str'. Putting None there makes a lot of
troubles: it gets encoded as 'None' in qubes.xml and then loaded back as
'None' string, not None value. Also it isn't possible to assign None
value to str property throgh Admin API.
kernel='' is equally good to specify "no kernel from dom0".
QubesOS/qubes-issues#2622
While libvirt handle locking itself, there is also Qubes-specific
startup part. Especially starting qrexec-daemon and waiting until
qrexec-agent connect to it. When someone will attempt to start VM the
second time (or simply assume it's already running) - qrexec will not be
connected yet and the operation will fail. Solve the problem by wrapping
the whole vm.start() function with a lock, including a check if VM is
running and waiting for qrexec.
Also, do not throw exception if VM is already running.
This way, after a call to vm.start(), VM will be started with qrexec
connected - regardless of who really started it.
Note that, it will not solve the situation when someone check if VM is
running manually, like:
if not vm.is_running():
yield from vm.start()
Such code should be changed to simply:
yield from vm.start()
FixesQubesOS/qubes-issues#2001FixesQubesOS/qubes-issues#2666
And place them in /qubes-service/ QubesDB directory. This allows
extensions to easily store some data not exposed to VM, but also have
control what VM will see. And at the same time, it make it compatible
with existing services framework
QubesOS/qubes-issues#1637
Setting VMProperty to None VM should be encoded as '' value (according
to VMProperty._none_value). But value validation rejected this value.
QubesOS/qubes-issues#2622
Use '<option name="option_name">option_value</option>' instead of
'<options option_name="option_value"/>'. It's more consistent with the
rest of qubes.xml - have one thing per element.
Also, add options deserialization test.
When libvirt domain is not defined, it isn't running for sure.
This commit fixes the case when vm.is_running() appears anywhere in the
code used during libvirt xml building. In this case, it's mostly about
PCI device description for libvirt.
Do not initialize it only at qubes.xml load time, but re-read vm.kernel
property each time the path is constructed. While at it, add support for
vm.kernel set to 'None' - simply don't include modules.img (xvdd) then.
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).
FixesQubesOS/qubes-issues#2686
This is required to get shutdown notification, when it wasn't initiated
by qubesd (for example 'poweroff' command inside of VM).
Libvirt event loop implementation must be registered before making
connection to libvirt, so move it to the beginning of main().
For now, only 'domain-shutdown' event is emited.
User session may not be started at all (for example no qubes packages
installed there), so don't block it in all the cases. Also this would
prevent running 'qubes.WaitForSession' service...
In practice, default value for 'gui' argument is False, so in most cases
user session will be ignored. Which doesn't matter in most cases -
especially for services called by qubesd.
Keep it uniform - QubesVM() object is responsible for handling
vm.dir_path, Storage() is responsible for handling disk volumes (which
may live in that directory
QubesOS/qubes-issues#2256
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.
This commit does not touch any pool implementation - all of them are
still synchronous.
QubesOS/qubes-issues#2256
Some methods inherited from dict (pop and setdefault here) are covered
by placeholders raising NotImplementedError. Lets fix their signatures
(to match those of dict) to really get NotImplementedError, instead of
TypeError.
Management API gives access only to qubes.property. And this is
actually a good thing, so instead of extending it to access also
builtins.property, add a simple decorator to define read-only, stateless
qubes.property.
QubesOS/qubes-issues#2622
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.
The tool itself is already moved to core-mgmt-client repository.
QubesOS/qubes-issues#853
* kalkin/device-assignments: (21 commits)
PCI extension cache PCIDevice objects
Make pylint ♥
Fix pylint warning no-else-return
Fix pylint warning len-as-conditional
device-list-attached event returns a dev/options tupples list
DeviceAssignment options are now a dict
Remove WrongAssignment exception
Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
Update relaxng devices option element
Fix tests broken by the new assignment api
Fix qubes.tests.devices
Fix pci device integration tests
qvm-device add support for assignments
Update ext/pci to new api
BaseVM add DeviceAssignment xml serialization
Migrate DeviceCollection to new API
Add PersistentCollection helper to qubes.devices
Add DeviceAssignment
qvm-device validates device parameters
qvm-device fix handling of non block devices
...
* core3-policy:
Make pylint happy
tests: disable GTK tests on travis
qubespolicy: make pylint happy
qubespolicy: run GUI code inside user session and expose it as dbus object
tests: plug rpc-window tests into main test runner
qubespolicy: plug GUI code into qrexec-policy tool
rpm: add rpc-window related files to package
rpc-window: adjust for qubespolicy API
rpc-window: use pkg_resources for glade file
rpc-window: use 'edit-find' icon if no other is found
rpc-window: adjust for python3
rpc-window: code style adjustments
Import new rpc confirmation window code
qubesd: add second socket for in-dom0 internal calls
policy: qrexec-policy cli tool
tests: qubespolicy tests
qubespolicy: initial version for core3
vm/appvm: add dispvm_allowed property
dispvm: don't load separate Qubes() instance when handling DispVM
0) All those methods are now awaitable rather than synchronous.
1) The base method is run_service(). The method run() was rewritten
using run_service('qubes.VMShell', input=...). There is no provision
for running plain commands.
2) Get rid of passio*= arguments. If you'd like to get another return
value, use another method. It's as simple as that.
See:
- run_service_for_stdio()
- run_for_stdio()
Also gone are wait= and localcmd= arguments. They are of no use
inside qubesd.
3) The qvm-run tool and tests are left behind for now and will be fixed
later. This is because they also need event loop, which is not
implemented yet.
fixesQubesOS/qubes-issues#1900QubesOS/qubes-issues#2622
This also means we don't check if a VM with given name (in case of
VMProperty) exists in the system, at this stage. But this is ok, lets
not duplicate work of property setter.
QubesOS/qubes-issues#2622
Don't allow characters potentially interfering with qrexec. To be on the
safe side, allow only alphanumeric characters + very few selected
punctuations.
Split it into two functions: validate_name - context-less verification,
and actual _setter_name which perform additional verification in
context of actual VM.
Switch to qubes.exc.* exceptions where appropriate.
QubesDB daemon no longer remove socket created by new instance, so one
part of VM restart race condition is solved. The only remaining part is
to ensure that we really connect to the new instance, instead of talking
to the old one (soon to be terminated).
FixesQubesOS/qubes-issues#1694
VM files may be already removed. Don't fail on this while removing a
VM, it's probably the reason why domain is being removed.
qvm-remove tool have its own guard for this, but it isn't enough - if
rmtree(dir_path) fails, storage.remove() would not be called, so
non-file storages would not be cleaned up.
This is also needed to correctly handle template reinstallation - where
VM directory is moved away to call create_on_disk again.
QubesOS/qubes-issues#2412
Just calling pool.init_volume isn't enough - a lot of code depends on
additional data loaded into vm.storage object. Provide a convenient
wrapper for this.
At the same time, fix loading extra volumes from qubes.xml - don't fail
on volume not mentioned in initial vm.volume_config.
QubesOS/qubes-issues#2256
Set parameters for possibly hiding domain's real IP before attaching
network to it, otherwise we'll have race condition with vif-route-qubes
script.
QubesOS/qubes-issues#1143
This is the IP known to the domain itself and downstream domains. It may
be a different one than seen be its upstream domain.
Related to QubesOS/qubes-issues#1143`
This helps hiding VM IP for anonymous VMs (Whonix) even when some
application leak it. VM will know only some fake IP, which should be set
to something as common as possible.
The feature is mostly implemented at (Proxy)VM side using NAT in
separate network namespace. Core here is only passing arguments to it.
It is designed the way that multiple VMs can use the same IP and still
do not interfere with each other. Even more: it is possible to address
each of them (using their "native" IP), even when multiple of them share
the same "fake" IP.
Original approach (marmarek/old-qubes-core-admin#2) used network script
arguments by appending them to script name, but libxl in Xen >= 4.6
fixed that side effect and it isn't possible anymore. So use QubesDB
instead.
From user POV, this adds 3 "features":
- net/fake-ip - IP address visible in the VM
- net/fake-gateway - default gateway in the VM
- net/fake-netmask - network mask
The feature is enabled if net/fake-ip is set (to some IP address) and is
different than VM native IP. All of those "features" can be set on
template, to affect all of VMs.
Firewall rules etc in (Proxy)VM should still be applied to VM "native"
IP.
FixesQubesOS/qubes-issues#1143
Instead of excerpt from /proc/meminfo, use just one integer. This make
qmemman handling much easier and ease implementation for non-Linux OSes
(where /proc/meminfo doesn't exist).
For now keep also support for old format.
FixesQubesOS/qubes-issues#1312
There is no point in changing *public API* for just a change without any
better reason. It turned out most of those settings will be the same in
Qubes 4.0, so keep names the same.
This reverts commit 2d6ad3b60c.
QubesOS/qubes-issues#1812
This is migration of core2 commits:
commit d0ba43f253
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date: Mon Jun 6 02:21:08 2016 +0200
core: start guid as normal user even when VM started by root
Another attempt to avoid permissions-related problems...
QubesOS/qubes-issues#1768
commit 89d002a031
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date: Mon Jun 6 02:19:51 2016 +0200
core: use runuser instead of sudo for switching root->user
There are problems with using sudo in early system startup
(systemd-logind not running yet, pam_systemd timeouts). Since we don't
need full session here, runuser is good enough (even better: faster).
commit 2265fd3d52
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date: Sat Jun 4 17:42:24 2016 +0200
core: start qubesdb as normal user, even when VM is started by root
On VM start, old qubesdb-daemon is terminated (if still running). In
practice it happen only at VM startart (shutdown and quickly start
again). But in that case, if the VM was started by root, such operation
would fail.
So when VM is started by root, make sure that qubesdb-daemon will be
running as normal user (the first user in group 'qubes' - there should
be only one).
FixesQubesOS/qubes-issues#1745
* core3-devices:
Fix core2migration and tests for new devices API
tests: more qubes.devices tests
qubes/ext/pci: implement pci-no-strict-reset/BDF feature
qubes/tools: allow calling qvm-device as qvm-devclass (like qvm-pci)
qubes: make pylint happy
qubes/tools: add qvm-device tool (and tests)
tests: load qubes.tests.tools.qvm_ls
tests: PCI devices tests
tests: add context manager to catch stdout
qubes/ext/pci: move PCI devices handling to an extension
qubes/devices: use more detailed exceptions than just KeyError
qubes/devices: allow non-persistent attach
qubes/storage: misc fixes for VM-exposed block devices handling
qubes: new devices API
FixesQubesOS/qubes-issues#2257
