Test suite creates some VMs and needs to pass the knowledge about them
to qrexec policy checker. This is done using Admin API, so we need to
substitute qubesd with our own API server.
* qubesos/pr/111:
vm: drop 'internal' property
qmemman: make sure to release lock
qmemman: fix meminfo parsing for python 3
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
* qubesos/pr/110:
storage: use direct object references, not only identifiers
vm: fix volume_config
storage/lvm: prefix VM LVM volumes with 'vm-'
storage: fix VM rename
Make qubes.NotifyTools reuse logic of qubes.FeaturesRequest, then move
actual request processing to 'features-request' event handler. At the
same time implement handling 'qrexec' and 'gui' features request -
allowing to set template features when wasn't already there.
Behavior change: template is no longer allowed to change feature value
(regardless of being True or False). This means the user will always be
able to override what template have set.
Drop DeviceInfo.data - device extension should provide a subclass with
proper individual fields.
Drop DeviceAssignment.frontend_domain - this information is redundant -
frontend domain is defined by where DeviceAssignment is attached.
Rename DeviceCollection.devclass to bus - devclass if confusing here,
because this term is also used for DeviceInfo subclass.
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object
Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.
One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.
Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.
QubesOS/qubes-issues#2256
Setting VMProperty to None VM should be encoded as '' value (according
to VMProperty._none_value). But value validation rejected this value.
QubesOS/qubes-issues#2622
Accessing non-existing property is a common action (for example
hasattr() do try to access the property). So, introduce specific
exception, inheriting from AttributeError. It will behave very similar
to standard (non-Admin-API) property access.
This exception is reported to the Admin API user, so it will be possible
to distinguish between non-existing property and access denied. But it
isn't any significant information leak, as list of valid properties is
publicly available in the source code.
QubesOS/qubes-issues#853
Use '<option name="option_name">option_value</option>' instead of
'<options option_name="option_value"/>'. It's more consistent with the
rest of qubes.xml - have one thing per element.
Also, add options deserialization test.
When device extension do not return some "persistent" device as
currently attached, still return it, as it will be attached at next
domain startup. User can distinguish such devices by having
frontend_domain=None (or other VM).
Also, return a set from DeviceCollection.assignments().
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).
FixesQubesOS/qubes-issues#2686
This requires creating LVM volume group, so create on based on loop dev
in /tmp.
This is rather rough, but if any of this fails, run the tests anyway -
it will simply skip LVM tests.
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.
This commit does not touch any pool implementation - all of them are
still synchronous.
QubesOS/qubes-issues#2256
This allows avoid race condition between registering event handlers and
performing some action. The important thing is the event sent after
registering event handlers in qubesd. This means state changes (like
VM start/stop) after 'connection-established' event will be included in
event stream.
QubesOS/qubes-issues#2622
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.
The tool itself is already moved to core-mgmt-client repository.
QubesOS/qubes-issues#853
* kalkin/device-assignments: (21 commits)
PCI extension cache PCIDevice objects
Make pylint ♥
Fix pylint warning no-else-return
Fix pylint warning len-as-conditional
device-list-attached event returns a dev/options tupples list
DeviceAssignment options are now a dict
Remove WrongAssignment exception
Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
Update relaxng devices option element
Fix tests broken by the new assignment api
Fix qubes.tests.devices
Fix pci device integration tests
qvm-device add support for assignments
Update ext/pci to new api
BaseVM add DeviceAssignment xml serialization
Migrate DeviceCollection to new API
Add PersistentCollection helper to qubes.devices
Add DeviceAssignment
qvm-device validates device parameters
qvm-device fix handling of non block devices
...
This also means we don't check if a VM with given name (in case of
VMProperty) exists in the system, at this stage. But this is ok, lets
not duplicate work of property setter.
QubesOS/qubes-issues#2622
If kwargs contains dict as one of values, it isn't hashable and can't be
used as value in frozenset/tuple. Convert such values into
frozenset(dict.items()). Only one (more) level is supported, but it
should be enough.
Solution from http://stackoverflow.com/a/13264725
Use the right cow image and apply the second layer to provide read-write
access. The correct setup is:
- base image + base cow -> read-only snapshot (base changes "cached"
until committed)
- read-only snapshot + VM cow -> read-write snapshot (changes discarded
after VM shutdown)
This way, even VM without Qubes-specific startup scripts will can
benefit from Template VMs, while VMs with Qubes-specific startup scripts
may still see original root.img content (for possible signature
verification, when storage domain got implemented).
QubesOS/qubes-issues#2256
tar can't write archive with _contents_ of block device. We need this to
backup LVM-based disk images. To avoid dumping image to a file first,
create a simple tar archiver just for this purpose.
Python is not the fastest possible technology, it's 3 times slower than
equivalent written in C. But it's much easier to read, much less
error-prone, and still process 1GB image under 1s (CPU time, leaving
along actual disk reads). So, it's acceptable.
To ease all this, rework restore workflow: first create QubesVM objects,
and all their files (as for fresh VM), then override them with data
from backup - possibly redirecting some files to new location. This
allows generic code to create LVM volumes and then only restore its
content.
First part - handling firewall.xml and rules formatting.
Specification on https://qubes-os.org/doc/vm-interface/
TODO (for dom0):
- plug into QubesVM object
- expose rules in QubesDB (including reloading)
- drop old functions (vm.get_firewall_conf etc)
QubesOS/qubes-issues#1815
* core3-devices:
Fix core2migration and tests for new devices API
tests: more qubes.devices tests
qubes/ext/pci: implement pci-no-strict-reset/BDF feature
qubes/tools: allow calling qvm-device as qvm-devclass (like qvm-pci)
qubes: make pylint happy
qubes/tools: add qvm-device tool (and tests)
tests: load qubes.tests.tools.qvm_ls
tests: PCI devices tests
tests: add context manager to catch stdout
qubes/ext/pci: move PCI devices handling to an extension
qubes/devices: use more detailed exceptions than just KeyError
qubes/devices: allow non-persistent attach
qubes/storage: misc fixes for VM-exposed block devices handling
qubes: new devices API
FixesQubesOS/qubes-issues#2257
Instead of old per-VM flag 'pci_strictreset', now implement this as
per-device flag using features. To not fail on particular device
assignment set 'pci-no-strict-reset/DEVICE-BDF' to True. For
example 'pci-no-strict-reset/00:1b.0'.
QubesOS/qubes-issues#2257
Implement required event handlers according to documentation in
qubes.devices.
A modification of qubes.devices.DeviceInfo is needed to allow dynamic,
read-only properties.
QubesOS/qubes-issues#2257
Allow device plugin to list attached and available devices. Enforce
at API level every device being exposed by some domain.
This commit only changes devices API, but not update existing users
(pci) yet.
QubesOS/qubes-issues#2257
Some tests do not apply, as there is no savefile and attributes
propagation is much simpler. Dropped tests:
- test_000_firewall_propagation
- test_001_firewall_propagation
- test_000_prepare_dvm
QubesOS/qubes-issues#2253
Do not access vm.libvirt_domain after it being already removed - this
will redefine it again in libvirt, just to undefine it in a moment.
On the other hand, few lines below there is fallback libvirt cleanup, in
case of proper one not working.
Now failure to load external tests shows in which entry point the error
happened and a useful traceback. The traceback extends from the "try"
statement down to the actual error line, but it does not include the
frames above, ie. from the invocation to the load_tests routine. This is
a limitation of Python itself and usually not a problem.
1. It is unclear yet whether dispvm_netvm will be implemented in core3, but
probably not.
2. Remove tests for setting memory/cpu above host resouces - rejecting
those values at property set time would break backup restore on some
machines (when migrating from bigger to smaller system).
This commit eliminates import statements happening in the middle of the
file (between two classes definition). The cycles are still there. The
only magic module is qubes itself.
