Commit Graph

779 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
f223594f92
app: kill default_fw_netvm property
Having both default_netvm and default_fw_netvm cause a lot of confusion,
because it isn't clear for the user which one is used when. Additionally
changing provides_network property may also change netvm property, which
may be unintended effect. This as a whole make it hard to:
- cover all netvm-changing actions with policy for Admin API
- cover all netvm-changing events (for example to apply the change to
the running VM, or to check for netvm loops)

As suggested by @qubesuser, kill the default_fw_netvm property and
simplify the logic around it.
Since we're past rc1, implement also migration logic. And add tests for
said migration.

Fixes QubesOS/qubes-issues#3247
2017-12-07 01:40:29 +01:00
Marek Marczykowski-Górecki
f2cd7fb226
Merge branch 'tests-and-fixes-20171205' 2017-12-07 01:39:34 +01:00
Marek Marczykowski-Górecki
7b81d7affa
Merge remote-tracking branch 'qubesos/pr/166'
* qubesos/pr/166:
  create "lvm" pool using rootfs thin pool instead of hardcoding qubes_dom0-pool00
  change default pool code to be fast
  cache PropertyHolder.property_list and use O(1) property name lookups
  remove unused netid code
  cache isinstance(default, collections.Callable)
  don't access netvm if it's None in visible_gateway/netmask
2017-12-06 00:41:37 +01:00
Marek Marczykowski-Górecki
4d6bfbab4d
tests: improve spoof_ip test
Not only check if full round trip ping (does not) work, but also if just
echo-request get filtered.
2017-12-05 17:40:30 +01:00
Marek Marczykowski-Górecki
379add52ba
tests: skip network tests on whonix-gw and whonix-ws
whonix-ws also have non-standard firewall and require specific tests for
that.
2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
088c255355
tests: add create_qdb_entries() unit test 2017-12-05 17:39:34 +01:00
Marek Marczykowski-Górecki
0afee4b05e
Improve checking for netvm loop
There were many cases were the check was missing:
 - changing default_netvm
 - resetting netvm to default value
 - loading already broken qubes.xml

Since it was possible to create broken qubes.xml using legal calls, do
not reject loading such file, instead break the loop(s) by setting netvm
to None when loop is detected. This will be also useful if still not all
places are covered...

Place the check in default_netvm setter. Skip it during qubes.xml loading
(when events_enabled=False), but still keep it in setter, to _validate_ the
value before any property-* event got fired.
2017-12-05 17:39:33 +01:00
Marek Marczykowski-Górecki
19a1579a99
tests: fix deadlock in filecopy test
Error window (where test need to send Enter key) is opened while
qvm-move-to-vm is still running.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
99874a0a25
tests: make waiting for window asyncio aware
For now just replace sleep with asyncio.sleep. Later it may make sense
to change subprocess.call too.
2017-12-05 17:39:32 +01:00
Marek Marczykowski-Górecki
a92dd99fbb
Merge branch '20171107-storage'
* 20171107-storage:
  api/admin: add API for changing revisions_to_keep dynamically
  storage/file: move revisions_to_keep restrictions to property setter
  api/admin: hide dd statistics in admin.vm.volume.Import call
  storage/lvm: fix importing different-sized volume from another pool
  storage/file: fix preserving spareness on volume clone
  api/admin: add pool size and usage to admin.pool.Info response
  storage: add size and usage properties to pool object
2017-11-20 22:52:50 +01:00
Marek Marczykowski-Górecki
2c6c766968
Merge branch '20171107-tests-backup-api-misc'
* 20171107-tests-backup-api-misc:
  test: make race condition on xterm close less likely
  tests/backupcompatibility: fix handling 'internal' property
  backup: fix handling target write error (like no disk space)
  tests/backupcompatibility: drop R1 format tests
  backup: use offline_mode for backup collection
  qubespolicy: fix handling '$adminvm' target with ask action
  app: drop reference to libvirt object after undefining it
  vm: always log startup fail
  api: do not log handled errors sent to a client
  tests/backups: convert to new restore handling - using qubesadmin module
  app: clarify error message on failed domain remove (used somewhere)
  Fix qubes-core.service ordering
2017-11-20 22:52:26 +01:00
Marek Marczykowski-Górecki
cf92a576ad
Merge remote-tracking branch 'qubesos/pr/159'
* qubesos/pr/159:
  qubes/vm: Improve stopped event handling
2017-11-20 22:49:05 +01:00
Marek Marczykowski-Górecki
3217c3ac4e
Merge remote-tracking branch 'qubesos/pr/156'
* qubesos/pr/156:
  tests: Add unit tests for pvh virt_mode
  Don't allow attached PCI devices and virt_mode = 'pvh'
2017-11-20 22:40:53 +01:00
qubesuser
f2b8ad7d38 remove unused netid code
it's unused and has a netid property with name different than key
that would cause issues in the next commit
2017-11-11 02:37:37 +01:00
Marek Marczykowski-Górecki
45af0913f2
test: make race condition on xterm close less likely
xterm is very fast on closing when application inside terminates. It is
so fast with closing on keydown event that xdotool do not manage to send
keyup event, resulting in xdotool crash. Add a little more time for
that.
2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
07d4f4b340
tests/backupcompatibility: fix handling 'internal' property 2017-11-07 03:10:42 +01:00
Marek Marczykowski-Górecki
11fd2cf115
tests/backupcompatibility: drop R1 format tests
It is no longer supported.
2017-11-07 03:10:41 +01:00
Marek Marczykowski-Górecki
31a55dcd18
tests/backups: convert to new restore handling - using qubesadmin module
Besides converting itself, change how the test verify restore
correctness: first collect VM metadata (and hashes of data) into plain
dict, then compare against it. This allow to destroy old VMs objects
before restoring the backup, so avoid having duplicate objects of the
same VM - which results in weird effects like trying to undefine libvirt
object twice.
2017-11-07 03:10:40 +01:00
Marek Marczykowski-Górecki
c3afdde3ef
api/admin: add API for changing revisions_to_keep dynamically
This one pool/volume property makes sense to change dynamically. There
may be more such properties, but lets be on the safe side and take
whitelist approach - allow only selected (just one for now), instead of
blacklisting any harmful ones.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
81f455e15d
storage/file: move revisions_to_keep restrictions to property setter
Do not check for accepted value only in constructor, do that in property
setter. This will allow enforcing the limit regardless of how the value
was set.

This is preparation for dynamic revisions_to_keep change.

QubesOS/qubes-issues#3256
2017-11-07 02:57:59 +01:00
Marek Marczykowski-Górecki
f18f4c9bff
api/admin: add pool size and usage to admin.pool.Info response
QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
Marek Marczykowski-Górecki
e76372b934
storage: add size and usage properties to pool object
Add Pool.size and Pool.usage to the API. Implement them for LVM and File
pools. Add appropriate tests.

QubesOS/qubes-issues#3240
2017-11-07 02:57:58 +01:00
HW42
c695f3268c linux-stubdom: Set cmdline in HVM config 2017-10-30 16:19:01 +01:00
HW42
b907303a39 qubes/vm: Improve stopped event handling
The previous version did not ensure that the stopped/shutdown event was
handled before a new VM start. This can easily lead to problems like in
QubesOS/qubes-issues#3164.

This improved version now ensures that the stopped/shutdown events are
handled before a new VM start.

Additionally this version should be more robust against unreliable
events from libvirt. It handles missing, duplicated and delayed stopped
events.

Instead of one 'domain-shutdown' event there are now 'domain-stopped'
and 'domain-shutdown'. The later is generated after the former. This way
it's easy to run code after the VM is shutdown including the stop of
it's storage.
2017-10-21 05:57:57 +02:00
Marek Marczykowski-Górecki
e2eaa57f65
Merge branch 'tests-20171020'
* tests-20171020:
  tests: fix asyncio usage some more places
  tests: fix reporting in network tests
  tests: domain-shutdown event race condition
  tests: improve events tests
  tests: fix logic error in wait_for_window()
  tests: cleanup QubesDB connection on domain remove
2017-10-21 04:33:35 +02:00
Marek Marczykowski-Górecki
42f8619559
tests: fix asyncio usage some more places
gui_memory_pinning tests finally works. Also, fix DispVM and private
volume resize tests.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
511502ae7e
tests: fix reporting in network tests
Do not trash stdout with VM's command output. Log it as part of
(potential) failure message.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
c3430fe5f6
tests: domain-shutdown event race condition
Second tests, for actual events, not storage handling done by its
handler.

cc @HW42

QubesOS/qubes-issues#3164
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
96f2e503db
tests: improve events tests
Check property-related events, including their arguments.
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
71fc5217f7
tests: fix logic error in wait_for_window()
Revert change introduced in cbc5dbb9b1
2017-10-21 04:06:15 +02:00
Marek Marczykowski-Górecki
43c09467b1
tests: cleanup QubesDB connection on domain remove
If domain got removed during the tests (for example DispVM), vm.close()
wouldn't be called in cleanup and some file descriptors will be
leaked. Add event handler for cleaning this up. Do not use close()
method here, because it is destructive, but the object may still be used
by the test.
2017-10-21 04:06:14 +02:00
Marek Marczykowski-Górecki
8f1bfa65f8
tests: fix typo 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
4809c16e08
admin-api: add admin.*.property.GetDefault
Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
038ca6d5d7
base: add method for getting default property value
Allow to get default value even it isn't set currently. This will allow
(G)UI to present better view, without duplicating logic for default
value.

Fixes QubesOS/qubes-issues#3197
2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
85c3368a0d
comments fixes 2017-10-21 04:05:15 +02:00
Marek Marczykowski-Górecki
c2c3225622
tests: root fs resize tests - for all templates
Check resize on each template separately, because it involves VM's
scripts (either qubes.ResizeDisk service, or some startup script).

QubesOS/qubes-issues#3173
2017-10-21 04:05:14 +02:00
Marek Marczykowski-Górecki
77ce956c2d
tests: improve root volume resize tests
- clone all features, not just qrexec (especially include 'gui')
- do not leak VM reference on failed test
- add test for online root volume resize

QubesOS/qubes-issues#3173
2017-10-21 03:35:01 +02:00
Marek Marczykowski-Górecki
27b96cce4c
firewall: skip expired rules
Expired rules are skipped while loading the firewall. Do that also when
such rules expired after loading the firewall. This applies to both
Admin API and actually applying the rules (sending them to appropriate
VM).

Related QubesOS/qubes-issues#3020
2017-10-21 03:35:00 +02:00
Marek Marczykowski-Górecki
836d9f902a
Merge branch 'bug3164'
* bug3164:
  tests: add regression test for #3164
  storage/lvm: make sure volume cache is refreshed after changes
  storage/lvm: fix Volume.verify()
  storage/lvm: remove old volume only after successfully cloning new one
2017-10-16 04:28:26 +02:00
Marek Marczykowski-Górecki
c8519a700f
tests: add regression test for #3164
This is a race condition, so to make it more likely to fail (if it's
broken), make some things manually. In normal circumstances this order
of actions is also possible, just less likely to happen. But as seen in
the bug report, happens from time to time.

QubesOS/qubes-issues#3164
2017-10-16 00:48:25 +02:00
Marek Marczykowski-Górecki
2164a8d7b8
Change license to LGPL v2.1+
See this thread for reasoning and acceptance from contributors:
https://groups.google.com/d/topic/qubes-devel/G7KzrfU0lWY/discussion
"Changing qubes-core-admin license to LGPL v2.1+"
2017-10-12 00:11:50 +02:00
HW42
4a84d5b222 tests: Add unit tests for pvh virt_mode 2017-10-11 21:11:36 +02:00
Marek Marczykowski-Górecki
af60fbb25d
tests: dispvm tests fixes 2017-10-04 15:24:53 +02:00
Marek Marczykowski-Górecki
6990550884
tests: more tests fixes, drop expectedFailure where not needed anymore 2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
e0be7861f9
tests: extend qrexec_policy context manager - custom action
Allow to use custom action - for example with 'target=' argument.
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
cbc5dbb9b1
tests: various fixes
- Use proper features/services names (updates proxy test).
- Fix logic error in wait_for_window.
- Fix test for qvm-sync-clock (first sync clockvm, then dom0), also fix
  cleanup (unset clockvm before removing it)
- More fixes for asyncio usage
2017-10-04 15:24:52 +02:00
Marek Marczykowski-Górecki
64c5b2c00f
tests: cleanup DispVMs created during tests
Since those do not have specific prefix (test-) compare list of them
with original qubes.xml
2017-10-04 15:24:51 +02:00
Marek Marczykowski-Górecki
3a5e8482c0
Merge branch 'fixes-20170929'
* fixes-20170929:
  vm: do not start QubesDB watch instance multiple times
  vm: report storage.stop() errors to log
  vm: move comment
  storage: fix method name in LinuxModules volume
  Prevent removing domain that is referenced from anywhere
  vm: add vm.klass property
  Move QubesVM.{name,qid,uuid,label} to BaseVM
  vm: do not allow deleting template property from AppVM and DispVM
  vm/qubesvm: emit event on failed startup
  vm/qubesvm: remove duplicated qmemman_client.close()
  vm/dispvm: cleanup DispVM also on failed startup
  vm/dispvm: fix error message
  ext/block: properly list devtype=cdrom option
  block: fix handling non-existing devices
  block: improve handling device name and description
2017-10-03 12:04:00 +02:00
Marek Marczykowski-Górecki
12b7e22d27
vm: do not start QubesDB watch instance multiple times
vm.create_qdb_entries can be called multiple times - for example when
changing VM IP. Move starting qdb watcher to start(). And just in case,
cleanup old watcher (if still exists) before starting new one.
This fixes one FD leak.
2017-09-29 12:05:23 +02:00
Marek Marczykowski-Górecki
9172a7708d
tests: fix various object leaks cont
Do not use self.fail when handling exception - this will keep exception
object referenced, which in turn have reference to domain object (via
traceback).
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
f9da1f2dd7
tests: various fixes for networking tests
- Update feature name for enabling network-manager.
- Fix error reporting
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
fdf780aad0
tests/network: match vm names with variable names
Having testvm1 variable for "vm2" is really confusing.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
8b34b9a5e4
tests: hide unneeded messages 2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
3c54244fdc
tests: fix various object and FD leaks
- Prefer instance attributes over local variables - the former ones do
not leak into traceback object and are cleaned up by tests framework.
- Use 'with' syntax for handling files.
- Use subprocess.DEVNULL instead of open('/dev/null') where applicable
- Delete local variables when not needed anymore.
2017-09-29 11:53:59 +02:00
Marek Marczykowski-Górecki
d96d4a94e1
tests: various fixes
- Fix str/bytes
- Call skipTest as early as possible - before doing any setup
- Fix networking tests - configuration commands needs to be called as
root (missing user= argument).
- Fix setting firewall - policy is no longer changeable
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
ed3001da99
tests: fix asyncio usage cont...
- Add missing loop.run_until_complete() calls.
- Convert subprocess.Popen to asyncio.create_subprocess_exec where
needed (when called process needs to communicate with qubesd).
- Cleanup processes (call .wait()).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
f0da5b21df
tests: improve "revert template changes" test
There is no more significant difference between PV and HVM. VMs are HVM
by default anyway. More important for this test is difference between
Linux (with Qubes packages installed) and other OS-es. Rename tests
accordingly. The later one is still incomplete.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
4b51c9da07
tests/storage: don't fail the test when directory already exists
/var/tmp/test-pool should normally be removed by previous test cleanup,
but do not fail next test if it wasn't.
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
344bb16ae2
tests: fix various object leaks at tests framework level
The most important change is doing vm.close() when removing domain -
this means it wouldn't be cleaned later by iterating over app.domains.

Other changes include removing VMs in the right order, regarding netvm
dependency (otherwise killing or removing may fail). And one more
missing coroutine handling (in shutdown_and_wait).
2017-09-29 11:53:58 +02:00
Marek Marczykowski-Górecki
a929916ce8
tests/storage: do not depend on system default storage pool
Use just created storage pool for tests, instead of system default
(which depends on system configuration).
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
d8ff85eee6
tests: fix cleanup function
Don't crash when directory to cleanup do not exists.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
3075767bb8
tests: add more complex test for netvm cyclic connection
Something more complex than VM connecting to itself.
2017-09-29 11:53:57 +02:00
Marek Marczykowski-Górecki
9e8c40867b
Prevent removing domain that is referenced from anywhere
Check VM properties and global properties (all of them).

Fixes QubesOS/qubes-issues#3128
2017-09-29 11:53:01 +02:00
Marek Marczykowski-Górecki
9f88fa7f0c
Move QubesVM.{name,qid,uuid,label} to BaseVM
Reduce strange code in BaseVM (accessing non-existing self.name) and
code duplication.
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
dc0e1a5481
vm: do not allow deleting template property from AppVM and DispVM
There is intentionally no default template in terms of qubes.property
definition, to not cause problems when switching global default_template
property - like breaking some VMs, or forcing the user to shutdown all
of them for this. But this also means it shouldn't be allowed to reset
template to "default" value, because it will result in a VM without
template at all.

Fixes QubesOS/qubes-issues#3115
2017-09-29 11:53:00 +02:00
Marek Marczykowski-Górecki
3548ee1163
ext/block: properly list devtype=cdrom option 2017-09-29 11:52:59 +02:00
Marek Marczykowski-Górecki
e0cadfdd67
Merge remote-tracking branch 'qubesos/pr/150'
* qubesos/pr/150:
  qubes/tests: moar fixes
  test-packages: add missing libvirt classes
  qubes/tests: do not deadlock on .drain()
  qubes/vm: put name= first in __repr__
  tests: fix some memory leaks
  tests: complain about memory leaks
  tests: use one event loop and one libvirtaio impl
2017-09-22 20:54:11 +02:00
Wojtek Porczyk
589d567ebc qubes/tests: moar fixes 2017-09-21 14:18:35 +02:00
Wojtek Porczyk
b78ad1993c qubes/tests: do not deadlock on .drain() 2017-09-21 14:17:36 +02:00
Wojtek Porczyk
bd123a3812 tests: fix some memory leaks
This is work in progress. There are still more leaks.
2017-09-19 17:01:29 +02:00
Wojtek Porczyk
5aa9fa2db4 tests: complain about memory leaks
Currently this detects leaking VM (and all subclasses), Qubes and
libvirt-related objects. Usable only with --failfast.
2017-09-19 16:59:09 +02:00
Wojtek Porczyk
7df8f51011 tests: use one event loop and one libvirtaio impl
Recently libvirt removed support for changing event implementation.
Therefore we have to use a single, global one and we check if it is
empty between tests.
2017-09-19 16:47:14 +02:00
HW42
c6c6f681ae tests: Fix expected HVM xml
The last two commits (intentionally) changed the libxl xml. So adapt the
expected value in the test.
2017-09-15 16:19:10 +02:00
Marek Marczykowski-Górecki
55208a0c7c
Merge branch 'dispvm-policy' 2017-09-14 00:15:29 +02:00
Marek Marczykowski-Górecki
0d59965a7b
Merge branch 'cdrom-boot'
* cdrom-boot:
  devices: fix error reporting
  api/admin: implement admin.vm.device....Set.persistent
  devices: implement DeviceCollection.update_persistent()
  devices: move DeviceInfo definition earlier
  api: do not fail events when listener is cancelled in the meantime
2017-09-07 13:02:04 +02:00
Marek Marczykowski-Górecki
99bd193688
Rename 'dispvm_allowed' to 'template_for_dispvms'
'dispvm_allowed' name was confusing, because it suggested being able to
spawn new DispVMs, not being a template for DispVM.

Fixes QubesOS/qubes-issues#3047
2017-09-05 02:07:26 +02:00
Marek Marczykowski-Górecki
c247ddff72
dispvm: fix setting up new DispVM
Clone properties from DispVM template after setting base properties
(qid, name, uuid). This means we can use standard clone_properties()
function. Otherwise various setters may fail - for example
netvm setter require uuid property initialized (for VM lookup in VM
collection).
Also, make dispvm_allowed check more robust - include direct creation of
DispVM, and also check just before VM startup (if property was changed
in the meantime).

Fixes QubesOS/qubes-issues#3057
2017-09-05 02:07:26 +02:00
Marek Marczykowski-Górecki
687ca4092c
tests: various fixes to integration tests
Mostly add missing self.loop.run_until_complete().
2017-09-05 02:01:58 +02:00
Marek Marczykowski-Górecki
d676576600
tests: fix qvm-sync-clock test for asyncio and new time sync API
qvm-sync-clock in dom0 now synchronize only dom0 time. For VM time,
qvm-sync-clock needs to be called in VM. Also, both will communicate
with qubesd, so must be called asynchronously from tests.
2017-09-05 02:01:58 +02:00
Marek Marczykowski-Górecki
cea70748a6
api/admin: implement admin.vm.device....Set.persistent
This will allow converting persistent device (used to boot VM for
example) to non-persistent.

QubesOS/qubes-issues#3055
2017-09-05 01:33:47 +02:00
Marek Marczykowski-Górecki
9d062c4c66
devices: implement DeviceCollection.update_persistent()
Allow attached device to be converted from persistent to non-persistent
and the other way around.
This is to allow starting a VM with some device attached temporarily.
When VM is not running, it is possible to attach device only
persistently, so this change will allow to do that, then, after starting
the VM, change it to non-persistent - so it will not be attached again
at further startups.

QubesOS/qubes-issues#3055
2017-09-05 01:33:47 +02:00
Wojtek Porczyk
6ff1bfdc16 qubes/tests: convert some tearDown into addCleanup
This is because .tearDown() is not executed if the exception occurs in
setUp() [for example self.skipTest() raises an exception]. The lower
levels of .tearDown() being executed are critical to not leaking file
descriptors.
2017-08-31 20:30:24 +02:00
Wojtek Porczyk
d0f2fdba55 qubes/tests: fix some of the fd leaks
This is WIP, there are more fds leaking.
2017-08-31 20:30:24 +02:00
Wojtek Porczyk
f1334e5095 qubes/tests/run: --break-to-repl 2017-08-31 20:30:24 +02:00
Wojtek Porczyk
25670fbd61 qubes.tests.list_templates(), and .close() the app 2017-08-31 20:30:23 +02:00
Wojtek Porczyk
8547d06cc9 qubes/tests: skip env-dependent tests using decorator
Gtk segfaults without X11 display. Some tests need dedicated PCI device.
2017-08-31 20:29:34 +02:00
Marek Marczykowski-Górecki
7a9e5c3650
tests: qrexec-policy cli tool tests 2017-08-14 02:24:32 +02:00
Marek Marczykowski-Górecki
971c7d4ac9
api/admin: add admin.vm.CreateDisposable in place of internal.vm.Create.DispVM
Add public Admin API call to create Disposable VM that would be
automatically destroyed after shutdown. Do not keep this functionality
for qrexec-policy tool only.
Also, use admin.vm.Start there, instead of internal.vm.Start and
admin.vm.Kill instead of internal.vm.CleanupDispVM (this is enough,
because DispVM now have auto_cleanup property).

QubesOS/qubes-issues#2974
2017-08-06 20:54:10 +02:00
Marek Marczykowski-Górecki
691a6f4d8c
vm/dispvm: add auto_cleanup property, unify creating new DispVM
Add auto_cleanup property, which remove DispVM after its shutdown
- this is to unify DispVM handling - less places needing special
handling after DispVM shutdown.

New DispVM inherit all settings from respective AppVM. Move this from
classmethod `DispVM.from_appvm()`, to DispVM constructor. This unify
creating new DispVM with any other VM class.
Notable exception are attached devices - because only one running VM can
have a device attached, this would prevent second DispVM started from
the same AppVM. If one need DispVM with some device attached, one can
create DispVM with auto_cleanup=False. Such DispVM will still not have
persistent storage (as any other DispVM).

Tests included.

QubesOS/qubes-issues#2974
2017-08-06 20:54:09 +02:00
Marek Marczykowski-Górecki
36f1a3abaf
Merge branch 'services'
* services:
  tests: check clockvm-related handlers
  doc: include list of extensions
  qubesvm: fix docstring
  ext/services: move exporting 'service.*' features to extensions
  app: update handling features/service os ClockVM
2017-07-29 05:09:32 +02:00
Marek Marczykowski-Górecki
34ff40c721
tests: fix AdminVM test 2017-07-29 05:07:45 +02:00
Marek Marczykowski-Górecki
c8c32d4956
Merge branch 'stats'
* stats:
  tests: admin.vm.Stats
  api: fix handling interrupted calls
  api/admin: implement admin.vm.Stats
  app: refresh getting VM statistics, rename to QubesHost.get_vm_stats
  api: use str(subject) instead of explicit subject.name
2017-07-29 05:05:32 +02:00
Marek Marczykowski-Górecki
76640df091
Merge branch 'tests-storage'
* tests-storage:
  tests: register libvirt events
  tests: even more agressive cleanup in tearDown
  app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
  api: keep track of established connections
  tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
  storage: fix Storage.clone and Storage.clone_volume
  tests: more tests fixes
  firewall: raise ValueError on invalid hostname in dsthost=
  qmemman: don't load qubes.xml
  tests: fix AdminVM test
  tests: create temporary files in /tmp
  tests: remove renaming test - it isn't supported anymore
  tests: various fixes for storage tests
  tests: fix removing LVM volumes
  tests: fix asyncio usage in some tests
  tests: minor fixes to api/admin tests
  storage/file: create -cow.img only when needed
  storage: move volume_config['source'] filling to one place
  app: do not create 'default' storage pool
  app: add missing setters for default_pool* global properties
2017-07-29 05:01:42 +02:00
Marek Marczykowski-Górecki
639fa26079
Merge branch 'qdb-watch'
* qdb-watch:
  tests: add qdb_watch test
  ext/block: make use of QubesDB watch
  vm: add API for watching changes in QubesDB
  vm: optimize imports
  api/admin: don't send internal events in admin.Events
  Add explanation why admin.vm.volume.Import is a custom script
  Follow change of qubesdb path return type
  Rename vm.qdb to vm.untrusted_qdb
2017-07-29 05:01:13 +02:00
Marek Marczykowski-Górecki
1556814f59
backup: fix handling already encoded passphrase
When passphrase is retrieved from VM, it is already encoded. Do not try
to encode it again.

QubesOS/qubes-issues#2931
2017-07-29 04:47:43 +02:00
Marek Marczykowski-Górecki
73fb16dbda
tests: check clockvm-related handlers 2017-07-28 16:32:48 +02:00
Marek Marczykowski-Górecki
9bbefa0edb
tests: admin.vm.Stats
QubesOS/qubes-issues#853
2017-07-28 03:02:33 +02:00
Marek Marczykowski-Górecki
e8b875f552
app: refresh getting VM statistics, rename to QubesHost.get_vm_stats
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.

QubesOS/qubes-issues#853
2017-07-27 22:16:03 +02:00
Marek Marczykowski-Górecki
0d9574d9fc
api: use str(subject) instead of explicit subject.name
This allows better flexibility, when subject is not necessary a VM
object.
2017-07-27 22:15:02 +02:00
Marek Marczykowski-Górecki
6a85592b51
tests: register libvirt events
This is especially needed to have proper domain-shutdown event.
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
c53582b285
tests: even more agressive cleanup in tearDown
Remove some more references to objects holding (possibly indirectly)
reference to libvirt connection:
 - local variables in tearDown function
 - running Admin API calls (especially admin.Events)
 - vmm._libvirt_conn directly, in case some reference to Qubes()
   is still there
 - any instance attribute that is an object from 'qubes' python package
   (instead of just those descending from BaseVM)
 - do not create new Qubes() instance for removing VMs - if we already
   have one in self.app

Then trigger garbage collector to really cleanup those objects (and
close relevant file descriptors). It's important do do this before
closing event loop, because some of descructors may try to use it (for
example remove registered handlers).
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
4be325a383
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
When tearDownClass is executed, event loop is already closed. Since no
test really need it right now, drop support for test class-wide VMs and
convert those methods back to instance methods.
Also put coroutines (vm.remove_from_disk, vm.kill) onto event loop.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
1af7034d57
tests: more tests fixes 2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
214c646417
firewall: raise ValueError on invalid hostname in dsthost=
...instead of AssertionError
2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
5d9bc00885
tests: fix AdminVM test
Since it is no longer child of QubesVM, constructor do not take 'qid'
and 'name' arguments.
Also:
- remove other dropped properties tests (netvm, storage related)
- make the test working in non-dom0
2017-07-26 03:26:40 +02:00
Marek Marczykowski-Górecki
a9934316c1
tests: create temporary files in /tmp 2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8bae2d39c2
tests: remove renaming test - it isn't supported anymore
QubesOS/qubes-issues#2868
2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8b2db94b41
tests: various fixes for storage tests
- improve TestPool mock - init_volume now return appropriate mock type,
   instead of TestPool
 - improve patching base directory (/var/lib/qubes) - it is stored in
   more than one place...
 - fix inheritance in TC_01_ThinPool class
 - fix expected LVM volume names ('vm-' prefix)
 - fix cleanup after FilePool tests - remove temporary qubes.xml
 - asyncio usage
 - better reporting in integ.storage - include error message in the
   report, not only as a comment in code
2017-07-26 03:26:13 +02:00
Marek Marczykowski-Górecki
a8e2f3111d
tests: fix removing LVM volumes
VM volumes have 'vm-' name prefix now.
This is continuation of fd5386c storage/lvm: prefix VM LVM volumes with 'vm-'
2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
75394a1348
tests: fix asyncio usage in some tests 2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
ba2b034843
tests: minor fixes to api/admin tests
Drop debug print, fix mocking Storage.remove (it is a coroutine).
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
4208897146
storage/file: create -cow.img only when needed
Do not create it at volume creation time. It it needed only when VM is
running, so create it just before startup only.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
c5667791e8
storage: move volume_config['source'] filling to one place
Don't set 'source' volume in various places (each VM class constructor
etc), do it as part of volume initialization. And when it needs to be
re-calculated, call storage.init_volume again.

This code was duplicated, and as usual in such a case, those copies
were different - one have set 'size', the other one not.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
3a21e1f1b3
app: do not create 'default' storage pool
Since we have app.default_pool* properties, create appropriately named
pool and let those properties choose the right pool. This also means we
don't need to specify pool name in default volume config anymore

QubesOS/qubes-issues#2256
2017-07-25 06:20:42 +02:00
Marek Marczykowski-Górecki
1933a720d0
tests: add qdb_watch test
Use dom0 instance of QubesDB, to be independent of working (or not) VMs
at this stage.

QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
0fb7c1fbed
Follow change of qubesdb path return type
It's str/unicode, not bytes now.

QubesOS/qubes-issues#2937
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
1759bca00f
Rename vm.qdb to vm.untrusted_qdb
QubesDB can be freely modified by a VM, so one should take care when
reading any data retrieved from it.

Fixes QubesOS/qubes-issues#2934
2017-07-24 13:01:55 +02:00
Marek Marczykowski-Górecki
397a8263bd
tarwriter: use new PAX format, instead of old GNU for sparse files
The old format have many issues and is discouraged by tar developers. In
this case the most important one is header with possible non-ASCII
characters, which will result in UnicodeDecodeError (tarfile module
require header parts in utf-8).
PAX format is much cleaner, as it use standard mechanism for extended
headers.
2017-07-21 03:14:06 +02:00
Marek Marczykowski-Górecki
dbea03ad1f
tests: admin.backup.*
Those tests are only about Admin API integration, not about actual backup
backend code

QubesOS/qubes-issues#2931
2017-07-20 12:19:01 +02:00
Marek Marczykowski-Górecki
801d6acf5c
Rename 'net/*' features to 'net.*'
Same reasoning as with 'service/*' - '/' is not allowed in qrexec (so
Admin API) call argument.

Related to QubesOS/qubes-issues#2906
2017-07-18 03:50:02 +02:00
Marek Marczykowski-Górecki
9e4f2d0907
Merge remote-tracking branch 'qubesos/pr/131'
* qubesos/pr/131:
  When running tests clean up and remove qubesd.socket
  Do not display gi Gtk warning when running tests
2017-07-18 02:19:34 +02:00
Marek Marczykowski-Górecki
c6e09b5a99
storage: fix resize and revert handling
- fix coroutine calling from Admin API
 - fix off-by-one error in resize
 - adjust tests
2017-07-18 01:42:37 +02:00
Marek Marczykowski-Górecki
4037bf9abc
vm/qubesvm: do not assign all host vCPUs to a qube
Fixes QubesOS/qubes-issues#2900
2017-07-17 12:33:52 +02:00
Marek Marczykowski-Górecki
ef756eb05d
tests: libvirt xml rendering unit test 2017-07-17 12:28:24 +02:00
Marek Marczykowski-Górecki
9ba84ecdad
Add code and test for migrating vm.hvm into vm.virt_mode
This will allow to load old qubes.xml - otherwise previous change render
the system unusable (until manual qubes.xml edit).

QubesOS/qubes-issues#2912
2017-07-17 12:27:17 +02:00
Marek Marczykowski-Górecki
21940bef90
Change vm.hvm into vm.virt_mode
Fixes QubesOS/qubes-issues#2912
2017-07-17 12:26:10 +02:00
Marek Marczykowski-Górecki
cbc7241a93
api/admin: rename mgmt-permission:* event to admin-permission:*
Be more consistent with Admin API name and method names.
2017-07-17 02:41:42 +02:00
Bahtiar `kalkin-` Gadimov
3be7b23f46
When running tests clean up and remove qubesd.socket
Changed the inheritance hierarchy:

1. Renamed `SystemTestsMixin` to `SystemTestCase`
2. `SystemTestCase` is a child of `QubesTestCase`
3. All classes extending the prior `SystemTestsMixin` now just extend `object`
2017-07-14 14:54:04 +02:00
Marek Marczykowski-Górecki
a42e500108
Merge branch 'tests-fixes-1'
* tests-fixes-1:
  api: extract function to make pylint happy
  tests/vm: simplify AppVM storage test
  storage: do not use deepcopy on volume configs
  api: cleanup already started servers when some later failed
  tests: fix block devices tests when running on real system
  tests: fix some FD leaks
2017-07-14 01:05:21 +02:00
Marek Marczykowski-Górecki
2df81adab1
tests/vm: simplify AppVM storage test
Use minimal TestPool(), instead of Mock().
This allow effectively compare Volume instances (init_volume with the
same parameters return equal Volume instance).
2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
d7b968d867
tests: fix block devices tests when running on real system 2017-07-12 21:33:14 +02:00
Marek Marczykowski-Górecki
7aa8d74ad4
tests: fix some FD leaks
There are still much more of them...
2017-07-12 21:33:14 +02:00
Wojtek Porczyk
b04f612374 qubes: have "service" features' keys separated by period 2017-07-05 04:16:16 +02:00
Marek Marczykowski-Górecki
dddd94b339
Merge branch 'core3-storage3' 2017-07-04 16:44:53 +02:00
Marek Marczykowski-Górecki
6db39345fb
Merge remote-tracking branch 'origin/core3-quick-fixes-20170703' 2017-07-04 16:11:18 +02:00
Marek Marczykowski-Górecki
51022cada5
Merge remote-tracking branch 'origin/policy-adminvm' 2017-07-04 15:51:30 +02:00
Marek Marczykowski-Górecki
0e554296e3
storage: drop 'internal' and 'removable' volume properties
Since dynamic volumes (qvm-block) are moved to devices API, those two
are not needed anymore.

QubesOS/qubes-issues#2256
2017-07-04 14:25:08 +02:00
Marek Marczykowski-Górecki
075d991114
config: eliminate duplicated qubes_base_dir
Remove it from system_path dict, have it in one place.
2017-07-04 14:25:07 +02:00
Marek Marczykowski-Górecki
317d140f46
storage/file: major FilePool/FileVolume cleanup and documentation
This driver isn't used in default Qubes 4.0 installation, but if we do
have it, let it follow defined API and its own documentation. And also
explicitly reject not supported operations:
 - support only revisions_to_keep<=1, but do not support revert() anyway
 (implemented version were wrong on so many levels...)
 - use 'save_on_stop'/'snap_on_start' properties directly instead of
 obsolete volume types
 - don't call sudo - qubesd is running as root
 - consistently use path, path_cow, path_source, path_source_cow

Also, add tests for BlockDevice instance returned by
FileVolume.block_device().

QubesOS/qubes-issues#2256
2017-07-04 14:25:07 +02:00
Marek Marczykowski-Górecki
8318038a73
tests: workaround asyncio.iscouroutinefunction(Mock)=True
It's already fixed in Python 3.5.3, but on Travis we have Python 3.5.2.
https://bugs.python.org/issue28703
2017-07-04 13:04:24 +02:00
Marek Marczykowski-Górecki
d00e4538bf
devices: make attach/detach related events async
This will allow starting processes and calling RPC services in those
events. This if required for usb devices, which are attached using RPC
services.
Intentionally keep device listing events synchronous only - to
discourage putting long-running actions there.

This change also require some not-async attach method version for
loading devices from qubes.xml - have `load_persistent` for this.
2017-07-04 13:04:23 +02:00
Marek Marczykowski-Górecki
ea1a04cb19
events: add support for async event handlers
See documentation for details.
2017-07-04 13:04:22 +02:00
Marek Marczykowski-Górecki
6238254f49
events: merge fire_event and fire_event_pre functions
Those functions really do very similar things - lets merge them and add
simple parameter.
2017-07-04 13:04:21 +02:00
Marek Marczykowski-Górecki
9d17c52a8d
tests: add missing import 2017-07-04 05:22:39 +02:00
Marek Marczykowski-Górecki
f393ce62dc
Fix docstrings formatting
Fix warnings/errors reported by Sphinx
2017-07-04 04:27:36 +02:00
Marek Marczykowski-Górecki
dd1e05dc83
vm: drop rename related methods
Since VM name in immutable now, drop rename-related methods.

QubesOS/qubes-issues#2868
2017-07-04 04:11:13 +02:00
Marek Marczykowski-Górecki
2dbb51a38d
exc/block: fix attaching block device exposed by dom0
Don't set <backenddomain> element in libvirt XML when device is provided
by dom0.
2017-07-04 04:09:28 +02:00
Marek Marczykowski-Górecki
3748eb3e2b
Merge branch 'core3-firewall2' 2017-07-04 03:38:59 +02:00
Marek Marczykowski-Górecki
f83c516082
Merge branch 'two-stage-clone' 2017-07-04 03:37:41 +02:00
Marek Marczykowski-Górecki
26a9974432
api/admin: split vm.volume.Clone to CloneFrom and CloneTo
The first operation returns a token, which can be passed to the second
one to actually perform clone operation. This way the caller needs have
power over both source and destination VMs (or at least appropriate
volumes), so it's easier to enforce appropriate qrexec policy.

The pending tokens are stored on Qubes() instance (as QubesAdminAPI is
not persistent). It is design choice to keep them in RAM only - those
are one time use and this way restarting qubesd is a simple way to
invalidate all of them. Otherwise we'd need some additional calls like
CloneCancel or such.

QubesOS/qubes-issues#2622
2017-06-26 13:27:48 +02:00
Marek Marczykowski-Górecki
3dcd29afea
api/admin: remove admin.vm.Clone operation
The same can be achieved with Create+volume.Clone

QubesOS/qubes-issues#2622
2017-06-26 13:10:00 +02:00
Marek Marczykowski-Górecki
a748b393f4
storage: move remove() to Volume
This is continuation of 0f12870 "storage: use direct object references,
not only identifiers".

QubesOS/qubes-issues#2256
2017-06-26 13:09:26 +02:00
Marek Marczykowski-Górecki
0200fdadcb
api/admin: firewall-related methods
In the end firewall is implemented as .Get and .Set rules, with policy
statically set to 'drop'. This way allow atomic firewall updates.

Since we already have appropriate firewall format handling in
qubes.firewall module - reuse it from there, but adjust the code to be
prepared for potentially malicious input. And also mark such variables
with untrusted_ prefix.

There is also third method: .Reload - which cause firewall reload
without making any change.

QubesOS/qubes-issues#2622
Fixes QubesOS/qubes-issues#2869
2017-06-26 13:08:27 +02:00
Marek Marczykowski-Górecki
65d15e6040
api/admin: skip firewall in vm.Clone
This operation is going to be removed, so apply a quick fix for tests.

QubesOS/qubes-issues#2622
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
842efb577d
firewall: always use policy 'drop'
There is a problem with having separate default action ("policy") and
rules because it isn't possible to set both of them atomically at the
same time.
To solve this problem, always have policy 'drop' (as a safe default),
but by default have a single rule with action 'accept'

Fixes QubesOS/qubes-issues#2869
2017-06-26 13:07:19 +02:00
Marek Marczykowski-Górecki
a6c16d00be
vm/qubesvm: fix handling empty kernel value
Do not fail on empty kernel value - which means "use booloader on
root.img".

Fixes 3ddc052 "vm: move kernel presence validation to event handler"
2017-06-26 01:59:39 +02:00
Marek Marczykowski-Górecki
4208a98bd7
Merge branch 'core3-devel-20170619' 2017-06-23 02:53:17 +02:00
Marek Marczykowski-Górecki
c1f4c219f9
tests: adjust TC_00_QubesDaemonProtocol for reorganized api module 2017-06-21 06:59:58 +02:00
Marek Marczykowski-Górecki
ea0cbe3a56
tests: improve tests for qrexe exit code handling
Check if exit code retrieved from dom0 is really the one expected.

Fix typo in test_065_qrexec_exit_code_vm (testvm1/testvm2), adjust for
reporing remote exit code and remove expectedFailure.

QubesOS/qubes-issues#2861
2017-06-21 05:23:35 +02:00
Marek Marczykowski-Górecki
a73dcf6016
tests: wait for session in tests requiring running GUI
Since tests expose qubesd socket, qvm-start-gui should handle starting
GUI daemons (so, GUI session inside VM). Add synchronization with it
using qubes.WaitForSession service.
2017-06-21 04:45:46 +02:00
Marek Marczykowski-Górecki
376ac4b32d
tests: fix vm.run_for_stdio in some more places
When test expect to wait for remote process, use vm.run_for_stdio.
Additionally, when the call fail, (stdout, stderr) is not assigned - use
the one attached to exception object instead.
2017-06-21 04:33:10 +02:00
Marek Marczykowski-Górecki
a0f616f14e
tests: fix checking exit code
Since run_for_stdio raise an exception for non-zero exit code, it isn't
ignored anymore. So, check if qrexec-client-vm return expected value,
instead of keep ignoring it.

QubesOS/qubes-issues#2861
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
72240c13b6
tests: fix vm_qrexec_gui/TC_10_Generic/test_000_anyvm_deny_dom0
When method (as expected) raise an exception, service output would not be
assigned. Extract it from exception object.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
a469c565f4
tests: fix QrexecPolicyContext
Flush new policy file to the disk, otherwise it will stay only in write
buffer.
2017-06-21 00:17:43 +02:00
Marek Marczykowski-Górecki
4241b39b94
tests: fix tests cleanup 2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
51a17ba749
tests: do not reload qubes.xml
In core3 this isn't needed anymore (and unit tests already check if
that's really true).
2017-06-21 00:17:42 +02:00
Marek Marczykowski-Górecki
ea5ca79133
tests: fix removing test VMs
Do it before shutting down qubesd socket - some things may require it
for VM removal/shutdown.
2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
eee6ab0c01
tests: use copy of qubes.xml, instead of empty one 2017-06-21 00:17:41 +02:00
Marek Marczykowski-Górecki
984a070f3e
tests: move create_*_file to SystemTestsMixin 2017-06-21 00:17:40 +02:00
Wojtek Porczyk
f56f7d13fb
tests/integ/vm_qrexec_gui: skip test_051_qrexec_simple_eof_reverse
QubesOS/qubes-issues#2851
2017-06-21 00:17:39 +02:00
Wojtek Porczyk
139f18fa1d
qubes/tests/integ/vm_qrexec_gui: some fixes 2017-06-21 00:17:39 +02:00
Wojtek Porczyk
0c0b0ea6ef
qubes/tests/integ/vm_qrexec_gui: change time.sleep to asyncio.sleep 2017-06-21 00:17:38 +02:00
Wojtek Porczyk
96a66ac6bd
qubes/api: refactor creating multiple qubesd sockets
Now there is a single function to do this, shared with tests.
2017-06-21 00:17:37 +02:00
Wojtek Porczyk
bec58fc861
qubes/tests: start qrexec policy responder for system tests 2017-06-20 13:00:20 +02:00
Wojtek Porczyk
4b8e5c3704
qubes/tests/run: refuse to run tests if qubesd is running
Test suite creates some VMs and needs to pass the knowledge about them
to qrexec policy checker. This is done using Admin API, so we need to
substitute qubesd with our own API server.
2017-06-20 13:00:20 +02:00
Wojtek Porczyk
858e547525
qubes: reorganise API protocols
Now instantiating API servers is handled by common function. This is,
among other reasons, for creating ad-hoc sockets for tests.
2017-06-20 13:00:20 +02:00
Marek Marczykowski-Górecki
9242202db2
admin: implement admin.vm.tag.*
QubesOS/qubes-issues#2622
2017-06-20 00:54:16 +02:00
Marek Marczykowski-Górecki
aadbe223c3
admin: add admin.vm.volume.Clone
QubesOS/qubes-issues#2622
2017-06-20 00:54:15 +02:00
Marek Marczykowski-Górecki
86a935e779
qubes.NotifyTools: ignore '/qubes-tools/version' completely
It isn't used for anything, so simply ignore it for good.

https://github.com/QubesOS/qubes-core-admin/pull/109#discussion_r121421409
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
55669c350c
api/misc: fix retrieving requested features from QubesDB
qdb.list return list of bytes().
2017-06-14 10:44:24 +02:00
Marek Marczykowski-Górecki
160ab964bc
vm: improve sending events for tags and features
1. Send the event after setting tag/feature
2. Provide old value for feature
3. Rename 'key' kwarg to 'feature'
2017-06-14 10:44:23 +02:00
Marek Marczykowski-Górecki
93ccb8bbda
Merge remote-tracking branch 'qubesos/pr/111'
* qubesos/pr/111:
  vm: drop 'internal' property
  qmemman: make sure to release lock
  qmemman: fix meminfo parsing for python 3
  devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
2017-06-12 23:22:55 +02:00
Marek Marczykowski-Górecki
ba4da5cbe0
Merge remote-tracking branch 'qubesos/pr/110'
* qubesos/pr/110:
  storage: use direct object references, not only identifiers
  vm: fix volume_config
  storage/lvm: prefix VM LVM volumes with 'vm-'
  storage: fix VM rename
2017-06-12 23:22:41 +02:00
Marek Marczykowski-Górecki
37245acdcf
vm: change vm.updates_available to a 'updates-available' feature 2017-06-12 12:34:23 +02:00
Marek Marczykowski-Górecki
ee442c754f
api/misc: integrate qubes.NotifyTools logic with qubes.FeaturesRequest
Make qubes.NotifyTools reuse logic of qubes.FeaturesRequest, then move
actual request processing to 'features-request' event handler. At the
same time implement handling 'qrexec' and 'gui' features request -
allowing to set template features when wasn't already there.
Behavior change: template is no longer allowed to change feature value
(regardless of being True or False). This means the user will always be
able to override what template have set.
2017-06-12 12:22:39 +02:00
Marek Marczykowski-Górecki
9d99232515
vm: drop 'internal' property
It isn't used anywhere in the code right now. And when it will be
needed, it should be a "feature" not "property".
2017-06-12 10:06:43 +02:00
Marek Marczykowski-Górecki
3564250298
devices: drop 'data' and 'frontend_domain' fields, rename 'devclass' to 'bus'
Drop DeviceInfo.data - device extension should provide a subclass with
proper individual fields.
Drop DeviceAssignment.frontend_domain - this information is redundant -
frontend domain is defined by where DeviceAssignment is attached.
Rename DeviceCollection.devclass to bus - devclass if confusing here,
because this term is also used for DeviceInfo subclass.
2017-06-12 10:06:42 +02:00
Marek Marczykowski-Górecki
0f12870803
storage: use direct object references, not only identifiers
Reference objects, not their IDs - this way when object is modified, it
is visible everywhere where it is used. Main changes:
- volume.pool - Pool object
- volume.source - Volume object

Since volume have Pool object reference now, move volume related
functions into Volume class (from Pool class). This avoids horrible
`storage.get_pool(volume).something(volume)` construct.

One issue here is since volume.source reference a Volume object from a
different VM - VM's template, now VM load order is important. Since we
don't have control over it, initialize vm.storage when needed - possibly
while initializing storage of different VM. Since we don't have cycles
in AppVM-TemplateVM dependencies, it is safe.

Also, since this commit, volume.source (if defined) always points at
volume of the same name from VM's template. Using volumes with something
else as a source is no longer supported.

QubesOS/qubes-issues#2256
2017-06-09 04:55:06 +02:00
Marek Marczykowski-Górecki
fa1da422d1
vm/adminvm: don't base AdminVM on QubesVM
dom0 isn't real VM and most properties doesn't apply to it. Lets make it
more explicit.
2017-06-06 21:57:27 +02:00
Marek Marczykowski-Górecki
67fcda2cde
tests: basic tests for AppVM
Specifically, check if root volume is updated after template switch.
2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
313d7422e7
tests: remove obsolete tests, formatting 2017-06-06 03:33:31 +02:00
Marek Marczykowski-Górecki
28737d16ce
Migrate qubes.NotifyTools, qubes.FeaturesRequest, qubes.NotifyUpdates
Make them call into qubesd. Create separate socket for "misc" calls - VM
accessible, but not part of Admin API.
2017-06-06 03:33:30 +02:00
Marek Marczykowski-Górecki
1ed0b14d93
vm: make vm.hvm=True by default
QubesOS/qubes-issues#2185
2017-06-05 23:38:30 +02:00
Marek Marczykowski-Górecki
9bb5054e50
ext: BlockDevices extension
Handle block devices exposed by VMs
2017-06-05 23:33:58 +02:00
Wojtek Porczyk
d9f5192cc2 last fixes (thanks, @marmarek!) 2017-06-05 14:34:06 +02:00
Wojtek Porczyk
dc793be81f Fixes from marmarek's review 2017-06-01 12:31:14 +02:00
Wojtek Porczyk
cb3c9a82b3 qubes/tests: fix qrexec policy context
Now it removes policy file if it was created while entering the context.
2017-06-01 12:31:14 +02:00
Wojtek Porczyk
42cbd9ff68 qubes.tests asyncio, part 2
QubesOS/qubes-issues#2622
2017-05-29 19:08:38 +02:00
Wojtek Porczyk
df03800278 Fix miscellaneous warnings
Among them:
- explicitly close files (possibly using with: syntax)
- use non-deprecated methods
2017-05-29 17:34:15 +02:00
Wojtek Porczyk
b256af3bfb qubes.tests asyncio
QubesOS/qubes-issues#2622
2017-05-29 17:34:15 +02:00
Marek Marczykowski-Górecki
607dcbaf37
admin-api: fix handling admin.vm.property.Set with None VM value
Setting VMProperty to None VM should be encoded as '' value (according
to VMProperty._none_value). But value validation rejected this value.

QubesOS/qubes-issues#2622
2017-05-26 15:08:18 +02:00
Marek Marczykowski-Górecki
3177fc504f
tests: basic tests for admin.vm.volume.Import 2017-05-26 15:08:16 +02:00
Marek Marczykowski-Górecki
6c29c653a0
tests: improve qubes_basdir_patching
File storage pool use qubes.config.qubes_base_dir, not
qubes.config.system_path, so patch that too.
2017-05-26 15:08:15 +02:00
Marek Marczykowski-Górecki
64b83fa95a
admin: raise QubesNoSuchPropertyError for non-existing properties
Accessing non-existing property is a common action (for example
hasattr() do try to access the property). So, introduce specific
exception, inheriting from AttributeError. It will behave very similar
to standard (non-Admin-API) property access.

This exception is reported to the Admin API user, so it will be possible
to distinguish between non-existing property and access denied. But it
isn't any significant information leak, as list of valid properties is
publicly available in the source code.

QubesOS/qubes-issues#853
2017-05-23 04:32:59 +02:00
Marek Marczykowski-Górecki
f93583e2be
devices: adjust XML serialization of device options
Use '<option name="option_name">option_value</option>' instead of
'<options option_name="option_value"/>'. It's more consistent with the
rest of qubes.xml - have one thing per element.

Also, add options deserialization test.
2017-05-22 17:11:20 +02:00
Marek Marczykowski-Górecki
6bc44b43de
pci: adjust PCIDeviceExtension device identifier syntax
':' is not allowed in device identifier, replace it with '_'.

Warning: this breaks existing qubes.xml
2017-05-22 17:11:19 +02:00
Marek Marczykowski-Górecki
fe37e0933e
Devices-related formating fixes
QubesOS/qubes-issues#
2017-05-22 03:21:13 +02:00
Marek Marczykowski-Górecki
f7cb8f27c5
admin: add admin.vm.Remove method
QubesOS/qubes-issues#2622
2017-05-19 18:49:21 +02:00
Marek Marczykowski-Górecki
dc2563271c
tests: very basic tests for LinuxKernel pool 2017-05-17 11:17:22 +02:00
Marek Marczykowski-Górecki
774f4c8d49
tests: admin.vm.devices.* Admin API
QubesOS/qubes-issues#2622
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
661752bbc1
devices: list persistent devices also when not currently attached
When device extension do not return some "persistent" device as
currently attached, still return it, as it will be attached at next
domain startup. User can distinguish such devices by having
frontend_domain=None (or other VM).
Also, return a set from DeviceCollection.assignments().
2017-05-16 09:05:22 +02:00
Marek Marczykowski-Górecki
f3c7fb8129
admin: add missing parts to admin.vm.Clone
Clone all the things.
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
8de82f390e
tests: Tags() and Features() 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
ba86d6da79
vm: implement tag manager to fire events on change
While at it, adjust implementation to specification: tags don't have
value, only one bit of information (present/not present).

Fixes QubesOS/qubes-issues#2686
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
68a426f0ba
admin: add methods for global properties
QubesOS/qubes-issues#2622
2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
0160d6e2f0
admin: fix handling default template in admin.vm.Create.* methods 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
fcfb2e004a
tests: more for admin.Events method 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
e598735c3b
tests: minor formatting 2017-05-16 09:05:00 +02:00
Marek Marczykowski-Górecki
625c03c504
Rename MgmtAPI to AdminAPI - part 3: Admin API
QubesOS/qubes-issues#853
2017-05-12 19:28:08 +02:00
Marek Marczykowski-Górecki
cd489f46e1
Rename MgmtAPI to AdminAPI - part 1: classes
QubesOS/qubes-issues#853
2017-05-12 19:28:07 +02:00
Marek Marczykowski-Górecki
83eef56f9d
Remove tools that are moved to -client repository
QubesOS/qubes-issues#853
2017-05-12 18:43:35 +02:00
Marek Marczykowski-Górecki
da3eed582b
tests: rearrage LVM thin tests to run at least some in devel env
This requires creating LVM volume group, so create on based on loop dev
in /tmp.
This is rather rough, but if any of this fails, run the tests anyway -
it will simply skip LVM tests.
2017-05-12 18:10:17 +02:00
Marek Marczykowski-Górecki
53be79b3b7
mgmt: vm.Create* and vm.Clone
Create methods are dynamically created based on available VM classes.

QubesOS/qubes-issues#2622
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
52c3753d61
storage: support asynchronous storage pool implementations
Allow specific pool implementation to provide asynchronous
implementation. vm.storage.* methods will detect if given implementation
is synchronous or asynchronous and will act accordingly.
Then it's up to pool implementation how asynchronous should be achieved.
Do not force it using threads (`run_in_executor()`). But pool
implementation is free to use threads, if consider it safe in a
particular case.

This commit does not touch any pool implementation - all of them are
still synchronous.

QubesOS/qubes-issues#2256
2017-05-12 17:04:18 +02:00
Marek Marczykowski-Górecki
da7496794a
events: add support for per-instance handlers 2017-05-12 14:25:32 +02:00
Marek Marczykowski-Górecki
027df9567c
mgmt: add mgmt.label.Index method
QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
55037e0446
tests: tools.qubesd
For now only QubesDaemonProtocol test. main function still untested

QubesOS/qubes-issues#2622
2017-05-12 14:25:29 +02:00
Marek Marczykowski-Górecki
0b737b3a55
mgmt: vm.features handling (mgmt.vm.feature.*)
Actual implementation and tests.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
643cef1e65
mgmt: send artificial 'connection-established' event just after connection
This allows avoid race condition between  registering event handlers and
performing some action. The important thing is the event sent after
registering event handlers in qubesd. This means state changes (like
VM start/stop) after 'connection-established' event will be included in
event stream.

QubesOS/qubes-issues#2622
2017-05-12 14:25:28 +02:00
Marek Marczykowski-Górecki
8cc0af1eda
tests: mgmt.Events
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
73df1b8ea7
tests: VM lifecycle methods tests
QubesOS/qubes-issues#2622
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
2aa0de3d5b
tools: remove qvm-ls tool and related integration in qubes.property
qvm-ls tool (as all other tools) will be accessing properties through
API, so no need (nor sense) for this tool-specific attributes in
qubes.property. The only somehow used was ls_width, and in fact it made
the output unnecessary wide.

The tool itself is already moved to core-mgmt-client repository.

QubesOS/qubes-issues#853
2017-05-12 14:25:27 +02:00
Marek Marczykowski-Górecki
50b812190b
Merge remote-tracking branch 'kalkin/device-assignments' into core3-devel
* kalkin/device-assignments: (21 commits)
  PCI extension cache PCIDevice objects
  Make pylint ♥
  Fix pylint warning no-else-return
  Fix pylint warning len-as-conditional
  device-list-attached event returns a dev/options tupples list
  DeviceAssignment options are now a dict
  Remove WrongAssignment exception
  Rename qubes.devices.BlockDevice to qubes.storage.BlockDevice
  Update relaxng devices option element
  Fix tests broken by the new assignment api
  Fix qubes.tests.devices
  Fix pci device integration tests
  qvm-device add support for assignments
  Update ext/pci to new api
  BaseVM add DeviceAssignment xml serialization
  Migrate DeviceCollection to new API
  Add PersistentCollection helper to qubes.devices
  Add DeviceAssignment
  qvm-device validates device parameters
  qvm-device fix handling of non block devices
  ...
2017-04-21 17:09:33 +02:00
Bahtiar `kalkin-` Gadimov
79407a8717
Make pylint ♥ 2017-04-15 23:50:11 +02:00
Bahtiar `kalkin-` Gadimov
9da28c9c15
device-list-attached event returns a dev/options tupples list 2017-04-15 23:49:40 +02:00
Bahtiar `kalkin-` Gadimov
5a8cc9bdd3
Remove WrongAssignment exception
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:49:39 +02:00
Bahtiar `kalkin-` Gadimov
b1b005964f
Fix tests broken by the new assignment api 2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
827ca283f3
Fix qubes.tests.devices
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Bahtiar `kalkin-` Gadimov
e84114d3a9
Fix pci device integration tests
Signed-off-by: Bahtiar `kalkin-` Gadimov <bahtiar@gadimov.de>
2017-04-15 23:48:59 +02:00
Marek Marczykowski-Górecki
0ada6d0b64
tests: disable GTK tests on travis
The environment there is way too old.

QubesOS/qubes-issues#910
2017-04-07 17:07:31 +02:00
Marek Marczykowski-Górecki
e76ede3ad0
tests: plug rpc-window tests into main test runner
QubesOS/qubes-issues#910
2017-04-07 17:07:30 +02:00
Marek Marczykowski-Górecki
564408eb3f
tests: qubespolicy tests
Fixes QubesOS/qubes-issues#2460
2017-04-06 15:43:17 +02:00
Marek Marczykowski-Górecki
3388054eae
mgmt: move property value sanitization to property definition
This also means we don't check if a VM with given name (in case of
VMProperty) exists in the system, at this stage. But this is ok, lets
not duplicate work of property setter.

QubesOS/qubes-issues#2622
2017-03-31 11:55:24 +02:00
Marek Marczykowski-Górecki
8371ffcd0f
tests: storage and labels
QubesOS/qubes-issues#2622
2017-03-16 20:04:07 +01:00
Marek Marczykowski-Górecki
2a223811a5
tests: add missing import 2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
f1c8f3220d
tests: deal with events containing dict in kwargs
If kwargs contains dict as one of values, it isn't hashable and can't be
used as value in frozenset/tuple. Convert such values into
frozenset(dict.items()). Only one (more) level is supported, but it
should be enough.

Solution from http://stackoverflow.com/a/13264725
2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
beaa0e9348
tests/mgmt: check if argument/payload is rejected when should be
Instead of creating such tests for each method separately, use unittest
subTest functionality to handle all of them at once.
2017-03-16 20:04:06 +01:00
Marek Marczykowski-Górecki
944bb26369
tests/mgmt: VM property related functions 2017-03-16 20:04:05 +01:00
Marek Marczykowski-Górecki
2d2672ec58
vm/qubesvm: convert firewall_conf into dumb, read-only property
Don't allow anything else than firewall.xml.
2017-03-16 20:04:03 +01:00
Marek Marczykowski-Górecki
a6c7da6061
tests: be even more defensive on cleaning up VMs
Don't fail even if qubes-test.xml do not load at all because of syntax
error - for example empty file.
2017-02-27 02:37:52 +01:00
Marek Marczykowski-Górecki
45709b510a
backup: minor fixes after bringing back scrypt support 2017-02-27 02:37:51 +01:00
Marek Marczykowski-Górecki
13fc810363
tests: some more fixes for core3 API 2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
3ecc0a9bcb
tests: improve devices API unit test
Check fired events - inspired by qvm-device test.
2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
7f2ca33774
tests: fix importing template in non-default pool 2017-02-27 02:37:50 +01:00
Marek Marczykowski-Górecki
9ace4e66f1
tests: more py3k related fixes 2017-02-27 02:37:49 +01:00
Marek Marczykowski-Górecki
cae68f64ca
tests: just one more test in vm_qrexec_gui to core3 API 2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
3f29345d32
tests/storage: read-only volume should not have save_on_stop=True 2017-02-27 02:37:48 +01:00
Marek Marczykowski-Górecki
9cad353939
tests: py3k related fixes - bytes/str
Adjust usage of bytes vs str type.
2017-02-27 02:35:49 +01:00
Wojtek Porczyk
be53db4db9 qubes/events: they accept only keyword arguments
Positional arguments are hereby deprecated, with immediate effect.

QubesOS/qubes-issues#2622
2017-02-21 14:46:42 +01:00
Marek Marczykowski-Górecki
bd9300b38e
tests: copy pool configuration into qubes-test.xml
If template choosen for the tests is installed in non-default storage
pool, this pool also needs to be imported into qubes-test.xml.
2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
c3fc4062d8
tests: add basic test for qvm-features 2017-02-15 00:01:33 +01:00
Marek Marczykowski-Górecki
01aedb7f18
storage: fix handling snap_on_start=True file volumes
Use the right cow image and apply the second layer to provide read-write
access. The correct setup is:
 - base image + base cow -> read-only snapshot (base changes "cached"
   until committed)
 - read-only snapshot + VM cow -> read-write snapshot (changes discarded
   after VM shutdown)

This way, even VM without Qubes-specific startup scripts will can
benefit from Template VMs, while VMs with Qubes-specific startup scripts
may still see original root.img content (for possible signature
verification, when storage domain got implemented).

QubesOS/qubes-issues#2256
2017-02-14 23:59:07 +01:00
Marek Marczykowski-Górecki
48f78dfbc8
tests: check if snap_on_start=True volumes are not persistent
Content should be reset back to base volume at each VM startup.
Disposable VMs depend on this behaviour.

QubesOS/qubes-issues#2256
2017-02-14 23:59:07 +01:00
Wojtek Porczyk
1be75d9c83 misc python3 fixes 2017-02-07 17:07:52 +01:00
Wojtek Porczyk
d74567d65f qubes: port core to python3
fixes QubesOS/qubes-issues#2074
2017-01-20 16:42:51 +01:00
Wojtek Porczyk
68ad60c1b3 Merge remote-tracking branch 'origin/master' into core3-devel
Conflicts:
	core/qubes.py
	doc/Makefile
	doc/manpages/qvm-prefs.rst
	doc/qvm-tools/qvm-add-appvm.rst
	doc/qvm-tools/qvm-backup-restore.rst
	doc/qvm-tools/qvm-backup.rst
	doc/qvm-tools/qvm-block.rst
	doc/qvm-tools/qvm-clone.rst
	doc/qvm-tools/qvm-firewall.rst
	doc/qvm-tools/qvm-ls.rst
	doc/qvm-tools/qvm-pci.rst
	doc/qvm-tools/qvm-run.rst
	doc/qvm-tools/qvm-shutdown.rst
	doc/qvm-tools/qvm-start.rst
	doc/qvm-tools/qvm-template-commit.rst
	qvm-tools/qvm-ls
	qvm-tools/qvm-prefs
	qvm-tools/qvm-remove
	tests/__init__.py
	vm-config/xen-vm-template.xml
2016-11-30 03:07:39 +01:00
Wojtek Porczyk
37bfd0d2a3 Merge remote-tracking branch 'marmarek/core3-fake-ip' into core3-devel 2016-11-15 17:40:30 +01:00
Wojtek Porczyk
a4d50409df Merge remote-tracking branch 'marmarek/core3-storage-fixes' into core3-devel 2016-11-15 17:36:53 +01:00
Wojtek Porczyk
fd953f4f27 Merge remote-tracking branch 'marmarek/core3-backup2' into core3-devel 2016-11-15 17:34:12 +01:00
Marek Marczykowski-Górecki
b011cef8af
tests/storage: add tests for basic volumes properties
Things like if read-only volume is really read-only, volatile is
volatile etc.

QubesOS/qubes-issues#2256
2016-11-04 14:18:56 +01:00
Marek Marczykowski-Górecki
ec81b3046f
tests: add missing app.save() before starting a domain
Otherwise domain will be unknown to other processes (like qrexec
services).
2016-11-01 00:28:37 +01:00
Marek Marczykowski-Górecki
d999d91049
tests: few more tests for fake/custom IP
QubesOS/qubes-issues#1143
QubesOS/qubes-issues#1477
2016-10-31 03:39:46 +01:00
Marek Marczykowski-Górecki
ea33fef9cc
tests: drop dispvm_netvm tests
This property no longer exists in core3.
2016-10-31 03:10:12 +01:00
Marek Marczykowski-Górecki
5072acc8f2
tests: custom VM IP
QubesOS/qubes-issues#1477
2016-10-31 03:09:45 +01:00
Marek Marczykowski-Górecki
4585f2b503
tests: add tests for fake IP feature
QubesOS/qubes-issues#1143
2016-10-31 02:17:21 +01:00
Marek Marczykowski-Górecki
8cf19e3c92
tests/backupcompatibility: verify restored VM properties 2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
64ac7f6e8d
tests/backup: check non-ASCII passphrase
QubesOS/qubes-issues#2398
2016-10-28 11:53:34 +02:00
Marek Marczykowski-Górecki
fc00dd211e
tests/backup: test backup with non-ASCII passphrase 2016-10-28 11:53:33 +02:00
Marek Marczykowski-Górecki
6ee200236c
tests/backup: verify migration into LVM thin pool 2016-10-28 11:53:31 +02:00
Marek Marczykowski-Górecki
673fe4423a
tests: handle LVM thin pool 2016-10-28 11:53:31 +02:00
Marek Marczykowski-Górecki
fbecd08a58
tests/backup: exclude some VMs during restore 2016-10-28 11:53:31 +02:00
Wojtek Porczyk
5e62d3f7cb qubes/tests: substitute_entry_points
New context manager for temporary overriding entry point groups.

fixes QubesOS/qubes-issues#2111
2016-10-24 15:16:39 +02:00
Wojtek Porczyk
4c73c1b93a More green paint 2016-10-19 16:09:58 +02:00
Marek Marczykowski-Górecki
5babb68031
tests/backupcompatibility: verify if all files got restored
There is still no verification of disk images content, nor VM
properties...
2016-10-19 01:54:44 +02:00
Wojtek Porczyk
8097da7cab Paint the project green for testbench launch 2016-10-18 19:07:20 +02:00
Wojtek Porczyk
c81346ba51 qubes/test/run: Print tracebacks of expected fails
fixes QubesOS/qubes-issues#2376
2016-10-14 17:20:14 +02:00
Wojtek Porczyk
526f2c3751 Merge remote-tracking branch 'marmarek/core3-backup' into core3-devel 2016-10-14 15:29:37 +02:00
Wojtek Porczyk
c6c0a545e6 Merge remote-tracking branch 'origin/pull/58/head' into core3-devel 2016-10-11 11:37:15 +02:00
Marek Marczykowski-Górecki
36eb7f923f
qubes/tarwriter: add simple sparse-tar writer module
tar can't write archive with _contents_ of block device. We need this to
backup LVM-based disk images. To avoid dumping image to a file first,
create a simple tar archiver just for this purpose.

Python is not the fastest possible technology, it's 3 times slower than
equivalent written in C. But it's much easier to read, much less
error-prone, and still process 1GB image under 1s (CPU time, leaving
along actual disk reads). So, it's acceptable.
2016-10-05 01:54:41 +02:00
Marek Marczykowski-Górecki
c4632d6be8
tests/backup: test idea 2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
f2d79b9379
tests/backup: use round volume size
When handling LVM volumes, size must be multiply of 4MB.
2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
e938aa61ab
tests: cleanup test LVM volumes
Handle the case when vm.remove_from_disk does not cleanup all the
things.
2016-09-29 02:08:30 +02:00
Marek Marczykowski-Górecki
0a35bd06aa
backup: support relocating files to different storage pool
To ease all this, rework restore workflow: first create QubesVM objects,
and all their files (as for fresh VM), then override them with data
from backup - possibly redirecting some files to new location. This
allows generic code to create LVM volumes and then only restore its
content.
2016-09-29 02:08:29 +02:00
Marek Marczykowski-Górecki
4d45dd5549
tests/backup: check backup+restore of LVM based VM
The test fails for now...
2016-09-29 01:59:55 +02:00
Marek Marczykowski-Górecki
226695534b
tests/backup: handle non-default pool in BackupTestsMixin 2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
e1d9de1cc2
tests/backup: minor fix for python3 2016-09-29 01:59:54 +02:00
Marek Marczykowski-Górecki
6d5959b31d
tests/backup: use proper logging instead of print 2016-09-29 01:59:53 +02:00
Marek Marczykowski-Górecki
016c3d8e88
tests/backup: check restored disk images 2016-09-29 01:59:53 +02:00
Bahtiar `kalkin-` Gadimov
8d9b6f19fd
Add catch-all '*' to event handlers 2016-09-28 21:28:43 +02:00
Marek Marczykowski-Górecki
e499b529ad
tests: move BackupTestMixin to qubes.tests.int.backup
This is much more logical place, don't pollute main qubes.tests module.
2016-09-25 16:31:31 +02:00
Marek Marczykowski-Górecki
d4f29bb8b7
qubes: add qvm-firewall tool
Fixes QubesOS/qubes-issues#1815
2016-09-19 20:36:32 +02:00
Marek Marczykowski-Górecki
202042bd8d
tests: update for new firewall API
QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
e01f7b97d9
qubes/vm: plug in new firewall code, create QubesDB entries
QubesOS/qubes-issues#1815
2016-09-19 20:36:31 +02:00
Marek Marczykowski-Górecki
1da75a676f
qubes/firewall: new firewall interface
First part - handling firewall.xml and rules formatting.
Specification on https://qubes-os.org/doc/vm-interface/

TODO (for dom0):
 - plug into QubesVM object
 - expose rules in QubesDB (including reloading)
 - drop old functions (vm.get_firewall_conf etc)

QubesOS/qubes-issues#1815
2016-09-19 20:36:30 +02:00
Marek Marczykowski-Górecki
0ea7afd245
tests: adjust LVM tests for non-default VG name 2016-09-19 20:35:25 +02:00
Marek Marczykowski-Górecki
9ae3dc2ebf
tests/qvm-device: tolerate different 'column' tool versions
Newer one do output column even if empty, but older one (in Travis-CI
env) do not. Ignore trailing spaces to work with both version.
2016-09-13 02:01:22 +02:00
Marek Marczykowski-Górecki
16db68b053
Merge branch 'core3-devices' into core3-devel
* core3-devices:
  Fix core2migration and tests for new devices API
  tests: more qubes.devices tests
  qubes/ext/pci: implement pci-no-strict-reset/BDF feature
  qubes/tools: allow calling qvm-device as qvm-devclass (like qvm-pci)
  qubes: make pylint happy
  qubes/tools: add qvm-device tool (and tests)
  tests: load qubes.tests.tools.qvm_ls
  tests: PCI devices tests
  tests: add context manager to catch stdout
  qubes/ext/pci: move PCI devices handling to an extension
  qubes/devices: use more detailed exceptions than just KeyError
  qubes/devices: allow non-persistent attach
  qubes/storage: misc fixes for VM-exposed block devices handling
  qubes: new devices API

Fixes QubesOS/qubes-issues#2257
2016-09-04 21:32:01 +02:00
Marek Marczykowski-Górecki
5cf15ef3da
tests: improve DispVM error message 2016-09-04 21:00:02 +02:00
Marek Marczykowski-Górecki
11191ea694
Fix core2migration and tests for new devices API
QubesOS/qubes-issues#2257
2016-09-04 20:55:07 +02:00
Marek Marczykowski-Górecki
e536bcca30
tests: more qubes.devices tests
QubesOS/qubes-issues#2257
2016-09-03 20:41:06 +02:00
Marek Marczykowski-Górecki
93e88e0c22
qubes/ext/pci: implement pci-no-strict-reset/BDF feature
Instead of old per-VM flag 'pci_strictreset', now implement this as
per-device flag using features. To not fail on particular device
assignment set 'pci-no-strict-reset/DEVICE-BDF' to True. For
example 'pci-no-strict-reset/00:1b.0'.

QubesOS/qubes-issues#2257
2016-09-03 20:41:06 +02:00
Marek Marczykowski-Górecki
5c7f589330
qubes: make pylint happy 2016-09-03 20:41:05 +02:00
Marek Marczykowski-Górecki
e8d011b83f
qubes/tools: add qvm-device tool (and tests)
Add a tool to manipulate various devices.

QubesOS/qubes-issues#2257
2016-09-03 20:41:05 +02:00
Marek Marczykowski-Górecki
6aae6863b0
tests: load qubes.tests.tools.qvm_ls
The test module was implemented, but not loaded
2016-09-03 20:41:04 +02:00
Marek Marczykowski-Górecki
fbb086aed5
tests: PCI devices tests
Just basic things for now.

QubesOS/qubes-issues#2257
2016-09-03 20:41:04 +02:00
Marek Marczykowski-Górecki
4bfb523974
tests: add context manager to catch stdout
This will avoid code duplication in tools tests.
2016-09-03 20:41:03 +02:00
Marek Marczykowski-Górecki
aa67a4512e
qubes/ext/pci: move PCI devices handling to an extension
Implement required event handlers according to documentation in
qubes.devices.

A modification of qubes.devices.DeviceInfo is needed to allow dynamic,
read-only properties.

QubesOS/qubes-issues#2257
2016-09-03 20:41:03 +02:00
Marek Marczykowski-Górecki
d7a3c0d319
qubes: new devices API
Allow device plugin to list attached and available devices. Enforce
at API level every device being exposed by some domain.

This commit only changes devices API, but not update existing users
(pci) yet.

QubesOS/qubes-issues#2257
2016-09-03 20:40:39 +02:00
Marek Marczykowski-Górecki
0293c1c7ef
qubes/vm: move misc XML tags loading to separate method 2016-08-18 11:57:11 +02:00
Marek Marczykowski-Górecki
d8286b6dab
tests: port vm_qrexec_gui to core3 API 2016-08-17 22:41:57 +02:00
Marek Marczykowski-Górecki
e19df4c5b6
tests: port dispvm tests to core3 API
Some tests do not apply, as there is no savefile and attributes
propagation is much simpler. Dropped tests:
 - test_000_firewall_propagation
 - test_001_firewall_propagation
 - test_000_prepare_dvm

QubesOS/qubes-issues#2253
2016-08-17 04:15:35 +02:00
Marek Marczykowski-Górecki
6b546ca1e6
tests/backupcompatibility: no errors are expected on restoring R1 backup
Fallback to default kernel is enabled by default.
2016-08-17 01:31:33 +02:00
Marek Marczykowski-Górecki
690727fef8
tests/backup: remove VMs in reversed order
One place had missing `reversed()`.
2016-08-17 01:30:34 +02:00
Marek Marczykowski-Górecki
dd93650ea7
tests/backup: fix handling different VM types
Not all VM types have 'template' property.
2016-08-17 00:49:53 +02:00
Marek Marczykowski-Górecki
660c726315
tests: add qvm-check 2016-08-09 05:14:13 +02:00
Marek Marczykowski-Górecki
96d032919a
tests: minor fix to revert changes tests, mark as expected failure
While the test should be ok now, the functionality itself doesn't work (at
least on file volumes).
2016-08-09 05:14:12 +02:00
Marek Marczykowski-Górecki
ac68ffe39f
tests: correctly compare DeviceManager objects
Usefull for assertEqual(vm1.devices, vm2.devices) - clone tests, backup
tests etc.
2016-08-09 05:14:11 +02:00
Marek Marczykowski-Górecki
ec7a84ea9d
minor: formatting
QubesOS/qubes-issues#
2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
af7b4087f0
tests: port few more tests to core3 API 2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
faa341ffa2
tests/backup: adjust for core3 API properties to verify after restore 2016-08-09 05:14:10 +02:00
Marek Marczykowski-Górecki
7d1cbd6934
tests: handle per-template tests in BackupTestsMixin 2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
fba6eac07f
qubes/tests: do not undefine libvirt domain twice
Do not access vm.libvirt_domain after it being already removed - this
will redefine it again in libvirt, just to undefine it in a moment.
On the other hand, few lines below there is fallback libvirt cleanup, in
case of proper one not working.
2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
6d45b97357
tests: workaround 'int' name being shadowed by module name
qubes.tests.int shadows 'int' type in qubes.tests module.

QubesOS/qubes-issues#2241
2016-08-09 05:14:09 +02:00
Marek Marczykowski-Górecki
4e022382a5
Merge remote-tracking branch 'origin/master' into core3-devel 2016-08-08 00:11:46 +02:00
Bahtiar `kalkin-` Gadimov
7d11a7afbd
Add lvm tests 2016-08-03 02:57:41 +02:00
Wojtek Porczyk
f915115cfb Merge remote-tracking branch 'woju/pull/25/head' into core3-devel 2016-07-21 19:38:58 +02:00
Wojtek Porczyk
36e5bcd766 Merge remote-tracking branch 'origin/pull/39/head' into core3-devel 2016-07-21 16:43:25 +02:00
Bahtiar `kalkin-` Gadimov
71161bfb93
Add mock libvirt objects 2016-07-13 23:28:02 +02:00
Bahtiar `kalkin-` Gadimov
d380fb4aba
qubes.tests.app make pylint happy ♥ 2016-07-13 23:08:59 +02:00
Bahtiar `kalkin-` Gadimov
23e52775b6
qubes.tests.init make pylint happy ♥ 2016-07-13 23:08:58 +02:00
Bahtiar `kalkin-` Gadimov
61feb0ced7
Migrate backup to new storage api 2016-07-13 22:34:37 +02:00
Bahtiar `kalkin-` Gadimov
d1c606b952
qubes.storage.file use new storage API 2016-07-13 22:34:33 +02:00
Bahtiar `kalkin-` Gadimov
ca9797bb6b
qubes.tests.int.basic use new storage API 2016-07-13 22:34:32 +02:00
Bahtiar `kalkin-` Gadimov
4cc7b8d2a8
Fix qubes.tests.storage 2016-07-13 22:21:31 +02:00
Bahtiar `kalkin-` Gadimov
774cbd499e
Add TestVM.is_halted() to fix failing tests 2016-07-13 21:57:12 +02:00
Wojtek Porczyk
c899d1f960 qubes/tests: Show errors while loading external tests
Now failure to load external tests shows in which entry point the error
happened and a useful traceback. The traceback extends from the "try"
statement down to the actual error line, but it does not include the
frames above, ie. from the invocation to the load_tests routine. This is
a limitation of Python itself and usually not a problem.
2016-07-04 16:51:35 +02:00
Wojtek Porczyk
d28ab908cc Network and storage fixes 2016-06-29 16:42:35 +02:00
Bahtiar `kalkin-` Gadimov
e08ca1ff57
Remove QubesVM.is_template() 2016-06-23 14:23:31 +02:00
Wojtek Porczyk
66f46d8a25 Fix test error after bdfb85ac (refactor storage) 2016-06-21 10:10:45 +02:00
Wojtek Porczyk
d6ad8d34a6 Merge remote-tracking branch 'marmarek/patch-1' into core3-devel 2016-06-16 21:46:53 +02:00