Commit Graph

5621 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
76640df091
Merge branch 'tests-storage'
* tests-storage:
  tests: register libvirt events
  tests: even more agressive cleanup in tearDown
  app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
  api: keep track of established connections
  tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
  storage: fix Storage.clone and Storage.clone_volume
  tests: more tests fixes
  firewall: raise ValueError on invalid hostname in dsthost=
  qmemman: don't load qubes.xml
  tests: fix AdminVM test
  tests: create temporary files in /tmp
  tests: remove renaming test - it isn't supported anymore
  tests: various fixes for storage tests
  tests: fix removing LVM volumes
  tests: fix asyncio usage in some tests
  tests: minor fixes to api/admin tests
  storage/file: create -cow.img only when needed
  storage: move volume_config['source'] filling to one place
  app: do not create 'default' storage pool
  app: add missing setters for default_pool* global properties
2017-07-29 05:01:42 +02:00
Marek Marczykowski-Górecki
639fa26079
Merge branch 'qdb-watch'
* qdb-watch:
  tests: add qdb_watch test
  ext/block: make use of QubesDB watch
  vm: add API for watching changes in QubesDB
  vm: optimize imports
  api/admin: don't send internal events in admin.Events
  Add explanation why admin.vm.volume.Import is a custom script
  Follow change of qubesdb path return type
  Rename vm.qdb to vm.untrusted_qdb
2017-07-29 05:01:13 +02:00
Marek Marczykowski-Górecki
46bfa14d08
storage/lvm: make sure that volumes have /dev entries when export() is called
In some cases (even manual manipulation) those files may not be present
- especially because thin snapshots do not have nodes in /dev by default.
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
27ce27b6e9
backup: improve error reporting about passphrase retrieval
QubesOS/qubes-issues#2931
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
bde8d45cca
backup: update last backup timestamp only when backup succeed
QubesOS/qubes-issues#2931
2017-07-29 04:47:44 +02:00
Marek Marczykowski-Górecki
1556814f59
backup: fix handling already encoded passphrase
When passphrase is retrieved from VM, it is already encoded. Do not try
to encode it again.

QubesOS/qubes-issues#2931
2017-07-29 04:47:43 +02:00
Marek Marczykowski-Górecki
73fb16dbda
tests: check clockvm-related handlers 2017-07-28 16:32:48 +02:00
Marek Marczykowski-Górecki
3c052d9e3c
doc: include list of extensions 2017-07-28 16:32:48 +02:00
Marek Marczykowski-Górecki
5321e11003
qubesvm: fix docstring 2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
1a6728cb12
ext/services: move exporting 'service.*' features to extensions
This means core code will not publish any features by default.
2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
e738a7596d
app: update handling features/service os ClockVM
Threis no more ntpd service used - new approach do not conflict with
ntpd. Because of this, new feature is named 'service.clocksync', and
should be _enabled_ in ClockVM ('ntpd' was disabled there).

QubesOS/qubes-issues#1230
2017-07-28 16:32:47 +02:00
Marek Marczykowski-Górecki
9bbefa0edb
tests: admin.vm.Stats
QubesOS/qubes-issues#853
2017-07-28 03:02:33 +02:00
Marek Marczykowski-Górecki
2f4b4d97e7
api: fix handling interrupted calls
When an API call is interrupted, the relevant coroutine is cancelled -
which means it may throw CancelledError. At the same time, cancelled
call have related socket already closed (and transport set to None). But
QubesDaemonProtocol.respond try to close the transport again, which
fails. Fix handling this case.
2017-07-28 03:02:33 +02:00
Marek Marczykowski-Górecki
147bca1648
api/admin: implement admin.vm.Stats
QubesOS/qubes-issues#853
2017-07-28 03:02:32 +02:00
Marek Marczykowski-Górecki
e8b875f552
app: refresh getting VM statistics, rename to QubesHost.get_vm_stats
Get a VM statistics once. If previous measurements are provided,
calculate difference too. This is backend part of upcoming
admin.vm.Stats service.

QubesOS/qubes-issues#853
2017-07-27 22:16:03 +02:00
Marek Marczykowski-Górecki
0d9574d9fc
api: use str(subject) instead of explicit subject.name
This allows better flexibility, when subject is not necessary a VM
object.
2017-07-27 22:15:02 +02:00
Marek Marczykowski-Górecki
6a85592b51
tests: register libvirt events
This is especially needed to have proper domain-shutdown event.
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
c53582b285
tests: even more agressive cleanup in tearDown
Remove some more references to objects holding (possibly indirectly)
reference to libvirt connection:
 - local variables in tearDown function
 - running Admin API calls (especially admin.Events)
 - vmm._libvirt_conn directly, in case some reference to Qubes()
   is still there
 - any instance attribute that is an object from 'qubes' python package
   (instead of just those descending from BaseVM)
 - do not create new Qubes() instance for removing VMs - if we already
   have one in self.app

Then trigger garbage collector to really cleanup those objects (and
close relevant file descriptors). It's important do do this before
closing event loop, because some of descructors may try to use it (for
example remove registered handlers).
2017-07-26 03:27:00 +02:00
Marek Marczykowski-Górecki
d57120d809
app: do not wrap libvirt_conn.close() in auto-reconnect wrapper
close() function purpose is to close the connection...
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
074c705f77
api: keep track of established connections
This will be needed to gracefuly cleanup them between individual tests.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
4be325a383
tests: drop VM cleanup from tearDownClass, fix asyncio usage in tearDown
When tearDownClass is executed, event loop is already closed. Since no
test really need it right now, drop support for test class-wide VMs and
convert those methods back to instance methods.
Also put coroutines (vm.remove_from_disk, vm.kill) onto event loop.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
910f793c61
storage: fix Storage.clone and Storage.clone_volume
1. Volume.import_volume expect volume to be created first.
2. asyncio.wait do not accept generator, it must be concrete iterable.
2017-07-26 03:26:59 +02:00
Marek Marczykowski-Górecki
1af7034d57
tests: more tests fixes 2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
214c646417
firewall: raise ValueError on invalid hostname in dsthost=
...instead of AssertionError
2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
625c94d1f6
qmemman: don't load qubes.xml
Only qubesd should load qubes.xml directly. Put a TODO comments for now
in place of slow VM reporting, invent some better mechanism later.

This loading of qubes.xml caused deadlocks, because qmemnan kept open
file descriptor (in locked state).
2017-07-26 03:26:58 +02:00
Marek Marczykowski-Górecki
5d9bc00885
tests: fix AdminVM test
Since it is no longer child of QubesVM, constructor do not take 'qid'
and 'name' arguments.
Also:
- remove other dropped properties tests (netvm, storage related)
- make the test working in non-dom0
2017-07-26 03:26:40 +02:00
Marek Marczykowski-Górecki
a9934316c1
tests: create temporary files in /tmp 2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8bae2d39c2
tests: remove renaming test - it isn't supported anymore
QubesOS/qubes-issues#2868
2017-07-26 03:26:39 +02:00
Marek Marczykowski-Górecki
8b2db94b41
tests: various fixes for storage tests
- improve TestPool mock - init_volume now return appropriate mock type,
   instead of TestPool
 - improve patching base directory (/var/lib/qubes) - it is stored in
   more than one place...
 - fix inheritance in TC_01_ThinPool class
 - fix expected LVM volume names ('vm-' prefix)
 - fix cleanup after FilePool tests - remove temporary qubes.xml
 - asyncio usage
 - better reporting in integ.storage - include error message in the
   report, not only as a comment in code
2017-07-26 03:26:13 +02:00
Marek Marczykowski-Górecki
a8e2f3111d
tests: fix removing LVM volumes
VM volumes have 'vm-' name prefix now.
This is continuation of fd5386c storage/lvm: prefix VM LVM volumes with 'vm-'
2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
75394a1348
tests: fix asyncio usage in some tests 2017-07-26 03:26:12 +02:00
Marek Marczykowski-Górecki
ba2b034843
tests: minor fixes to api/admin tests
Drop debug print, fix mocking Storage.remove (it is a coroutine).
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
4208897146
storage/file: create -cow.img only when needed
Do not create it at volume creation time. It it needed only when VM is
running, so create it just before startup only.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
c5667791e8
storage: move volume_config['source'] filling to one place
Don't set 'source' volume in various places (each VM class constructor
etc), do it as part of volume initialization. And when it needs to be
re-calculated, call storage.init_volume again.

This code was duplicated, and as usual in such a case, those copies
were different - one have set 'size', the other one not.

QubesOS/qubes-issues#2256
2017-07-26 03:26:11 +02:00
Marek Marczykowski-Górecki
3a21e1f1b3
app: do not create 'default' storage pool
Since we have app.default_pool* properties, create appropriately named
pool and let those properties choose the right pool. This also means we
don't need to specify pool name in default volume config anymore

QubesOS/qubes-issues#2256
2017-07-25 06:20:42 +02:00
Marek Marczykowski-Górecki
73a78ca0db
app: add missing setters for default_pool* global properties
Function itself was already written, but not set in the properties.

QubesOS/qubes-issues#2256
2017-07-25 05:21:32 +02:00
Marek Marczykowski-Górecki
1933a720d0
tests: add qdb_watch test
Use dom0 instance of QubesDB, to be independent of working (or not) VMs
at this stage.

QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
67c382c8b0
ext/block: make use of QubesDB watch
Actually use just introduced API.

Also document new `device-list-change:class` event.

QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
b7f0cf7d82
vm: add API for watching changes in QubesDB
Provide an API for use QubesDB.watch() inside of qubesd.

Fixes QubesOS/qubes-issues#2940
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
d5b94d1cbd
vm: optimize imports
Remove unused, sort others.
2017-07-25 05:20:39 +02:00
Marek Marczykowski-Górecki
354aa14c53
api/admin: don't send internal events in admin.Events
Some events are internal for a sole purpose of getting some data from
extension. Since listeners of admin.Events cannot return anything, there
is no sense in sending those events there.
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
b25cf4b085
Add explanation why admin.vm.volume.Import is a custom script
Cc @rootkovska
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
0fb7c1fbed
Follow change of qubesdb path return type
It's str/unicode, not bytes now.

QubesOS/qubes-issues#2937
2017-07-25 05:20:38 +02:00
Marek Marczykowski-Górecki
1759bca00f
Rename vm.qdb to vm.untrusted_qdb
QubesDB can be freely modified by a VM, so one should take care when
reading any data retrieved from it.

Fixes QubesOS/qubes-issues#2934
2017-07-24 13:01:55 +02:00
Marek Marczykowski-Górecki
cdbe1d14b3
rpm: include directory for backup profiles
QubesOS/qubes-issues#2931
2017-07-21 03:16:57 +02:00
Marek Marczykowski-Górecki
397a8263bd
tarwriter: use new PAX format, instead of old GNU for sparse files
The old format have many issues and is discouraged by tar developers. In
this case the most important one is header with possible non-ASCII
characters, which will result in UnicodeDecodeError (tarfile module
require header parts in utf-8).
PAX format is much cleaner, as it use standard mechanism for extended
headers.
2017-07-21 03:14:06 +02:00
Marek Marczykowski-Górecki
abdad8c2b2
backup: allow backup running VMs
Since we have LVM by default, it is possible to backup VMs while they
are running. For now it will include its state from before startup, but
later we may implement some other logic (a snapshot of running VM).
2017-07-21 03:12:23 +02:00
Marek Marczykowski-Górecki
1ec0060577
backup: check inner archiver exit code
If inner tar returns with non-zero code, mostly like the backup is
incomplete, so it is very important to report it.
2017-07-21 03:11:02 +02:00
Marek Marczykowski-Górecki
82918ee704
api/admin: gracefuly report cancelled backup operation
QubesOS/qubes-issues#2931
2017-07-21 03:10:26 +02:00
Marek Marczykowski-Górecki
088c838ac3
storage: typo fix in comment 2017-07-20 12:19:02 +02:00