Commit Graph

6031 Commits

Author SHA1 Message Date
Rusty Bird
86998b0815
lvm_thin: really remove revision 2018-02-16 21:47:33 +00:00
Rusty Bird
9f468f93f5
lvm_thin: fill in volume's revisions_to_keep from pool
Otherwise, the base class defaults revisions_to_keep to 0.
2018-02-16 21:47:32 +00:00
Marek Marczykowski-Górecki
7c50bd5104
qubespolicy: unify calling qrexec service
Rely on qrexec-client resolving QUBESRPC keyword, same as in case of VM
call. This will allow applying special treatment to such calls, like
calling qubes-rpc-multiplexer directly (avoiding shell), because we have
defined protocol what can be used here.
2018-02-16 04:30:32 +01:00
Marek Marczykowski-Górecki
b00bbb73e4
Merge remote-tracking branch 'qubesos/pr/190'
* qubesos/pr/190:
  Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90
  replaced underscore by dash and update test accordingly
  Updated assert content for test_620_qdb_standalone in TC_90_QubesVM
  Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372
2018-02-14 01:29:08 +01:00
Marek Marczykowski-Górecki
e3d3e149f0
Fix too long line 2018-02-13 11:27:59 +01:00
Marek Marczykowski-Górecki
14c6aad153
version 4.0.22 2018-02-13 05:30:11 +01:00
Marek Marczykowski-Górecki
209af07fd0
Merge remote-tracking branch 'qubesos/pr/188'
* qubesos/pr/188:
  file-reflink, a storage driver optimized for CoW filesystems
  Make AppVM/DispVM root volume rw to avoid CoW-on-CoW
2018-02-13 05:20:52 +01:00
Marek Marczykowski-Górecki
5afd5fb04f
Merge remote-tracking branch 'qubesos/pr/191'
* qubesos/pr/191:
  qubes/tests: fix QubesTestCase.qrexec_policy
2018-02-13 05:20:39 +01:00
Marek Marczykowski-Górecki
717bc4ace3
vm/appvm: forbid changing template if the are children DispVMs
Changing AppVM's template will not update root volume reference in
DispVMs based on it. For now forbid such change.

Fixes QubesOS/qubes-issues#3576
2018-02-13 04:53:39 +01:00
Rusty Bird
1695a732b8
file-reflink, a storage driver optimized for CoW filesystems
This adds the file-reflink storage driver. It is never selected
automatically for pool creation, especially not the creation of
'varlibqubes' (though it can be used if set up manually).

The code is quite small:

               reflink.py  lvm.py      file.py + block-snapshot
    sloccount  334 lines   447 (134%)  570 (171%)

Background: btrfs and XFS (but not yet ZFS) support instant copies of
individual files through the 'FICLONE' ioctl behind 'cp --reflink'.
Which file-reflink uses to snapshot VM image files without an extra
device-mapper layer. All the snapshots are essentially freestanding;
there's no functional origin vs. snapshot distinction.

In contrast to 'file'-on-btrfs, file-reflink inherently avoids
CoW-on-CoW. Which is a bigger issue now on R4.0, where even AppVMs'
private volumes are CoW. (And turning off the lower, filesystem-level
CoW for 'file'-on-btrfs images would turn off data checksums too, i.e.
protection against bit rot.)

Also in contrast to 'file', all storage features are supported,
including

    - any number of revisions_to_keep
    - volume.revert()
    - volume.is_outdated
    - online fstrim/discard

Example tree of a file-reflink pool - *-dirty.img are connected to Xen:

    - /var/lib/testpool/appvms/foo/volatile-dirty.img
    - /var/lib/testpool/appvms/foo/root-dirty.img
    - /var/lib/testpool/appvms/foo/root.img
    - /var/lib/testpool/appvms/foo/private-dirty.img
    - /var/lib/testpool/appvms/foo/private.img
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T03:04:05Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T04:05:06Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T05:06:07Z
    - /var/lib/testpool/appvms/bar/...
    - /var/lib/testpool/appvms/...
    - /var/lib/testpool/template-vms/fedora-26/...
    - /var/lib/testpool/template-vms/...

It looks similar to a 'file' pool tree, and in fact file-reflink is
drop-in compatible:

    $ qvm-shutdown --all --wait
    $ systemctl stop qubesd
    $ sed 's/ driver="file"/ driver="file-reflink"/g' -i.bak /var/lib/qubes/qubes.xml
    $ systemctl start qubesd
    $ sudo rm -f /path/to/pool/*/*/*-cow.img*

If the user tries to create a fresh file-reflink pool on a filesystem
that doesn't support reflinks, qvm-pool will abort and mention the
'setup_check=no' option. Which can be passed to force a fallback on
regular sparse copies, with of course lots of time/space overhead. The
same fallback code is also used when initially cloning a VM from a
foreign pool, or from another file-reflink pool on a different
mountpoint.

'journalctl -fu qubesd' will show all file-reflink copy/rename/remove
operations on VM creation/startup/shutdown/etc.
2018-02-12 21:20:05 +00:00
Rusty Bird
7a75e1090d
Make AppVM/DispVM root volume rw to avoid CoW-on-CoW 2018-02-12 21:20:04 +00:00
Marek Marczykowski-Górecki
340b8dbfe2
tests: add a test for removing expired firewall rules
QubesOS/qubes-issues#1173
2018-02-07 02:48:12 +01:00
Marek Marczykowski-Górecki
5e89b23288
firewall: use asyncio's call_later instead of systemd to reload rules
When some expiring rules are present, it is necessary to reload firewall
when those rules expire. Previously systemd timer was used to trigger
this action, but since we have own daemon now, it isn't necessary
anymore - use this daemon for that.
Additionally automatically removing expired rules was completely broken
in R4.0.

Fixes QubesOS/qubes-issues#1173
2018-02-07 02:48:11 +01:00
Marek Marczykowski-Górecki
e6bb2828f1
storage/lvm: add more details to refused volume shrink message
Include old and new size.
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
4e314c9792
tests: storage/lvm resize
QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
7903dc53f9
storage/lvm: fix resizing not persistent volumes
Even when volume is not persistent (like TemplateBasedVM:root), it
should be resizeable. Just the new size, similarly to the volume
content, will be reverted after qube shutdown.

Additionally, when VM is running, volume resize should affect _only_ its
temporary snapshot. This way resize can be properly reverted together
with actual volume changes (which include resize2fs call).

Fixes QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Wojtek Porczyk
8083d6dece qubes/tests: fix QubesTestCase.qrexec_policy
The context manager left empty file.

fixes QubesOS/qubes-issues#3535
2018-02-04 14:03:12 +01:00
Yassine Ilmi
65d5c41fdb
Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90 2018-02-01 01:03:05 +00:00
Yassine Ilmi
a0d45aac9c
replaced underscore by dash and update test accordingly 2018-02-01 00:50:42 +00:00
Yassine Ilmi
0eb978c59b
Updated assert content for test_620_qdb_standalone in TC_90_QubesVM 2018-02-01 00:33:12 +00:00
Yassine Ilmi
1c3b412ef8
Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372 2018-02-01 00:12:51 +00:00
Marek Marczykowski-Górecki
9b5256f002
version 4.0.21 2018-01-30 00:18:20 +01:00
Marek Marczykowski-Górecki
a6a7efc9a7
vm/mix/net: fix handling network detach/attach on VM startup
- catch both QubesException and libvirtError - do not kill starting VM
just because an error while connecting _other_ VMs to it
- try to detach network first (and do not abort on error) - if
libvirt/libxl will manage to cleanup stale interface this way, the
attach operation below may succeed.

Fixes QubesOS/qubes-issues#3163
2018-01-29 23:06:21 +01:00
Marek Marczykowski-Górecki
86026e364f
Fix starting PCI-having HVMs on early system boot and later
1. Make sure VMs are started after dom0 actual memory usage is reported
to qmemman, otherwise dom0 will hold 4GB, even if just a little over 1GB
is needed at that time.

2. Request only vm.memory MB from qmemman, instead of vm.maxmem. While
HVM with PCI devices indeed do not support populate-on-demand, this is
already handled in libvirt XML.

The later may often cause VM startup fail on systems with 8GB of memory,
because maxmem is 4GB there and with dom0 keeping the other 4GB (see
point 1) there is not enough memory to start any sych VM.

Fixes QubesOS/qubes-issues#3462
2018-01-29 22:57:32 +01:00
Marek Marczykowski-Górecki
2a8fd9399e
version 4.0.20 2018-01-27 01:47:54 +01:00
Marek Marczykowski-Górecki
2c2b89e2e0
tests: adjust for type=pvh change 2018-01-27 01:34:41 +01:00
Marek Marczykowski-Górecki
76836d4ff6
rpm: require libvirt new enough for type-pvh 2018-01-27 01:21:05 +01:00
Marek Marczykowski-Górecki
99134efc3c
tests: set virt_mode='hvm' for PCI tests 2018-01-26 23:49:11 +01:00
Simon Gaiser
c3d287a33c Switch to new PVH config variant 2018-01-24 02:48:00 +01:00
Marek Marczykowski-Górecki
30ba4b8e3b
version 4.0.19 2018-01-23 14:14:09 +01:00
Marek Marczykowski-Górecki
eb846f6647
Merge remote-tracking branch 'qubesos/pr/187'
* qubesos/pr/187:
  Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists
  Make 'qvm-volume revert' really use the latest revision
  Fix wrong mocks of Volume.revisions
2018-01-22 15:39:13 +01:00
Marek Marczykowski-Górecki
e577de7d49
Merge remote-tracking branch 'qubesos/pr/186'
* qubesos/pr/186:
  tests: check if udev do not touch VM's volumes
2018-01-22 15:33:50 +01:00
Marek Marczykowski-Górecki
74eb3f3208
Merge remote-tracking branch 'qubesos/pr/185'
* qubesos/pr/185:
  vm: remove doc for non-existing event `monitor-layout-change`
  vm: include tag/feature name in event name
  events: add support for wildcard event handlers
2018-01-22 15:32:57 +01:00
Marek Marczykowski-Górecki
2041e6f01c
rpm: require new enough libvirt for pci permissive setting 2018-01-22 15:31:12 +01:00
Marek Marczykowski-Górecki
9bd75d7808
Merge remote-tracking branch 'qubesos/pr/184'
* qubesos/pr/184:
  pci: Add permissive options

Fixes QubesOS/qubes-issues#3476
2018-01-22 15:30:26 +01:00
Rusty Bird
4ae854fdaf
Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists 2018-01-21 22:28:47 +00:00
Rusty Bird
fe77b0ec85
Make 'qvm-volume revert' really use the latest revision
admin.vm.volume.ListSnapshots returned volume revisions in undefined
order, but 'qvm-volume revert' assumes the list to be in chronological
order. Make that assumption true.
2018-01-20 23:20:23 +00:00
Rusty Bird
bf1f1ac5ff
Fix wrong mocks of Volume.revisions
It's a dict, not a list.
2018-01-20 23:20:22 +00:00
Marek Marczykowski-Górecki
ab15d43622
tests: check if udev do not touch VM's volumes
Based on shell version by @rustybird
2018-01-19 18:05:50 +01:00
Marek Marczykowski-Górecki
91639d7933
app: define values for default_dispvm=None, updatevm=None
If those VMs are not set, default to None, instead of throwing
AttributeError.

Fixes QubesOS/qubes-issues#3475
2018-01-19 04:50:04 +01:00
Marek Marczykowski-Górecki
206d3f136d
version 4.0.18 2018-01-19 02:45:34 +01:00
Marek Marczykowski-Górecki
2691efda8e
Merge branch 'devel-20180115'
* devel-20180115:
  tests: update PCI devices tests for core3 API
  tests: make tests.extra.VMWrapper hashable
  qmemman: do not close stdout/stderr in daemon mode
  qmemman: fix early crash
  qubesvm: do not try to define libvirt object in offline mode
  tests: one more missing virt_mode=hvm
  qubesvm: PVH minor improvements
  tests: minor fixes
  tests: add run_service and qrexec_policy wrappers to ExtraTestCase
  tests: clear PCIDevice cache after each test
2018-01-19 02:45:11 +01:00
Simon Gaiser
132f0608aa pci: Add permissive options 2018-01-19 02:03:45 +01:00
Marek Marczykowski-Górecki
faaebcc268
version 4.0.17 2018-01-18 22:51:56 +01:00
Marek Marczykowski-Górecki
5ea8eda3ea
replace console entry points with just importing the module
importing pkg_resources and looking up entry points wastes 100ms+
of time, which is totally unnecessary

This is based on
QubesOS/qubes-core-admin-client@b731ef3885
by @qubesuser
2018-01-18 22:13:37 +01:00
Marek Marczykowski-Górecki
e9cc6ee3db
version 4.0.16 2018-01-18 21:07:34 +01:00
Marek Marczykowski-Górecki
b245bbca6f
tests: update PCI devices tests for core3 API
- use asyncio where needed
- attach now takes DeviceAssignment, not DeviceInfo
- PCI ident have ':' replaced with '_'
2018-01-18 17:36:38 +01:00
Marek Marczykowski-Górecki
edbfd3843e
tests: make tests.extra.VMWrapper hashable
Allow using VMWrapper as dict key, same as QubesVM.
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
46177c7c9f
qmemman: do not close stdout/stderr in daemon mode
Allow exceptions to be logged to syslog/journald
2018-01-18 17:36:37 +01:00
Marek Marczykowski-Górecki
ca41ca66cd
qmemman: fix early crash
clear_outdated_error_markers crashes if memory stats are not retrieved
yet. In practice it crashes at the very first call during daemon
startup, making the whole qmemman unusable.

This fixes bf4306b815
    qmemman: clear "not responding" flags when VM require more memory

QubesOS/qubes-issues#3265
2018-01-18 17:36:37 +01:00