Commit Graph

6150 Commits

Author SHA1 Message Date
Rusty Bird
6886e70aa9
backup.py: add vmN/empty file if no other files to backup 2018-02-24 22:43:53 +00:00
Rusty Bird
59abdeb30e
Allow include=None to be passed to admin.backup.Info
This is forwarded to qubes.backup.Backup(), which uses None to choose
VMs according to their include_in_backups property.

('git show -w' recommended)
2018-02-23 21:29:15 +00:00
Rusty Bird
dbaf60ca24
Add include_in_backups property for AdminVM 2018-02-23 21:29:15 +00:00
Rusty Bird
84ca0c6df5
Use !auto_cleanup as DispVM include_in_backups default 2018-02-23 21:29:15 +00:00
Marek Marczykowski-Górecki
716114f676
Merge remote-tracking branch 'qubesos/pr/197'
* qubesos/pr/197:
  Don't fire domain-stopped/-shutdown while VM is still Dying
2018-02-22 21:14:55 +01:00
Rusty Bird
f96fd70f76
Don't fire domain-stopped/-shutdown while VM is still Dying
Lots of code expects the VM to be Halted after receiving one of these
events, but it could also be Dying or Crashed. Get rid of the Dying case
at least, by waiting until the VM has transitioned out of it.

Fixes e.g. the following DispVM cleanup bug:

    $ qvm-create -C DispVM --prop auto_cleanup=True -l red dispvm
    $ qvm-start dispvm
    $ qvm-shutdown --wait dispvm  # this won't remove dispvm
    $ qvm-start dispvm
    $ qvm-kill dispvm  # but this will
2018-02-22 19:53:29 +00:00
Marek Marczykowski-Górecki
1562defd42
Merge remote-tracking branch 'qubesos/pr/193'
* qubesos/pr/193:
  tests: add a test for removing expired firewall rules
  firewall: use asyncio's call_later instead of systemd to reload rules
2018-02-22 19:47:37 +01:00
Marek Marczykowski-Górecki
7731a56daa
Merge remote-tracking branch 'qubesos/pr/192'
* qubesos/pr/192:
  storage/lvm: add more details to refused volume shrink message
  tests: storage/lvm resize
  storage/lvm: fix resizing not persistent volumes
2018-02-22 19:46:11 +01:00
Marek Marczykowski-Górecki
b283ab2b52
Merge remote-tracking branch 'qubesos/pr/195'
* qubesos/pr/195:
  Fix is_running non-boolean
2018-02-22 17:52:09 +01:00
Marek Marczykowski-Górecki
033100fa7e
Merge remote-tracking branch 'qubesos/pr/194'
* qubesos/pr/194:
  reflink: style fix
  storage: typo fix
  lvm_thin: _remove_revisions() on revisions_to_keep==0
  lvm_thin: don't purge one revision too few
  lvm_thin: really remove revision
  lvm_thin: fill in volume's revisions_to_keep from pool
2018-02-22 17:51:37 +01:00
Christopher Laprise
75d8c553f9
Fix is_running non-boolean 2018-02-20 22:30:47 -05:00
Marek Marczykowski-Górecki
4f52a3b62e
version 4.0.23 2018-02-20 01:18:31 +01:00
Marek Marczykowski-Górecki
522a9594ee
rpm: adjust dependencies 2018-02-20 01:18:15 +01:00
Marek Marczykowski-Górecki
68b6f1ec76
qubespolicy: use '@' instead of '$' for policy keywords
Using '$' is easy to misuse in shell scripts, shell commands etc. After
all this years, lets abandon this dangerous character and move to
something safer: '@'. The choice was made after reviewing specifications
of various shells on different operating systems and this is the
character that have no special meaning in none of them.

To preserve compatibility, automatically translate '$' to '@' when
loading policy files.
2018-02-19 03:33:40 +01:00
Marek Marczykowski-Górecki
c87fcd7e2e
qubespolicy: use separate arguments for original target type and value
Provide original target as two arguments: type, value
This will ease handling special keywords without risking hitting shell
special characters or other problems.
2018-02-19 03:32:44 +01:00
Rusty Bird
37e1aedfa3
reflink: style fix 2018-02-16 21:47:39 +00:00
Rusty Bird
c871424fb0
storage: typo fix 2018-02-16 21:47:37 +00:00
Rusty Bird
f6542effc5
lvm_thin: _remove_revisions() on revisions_to_keep==0
If revisions_to_keep is 0, it may nevertheless have been > 0 before, so
it makes sense to call _remove_revisions() and hold back none (not all)
of the revisions in this case.
2018-02-16 21:47:36 +00:00
Rusty Bird
ade49d44b7
lvm_thin: don't purge one revision too few
We just added one that must be taken into account.
2018-02-16 21:47:35 +00:00
Rusty Bird
86998b0815
lvm_thin: really remove revision 2018-02-16 21:47:33 +00:00
Rusty Bird
9f468f93f5
lvm_thin: fill in volume's revisions_to_keep from pool
Otherwise, the base class defaults revisions_to_keep to 0.
2018-02-16 21:47:32 +00:00
Marek Marczykowski-Górecki
7c50bd5104
qubespolicy: unify calling qrexec service
Rely on qrexec-client resolving QUBESRPC keyword, same as in case of VM
call. This will allow applying special treatment to such calls, like
calling qubes-rpc-multiplexer directly (avoiding shell), because we have
defined protocol what can be used here.
2018-02-16 04:30:32 +01:00
Marek Marczykowski-Górecki
b00bbb73e4
Merge remote-tracking branch 'qubesos/pr/190'
* qubesos/pr/190:
  Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90
  replaced underscore by dash and update test accordingly
  Updated assert content for test_620_qdb_standalone in TC_90_QubesVM
  Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372
2018-02-14 01:29:08 +01:00
Marek Marczykowski-Górecki
e3d3e149f0
Fix too long line 2018-02-13 11:27:59 +01:00
Marek Marczykowski-Górecki
14c6aad153
version 4.0.22 2018-02-13 05:30:11 +01:00
Marek Marczykowski-Górecki
209af07fd0
Merge remote-tracking branch 'qubesos/pr/188'
* qubesos/pr/188:
  file-reflink, a storage driver optimized for CoW filesystems
  Make AppVM/DispVM root volume rw to avoid CoW-on-CoW
2018-02-13 05:20:52 +01:00
Marek Marczykowski-Górecki
5afd5fb04f
Merge remote-tracking branch 'qubesos/pr/191'
* qubesos/pr/191:
  qubes/tests: fix QubesTestCase.qrexec_policy
2018-02-13 05:20:39 +01:00
Marek Marczykowski-Górecki
717bc4ace3
vm/appvm: forbid changing template if the are children DispVMs
Changing AppVM's template will not update root volume reference in
DispVMs based on it. For now forbid such change.

Fixes QubesOS/qubes-issues#3576
2018-02-13 04:53:39 +01:00
Rusty Bird
1695a732b8
file-reflink, a storage driver optimized for CoW filesystems
This adds the file-reflink storage driver. It is never selected
automatically for pool creation, especially not the creation of
'varlibqubes' (though it can be used if set up manually).

The code is quite small:

               reflink.py  lvm.py      file.py + block-snapshot
    sloccount  334 lines   447 (134%)  570 (171%)

Background: btrfs and XFS (but not yet ZFS) support instant copies of
individual files through the 'FICLONE' ioctl behind 'cp --reflink'.
Which file-reflink uses to snapshot VM image files without an extra
device-mapper layer. All the snapshots are essentially freestanding;
there's no functional origin vs. snapshot distinction.

In contrast to 'file'-on-btrfs, file-reflink inherently avoids
CoW-on-CoW. Which is a bigger issue now on R4.0, where even AppVMs'
private volumes are CoW. (And turning off the lower, filesystem-level
CoW for 'file'-on-btrfs images would turn off data checksums too, i.e.
protection against bit rot.)

Also in contrast to 'file', all storage features are supported,
including

    - any number of revisions_to_keep
    - volume.revert()
    - volume.is_outdated
    - online fstrim/discard

Example tree of a file-reflink pool - *-dirty.img are connected to Xen:

    - /var/lib/testpool/appvms/foo/volatile-dirty.img
    - /var/lib/testpool/appvms/foo/root-dirty.img
    - /var/lib/testpool/appvms/foo/root.img
    - /var/lib/testpool/appvms/foo/private-dirty.img
    - /var/lib/testpool/appvms/foo/private.img
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T03:04:05Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T04:05:06Z
    - /var/lib/testpool/appvms/foo/private.img@2018-01-02T05:06:07Z
    - /var/lib/testpool/appvms/bar/...
    - /var/lib/testpool/appvms/...
    - /var/lib/testpool/template-vms/fedora-26/...
    - /var/lib/testpool/template-vms/...

It looks similar to a 'file' pool tree, and in fact file-reflink is
drop-in compatible:

    $ qvm-shutdown --all --wait
    $ systemctl stop qubesd
    $ sed 's/ driver="file"/ driver="file-reflink"/g' -i.bak /var/lib/qubes/qubes.xml
    $ systemctl start qubesd
    $ sudo rm -f /path/to/pool/*/*/*-cow.img*

If the user tries to create a fresh file-reflink pool on a filesystem
that doesn't support reflinks, qvm-pool will abort and mention the
'setup_check=no' option. Which can be passed to force a fallback on
regular sparse copies, with of course lots of time/space overhead. The
same fallback code is also used when initially cloning a VM from a
foreign pool, or from another file-reflink pool on a different
mountpoint.

'journalctl -fu qubesd' will show all file-reflink copy/rename/remove
operations on VM creation/startup/shutdown/etc.
2018-02-12 21:20:05 +00:00
Rusty Bird
7a75e1090d
Make AppVM/DispVM root volume rw to avoid CoW-on-CoW 2018-02-12 21:20:04 +00:00
Marek Marczykowski-Górecki
340b8dbfe2
tests: add a test for removing expired firewall rules
QubesOS/qubes-issues#1173
2018-02-07 02:48:12 +01:00
Marek Marczykowski-Górecki
5e89b23288
firewall: use asyncio's call_later instead of systemd to reload rules
When some expiring rules are present, it is necessary to reload firewall
when those rules expire. Previously systemd timer was used to trigger
this action, but since we have own daemon now, it isn't necessary
anymore - use this daemon for that.
Additionally automatically removing expired rules was completely broken
in R4.0.

Fixes QubesOS/qubes-issues#1173
2018-02-07 02:48:11 +01:00
Marek Marczykowski-Górecki
e6bb2828f1
storage/lvm: add more details to refused volume shrink message
Include old and new size.
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
4e314c9792
tests: storage/lvm resize
QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Marek Marczykowski-Górecki
7903dc53f9
storage/lvm: fix resizing not persistent volumes
Even when volume is not persistent (like TemplateBasedVM:root), it
should be resizeable. Just the new size, similarly to the volume
content, will be reverted after qube shutdown.

Additionally, when VM is running, volume resize should affect _only_ its
temporary snapshot. This way resize can be properly reverted together
with actual volume changes (which include resize2fs call).

Fixes QubesOS/qubes-issues#3519
2018-02-07 02:07:25 +01:00
Wojtek Porczyk
8083d6dece qubes/tests: fix QubesTestCase.qrexec_policy
The context manager left empty file.

fixes QubesOS/qubes-issues#3535
2018-02-04 14:03:12 +01:00
Yassine Ilmi
65d5c41fdb
Missed one test, adding default-user in assert for test test_621_qdb_vm_with_network in TC_90 2018-02-01 01:03:05 +00:00
Yassine Ilmi
a0d45aac9c
replaced underscore by dash and update test accordingly 2018-02-01 00:50:42 +00:00
Yassine Ilmi
0eb978c59b
Updated assert content for test_620_qdb_standalone in TC_90_QubesVM 2018-02-01 00:33:12 +00:00
Yassine Ilmi
1c3b412ef8
Added the default_user property from the Qube to the qubesdb so it is available when starting X. This is the 1st part of a fix for issue https://github.com/QubesOS/qubes-issues/issues/2372 2018-02-01 00:12:51 +00:00
Marek Marczykowski-Górecki
9b5256f002
version 4.0.21 2018-01-30 00:18:20 +01:00
Marek Marczykowski-Górecki
a6a7efc9a7
vm/mix/net: fix handling network detach/attach on VM startup
- catch both QubesException and libvirtError - do not kill starting VM
just because an error while connecting _other_ VMs to it
- try to detach network first (and do not abort on error) - if
libvirt/libxl will manage to cleanup stale interface this way, the
attach operation below may succeed.

Fixes QubesOS/qubes-issues#3163
2018-01-29 23:06:21 +01:00
Marek Marczykowski-Górecki
86026e364f
Fix starting PCI-having HVMs on early system boot and later
1. Make sure VMs are started after dom0 actual memory usage is reported
to qmemman, otherwise dom0 will hold 4GB, even if just a little over 1GB
is needed at that time.

2. Request only vm.memory MB from qmemman, instead of vm.maxmem. While
HVM with PCI devices indeed do not support populate-on-demand, this is
already handled in libvirt XML.

The later may often cause VM startup fail on systems with 8GB of memory,
because maxmem is 4GB there and with dom0 keeping the other 4GB (see
point 1) there is not enough memory to start any sych VM.

Fixes QubesOS/qubes-issues#3462
2018-01-29 22:57:32 +01:00
Marek Marczykowski-Górecki
2a8fd9399e
version 4.0.20 2018-01-27 01:47:54 +01:00
Marek Marczykowski-Górecki
2c2b89e2e0
tests: adjust for type=pvh change 2018-01-27 01:34:41 +01:00
Marek Marczykowski-Górecki
76836d4ff6
rpm: require libvirt new enough for type-pvh 2018-01-27 01:21:05 +01:00
Marek Marczykowski-Górecki
99134efc3c
tests: set virt_mode='hvm' for PCI tests 2018-01-26 23:49:11 +01:00
Simon Gaiser
c3d287a33c Switch to new PVH config variant 2018-01-24 02:48:00 +01:00
Marek Marczykowski-Górecki
30ba4b8e3b
version 4.0.19 2018-01-23 14:14:09 +01:00
Marek Marczykowski-Górecki
eb846f6647
Merge remote-tracking branch 'qubesos/pr/187'
* qubesos/pr/187:
  Don't fail create/clone if /var/lib/qubes/TYPE/NAME/ exists
  Make 'qvm-volume revert' really use the latest revision
  Fix wrong mocks of Volume.revisions
2018-01-22 15:39:13 +01:00