Commit Graph

186 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
c781a522d8 backups: move backup code to separate file
Also some major cleanups: Reduce some more code duplication
(verify_hmac, simplify backup_restore_prepare). Rename
backup_dir/backup_tmpdir variables to better match its purpose. Rename
backup_do_copy back to backup_do.  Require QubesVm object (instead of VM
name) as appvm param.
2013-11-25 05:41:13 +01:00
Marek Marczykowski-Górecki
657beaf655 backups: move extracted dom0 home from /var/tmp instead of copy 2013-11-25 01:11:29 +01:00
Marek Marczykowski-Górecki
e31c3ae8e7 backup: reduce volume size to 100M and limit queue length
This way backup process won't need more than 1GB for temporary files and
also will give more precise progress information. For now it looks like
the slowest element is qrexec, so without such limit, all the data would
be prepared (basically making second copy of it in dom0) while only
first few files would be transfered to the VM.
Also backup progress is calculated based on preparation thread, so when
it finishes there is some other time needed to flush all the data to the
VM. Limiting this amount makes progress somehow more accurate (but still
off by 1GB...).
2013-11-25 00:55:59 +01:00
Marek Marczykowski-Górecki
10a01010bb backups: fix handling multi-volume archive during restore
We can't wait for tar next volume prompt using stderr.readline(),
because tar don't output EOL marker after this prompt. The other way
would be switching file descriptor to non-blocking mode and using lower
level os.read(), but this looks like more error-prone way (races...).
So change idea of handling such archives: after switching to next
archive volume, simply send '\n' to tar (which will receive when
needed). When getting "*.000" file, assume that previous archive was
over and wait for previous tar process. Then start the new one.

Also don't give explicit tape length, only turn multi-volume mode on. So
will correctly handle all multi-volume archives, regardless of its size.
2013-11-25 00:48:54 +01:00
Marek Marczykowski-Górecki
7229b78bbf backups: minor reduce code duplication 2013-11-25 00:48:00 +01:00
Marek Marczykowski-Górecki
bc59d7e054 backups: include file path in internal archive, implement dom0 home restore
This is mostly revert of "3d1b40f backups: keep file without path in
inner tar archive" in terms of archive format, but the code is more
robust than old one. Especially reuse already computed dir paths. Also
restore only requested files (based on selected VMs and its qubes.xml
data). Change the restore workflow to restore files first to temporary
directory, then move to final dirs. This approach:
 - will be compatible with hashed vm name in the archive path
 - is required to handle dom0 home backup (directory outside of
   /var/lib/qubes)
 - it should be also more defensive - make any changes in /var/lib/qubes
 only after successful extraction of files and creating Qubes*Vm object

Second change in this commit is implement of dom0 home backup/restore.
As qubes.xml now contains data about dom0, we have information whether
it is included in the backup (before getting actual files).
2013-11-25 00:36:40 +01:00
Marek Marczykowski-Górecki
dc6fd3c8f3 core: store dom0 info in qubes.xml
At least to have there info about its backup.
2013-11-24 23:50:39 +01:00
Marek Marczykowski-Górecki
a64f7c12ad backups: desperate try to improve readability
Especially kill long lines.
2013-11-24 23:49:53 +01:00
Marek Marczykowski-Górecki
c306b9c00a backups: increase readability of long function calls 2013-11-24 23:49:53 +01:00
Marek Marczykowski-Górecki
5477aea877 backups: increase buffer size for better performance
After this change the bottleneck is qrexec throughput.
2013-11-24 23:49:53 +01:00
Marek Marczykowski-Górecki
c64b6c04ce backups: make all the debug easy to disable 2013-11-24 23:49:39 +01:00
Marek Marczykowski-Górecki
005db6a5ab backups: fix race condition in "tape" change event during backup
Ensure that outer tar/encryptor gets all the data *and EOF* before
signalling inner tar to continue. Previously it could happen that inner
tar begins to write next data chunk, while qvm-backup still holds
previous data chunk open.
2013-11-24 03:28:21 +01:00
Marek Marczykowski-Górecki
3d1b40f25c backups: keep file without path in inner tar archive
It is senseless to have full file path in multiple locations:
 - external archive
 - qubes.xml
 - internal archive
Also it is more logical to have only "private.img" file in archive
placed in "appvms/untrusted/private.img.000". Although this is rather
cosmetic change for VMs data, it is required to backup arbitrary
directory, like dom0 user home.

Also use os.path.* instead of manual string operations (split,
partition). It is more foolproof.
2013-11-24 03:23:27 +01:00
Marek Marczykowski-Górecki
61b3a81e82 backup: remove unused argument from backup_prepare 2013-11-24 03:17:15 +01:00
Marek Marczykowski-Górecki
e7701d9c5d backup: check for disk space if target is local directory 2013-11-24 03:15:44 +01:00
Olivier MEDOC
51f119326b backup: improved error handling during restore 2013-11-23 02:44:05 +01:00
Olivier MEDOC
e875ae9d06 backups: use tar2qfile filtering to enable partial backup restore 2013-11-23 02:44:05 +01:00
Marek Marczykowski-Górecki
2005207462 Template support for HVM (#719)
Any HVM (which isn't already template-based) can be a template for
another HVM. For now do not allow simultaneous run of template and its
VM (this assumption simplify the implementation, as no root-cow.img is
needed).
2013-11-19 18:42:59 +01:00
Marek Marczykowski-Górecki
dfa9e7c0df Add device-mapper devices to name->(major,minor) mapping
qvm-block will still not list device-mapper devices, but it would be
much easier to modify it.
2013-11-16 01:29:50 +01:00
Marek Marczykowski-Górecki
3666d6ced9 backup: wait for process termination in restore header phase
One more race condition, which could cause qvm-backup-restore hang.
2013-11-09 18:56:08 +01:00
Marek Marczykowski-Górecki
319158d5b1 backup: restore: Ignore qubes.xml in the second restore pass
Already processed in backup prepare phase). This is only because
qfile-dom0-unpacker doesn't support selective unpack (like tar do).
This should be extended to skip also VMs not selected for restore.
2013-11-09 18:54:52 +01:00
Marek Marczykowski-Górecki
1880f61c2d backup: restore: process files until EOF received
Not only until unpacker process is running. This is another race
condition, which would cause some data left in the pipe buffer not
processed.
2013-11-09 18:53:57 +01:00
Marek Marczykowski-Górecki
3c993c619c backup: restore: handle VM data with qfile format instead of simple tar
This was already partially implemented, but only for backup header
(qubes.xml).
Fix handling of vmproc object (available only when backup in another
VM).
Also fix some race conditions - wait for process termination, not only
check its exit code (which would be None if process still running).
2013-11-09 17:20:18 +01:00
Marek Marczykowski-Górecki
c61a4570e8 backup: use 'dom0' as source domain for RPC calls 2013-11-09 17:12:41 +01:00
Marek Marczykowski-Górecki
661a1ba4af backup: comment update 2013-11-09 17:12:14 +01:00
Marek Marczykowski-Górecki
54f08e00b9 backup: fix race condition with inner tar process
Do not assume that tar will finish quickly - explicitly wait for either
process termination or request for the next archive part.
2013-11-09 17:09:44 +01:00
Marek Marczykowski-Górecki
89b6069bda backup: whitespace fixes 2013-11-09 17:06:29 +01:00
Marek Marczykowski-Górecki
b84ba998a3 backup: fix misused variables - most likely copy&paste error 2013-11-09 16:56:59 +01:00
Marek Marczykowski-Górecki
5cebff34bd backup: include qubes.xml
It was commented out by mistake.
2013-11-09 16:51:25 +01:00
Marek Marczykowski-Górecki
34b03fe2b3 backup: fix setting backup qubes.xml attributes
Cannot compare QubesVm objects, because we have different instances of
QubesVmCollection. So compare QID instead.
2013-11-09 16:46:06 +01:00
Marek Marczykowski-Górecki
a56ceb92c5 backup: update for new API - use system_path hash
Instead of a bunch of global variables
2013-11-09 15:55:17 +01:00
Marek Marczykowski-Górecki
a9a8335403 Merge remote-tracking branch 'oliv/master' into new-backups
Conflicts:
	core/qubesutils.py
	dom0/qvm-core/qubes.py
2013-11-07 22:41:16 +01:00
Marek Marczykowski-Górecki
e6012a8fd2 Fix qubes.xml unlock code (#748)
Do not call explicit lockf(lOCK_UN) to not unlock the file
buffers are flushed. This is simpler than adding flush() call.
2013-10-02 04:54:10 +02:00
Marek Marczykowski-Górecki
76144df2e7 qvm-backup: respect "include in backups by default" setting (#746) 2013-10-01 03:31:12 +02:00
Marek Marczykowski
28b8eb0445 Send monitor layout at VM startup.
Gui daemon isn't aware of multihead parameters, also gui protocol
doesn't support such information - currently by design it is configured
via Qubes RPC service.
At GUI startup send monitor layout to the VM.
2013-08-11 04:11:34 +02:00
Marek Marczykowski
a84886db07 Move all files one level up 2013-03-16 19:56:51 +01:00