This requires having at least 1GB free on /tmp, but it is fair
assumption - it's tmpfs in dom0 and while performing the backup most of
the VMs aren't running, so shouldn't be a problem. Anyway it is always
possible to set TMPDIR variable or pass --tmpdir cmdline option.
Using tmpfs based temporary directory should speedup the backup.
QubesOS/qubes-issues#1652
If SendWorker queue is full, check if that thread is still alive.
Otherwise it would deadlock on putting an entry to that queue.
This also requires that SendWorker must ensure that the main thread
isn't currently waiting for queue space when it fails. We can do this by
simply removing an entry from a queue - so on the next iteration
SendWorker would be already dead and main thread would notice it.
Getting an entry from queue in such (error) situation is harmless,
because other checks will notice it's an error condition.
FixesQubesOS/qubes-issues#1359
For a long time Qubes backup did not include symlinked files, which
apparently is quite common practice for users with multiple disks (for
example HDD + SSD). It is covered in documentation
(https://www.qubes-os.org/doc/secondary-storage/), but better solution
would be to simply include symlinked files.
Restore of such files would (of course) not preserve the symlinks -
normal files will be restored instead. But that's fine. If the user want
to move the data to another location, he/she can do that and restore the
symlink.
The only possible breakage from this change is having a copy (instead of
symlink) to a VM icon. But storing that symlink in a backup was broken
for some time (because of --xform usage) and it is handled during
restore, so not a real problem.
This doesn't cover all the problems with symlinked VM images - the other
one is qvm-block behaviour, which would treat such images as non-system
disks, so easily detachable (which would break VM operation). But that's
another story.
FixesQubesOS/qubes-issues#1384
In most cases it would be some leftover after failed restore, or even
the reason why the user is restoring a VM in the first place. Move it to
nearby directory, but do not remove - backup tool should _never_ remove
any data.
When the pre-existing directory would not be moved, restore utility
(`shutil.move`) would place the data inside of that directory, with
additional directory level (for example `/var/lib/qubes/appvms/work/work`),
which would be wrong and would later fail on `vm.verify_files`. And more
importantly - such VM would not work.
FixesQubesOS/qubes-issues#1386
When qfile-dom0-unpacker detects an error, it sends error report to
stdout and terminate (so stdout is closed). That close should be
transferred to the VM process (as EOF on its stdin), which will signal
it to stop sending the data and handle error report.
Also qrexec-client holds the connection until both stdin and
stdout are closed.
So when that EOF is missing, tar2qfile will not detect error report and
still tries to send the data and qrexec-client will hold the
connection while receiving process is long dead.
To prevent that deadlock from happening, close FD in python code, so
qfile-dom0-unpacker will be the last owner of write end of the pipe.
When it closes its stdout, qrexec-client will receive EOF at its stdin.
Since tar multi-archive no longer used, we can simply instruct tar to
pipe output through gzip (or whatever compressor we want). Include used
compressor command in backup header.
Tar multi-volume support is broken when used with sparse files[1], so do
not use it. Instead simply cut the archive manually and concatenate at
restore time. This change require a little modification in restore
process, so make this new backup format ("3"). Also add backup format
version to the header, instead of some guessing code.
For now only cleartext and encrypted backups implemented, compression
will come as a separate commit.
Backups should be safe also for long-term, so change HMAC to SHA512,
which should be usable much longer than SHA1.
See this thread for discussion:
https://groups.google.com/d/msg/qubes-devel/5X-WjdP9VqQ/4zI8-QWd0S4J
Additionally save guessed HMAC in artificial header data (when no real
header exists).
Previously this message goes to /dev/null (unless BACKUP_DEBUG enabled),
so the user got cryptic "Restore failed" message without any clue about
the cause.
When non-english language is set, some processes can output non-ASCII
characters in error messages. Handle them nicely.
Also make error messages more consistent about "ERROR:" prefix. Do not
use this prefix in QubesException message, add it just before showing
the message to the user.
The --xform option affects all the names - including symlink targets. So
make the pattern as precise as possible to not break symlinks in dom0
home backup. Still not ideal, but at least limit damage to relative
symlinks pointing at <username> directory (which hopefully user will
not create). Previous version broke all relative symlinks...
The backup_cancel() method kills processes registered by main thread and
set "running_backup_operation.canceled" to True. Then main thread get an
error because of killed processes and check if that was because of
cancel request.
Introduce BackupCanceledError, which can report temporary dir to remove.
Do wait for nest reported filename only when restoring directly from
dom0. In VM case it isn't necessary and will cause false error report
(because filename will be set to nextfile at the end of restore process,
so will be treated as spurious file without hmac).
