QubesVmCollection.save() overrides qubes.xml by creating new file, then
renaming it over the old one. If any process has that (old) file open
at the same time - especially while waiting on lock_db_for_writing() -
it will end up in accessing old, already unlinked file.
The exact calls would look like:
P1 P2
lock_db_for_writing
fd = open('qubes.xml')
fcntl(fd, F_SETLK, ...)
lock_db_for_writing
fd = open('qubes.xml')
fcntl(fd, F_SETLK, ...)
...
save():
open(temp-file)
write(temp-file, ...)
...
flush(temp-file)
rename(temp-file, 'qubes.xml')
close(fd) // close old file
lock_db_for_writing succeed
*** fd points at already unlinked
file
unlock_db
close(qubes.xml)
To fix that problem, added a check if (already locked) file is still the
same as qubes.xml.
Check maxmem taking into account the minimum init memory that allows
that requested maximum memory.
Explanation:
Linux kernel needs space for memory-related structures created at boot.
If init_mem is just 400MB, then max_mem can't balloon above 4.3GB (at
which poing it yields "add_memory() failed: -17" messages and apps
crash), regardless of the max_mem_size value.
Based on Marek's findings and my tests on a 16GB PC, using several
processes like:
stress -m 1 --vm-bytes 1g --vm-hang 100
result in the following points:
init_mem ==> actual max memory
400 4300
700 7554
800 8635
1024 11051
1200 12954
1300 14038
1500 14045 <== probably capped on my 16GB system
The actual ratio of max_mem_size/init_mem is surprisingly constant at
10.79
If less init memory is set than that ratio allows, then the set
maxmem is unreachable and the VM becomes unstable (app crashes)
Based on qubes-devel discussion titled "Qubes Dom0 init memory against
Xen best practices?" at:
https://groups.google.com/d/msg/qubes-devel/VRqkFj1IOtA/UgMgnwfxVSIJ
Since tar multi-archive no longer used, we can simply instruct tar to
pipe output through gzip (or whatever compressor we want). Include used
compressor command in backup header.
Tar multi-volume support is broken when used with sparse files[1], so do
not use it. Instead simply cut the archive manually and concatenate at
restore time. This change require a little modification in restore
process, so make this new backup format ("3"). Also add backup format
version to the header, instead of some guessing code.
For now only cleartext and encrypted backups implemented, compression
will come as a separate commit.
When system is going down, systemd kills all the users processes,
including 'xl' daemons waiting for domain shutdown. This results in
zombie domains not cleaned up. The proper fix would be somehow extract
those processes from user session scope (most likely by starting them as
a service).
But because it applies only to system shutdown (qvm-shutdown
call there), it is simpler to add appropriate handling code to
qvm-shutdown.
In R3 the problem will vanish, because of use libvirtd deamon, so no
user processes required to track domains state.
loop device parsing should have "dXpY_style = True" in order to
correctly parse partitions on loop devices.
Reasoning:
==========
Using losetup to create a virtual SD card disk into a loop device and
creating partitions for it results in new devices within an AppVM that
look like: /dev/loop0p1 /dev/loop0p2 and so on.
However as soon as they are created, Qubes Manager rises an exception
and becomes blocked with the following message (redacted):
"QubesException: Invalid device name: loop0p1
at line 639 of file /usr/lib64/python2.7/site-
packages/qubesmanager/main.py
Details:
line: raise QubesException....
func: block_name_to_majorminor
line no.: 181
file: ....../qubes/qubesutils.py
Backups should be safe also for long-term, so change HMAC to SHA512,
which should be usable much longer than SHA1.
See this thread for discussion:
https://groups.google.com/d/msg/qubes-devel/5X-WjdP9VqQ/4zI8-QWd0S4J
Additionally save guessed HMAC in artificial header data (when no real
header exists).
Previously this message goes to /dev/null (unless BACKUP_DEBUG enabled),
so the user got cryptic "Restore failed" message without any clue about
the cause.
When non-english language is set, some processes can output non-ASCII
characters in error messages. Handle them nicely.
Also make error messages more consistent about "ERROR:" prefix. Do not
use this prefix in QubesException message, add it just before showing
the message to the user.
