Add support for importing volume data with qvm-volume tool.
This could be also used to clear volume by issuing:
qvm-volume import --no-resize some-vm:private /dev/null
QubesOS/qubes-issues#5192
Reinstalling template is a recommended way to get it back to a clean
state after potential compromise. In that case it is essential to
discard any persistent storage of old template, as it could be used by
the attacker to re-compromise it after reinstall.
Do this similar as root volume is overridden - via volume import
function.
FixesQubesOS/qubes-issues#5192
Fix syntax workaround for python3.7 + python2.7. This code can't use
'yield from' to be still importable on python2.7, but asyncio.sleep() is
no longer iterable on python3.7. Workaround it by manually calling
coroutine.send(None) in a loop - ugly as hell, but works. I can't wait
until we could drop python2 support...
Addresses review comments, specifically:
* updates qvm-clone tests to handle ignore_errors option
* adds manpage reference to new --ignore-errors option
* trims line length in qvm-clone changes
All tests should now be passing, including the new qvm-clone
functionality.
The method will ensure the parameters are properly handled as actual
separate arguments, regardless of shell special characters in them.
For now implement this with shlex.quote, later dedicated qrexec service
could be used.
QubesOS/qubes-issues#4850
Retry connection on any OSError, instead of only few selected error
types. For example PermissionError may happen too (when socket is
created, but before it gets appropriate mode), which wasn't listed.
Specify compression explicitly in the backup profile, not only when
requested with --compress or --compress-filter.
This will allow to disable compression with --no-compress option, as the
default if no compression is specified in the profile is to use gzip.
FixesQubesOS/qubes-issues#4803
Since no output from VM is passed (and even if it would, it's redirected
to /dev/null), there is no need to switch output color.
This fixes the case when qvm-run is started in background - the color
change would affect further shell output.
FixesQubesOS/qubes-issues#4808
For various reasons, decryption may be slower than initial encryption
(different hardware, different system load etc). Do not fail the restore
operation if scrypt anticipate it will take considerably more time or
memory, than while creating the backup (where the scrypt parameters were
originally set).
Thanks to @jharveyb for the report and suggested solution.
FixesQubesOS/qubes-issues#4683
* origin/pr/84:
Fixed typo in documentation
Documentation is important! Added shortcuts to manpages
added tests for shortcuts
diff doesn't set template
added --standalone and --disp shortcuts
* origin/pr/85:
isinstance instead of type
reverted to comparing .index
forgot braces
forgot braces
get_label now accepts integer and has correct return-type
Pylint complains that main() is too long and that's indeed true. Factor
out single process call (together with all the prepartion) into separate
function and keep only common code in main().
When cloning VM, create it in the same pool as the source one.
Previously it always used default pool, which means for example renaming
a VM in non-default pool moved it back to the default one.
FixesQubesOS/qubes-issues#4145FixesQubesOS/qubes-issues#4523
Print some error even without --pass-io, otherwise the only way to learn
the failure is checking $?, as no other visual sign is there.
FixesQubesOS/qubes-issues#4533
qubes.VMShell service, used by qvm-run, expects the command on the first
input line. Previously, when --localcmd was used, the command wasn't
written anywhere and the local command was connected directly to
qubes.VMShell service. And the first line of its output was interpreted
as a command.
Fix this by starting the local command separately, after sending the
command to qubes.VMShell service.
While at it, unify handling shell command and service calls in the process.
vm.run_service(..., localcmd= ) isn't that useful in general case,
because for qubes.VMShell the caller first need to send the command
before starting local process. Since the qvm-run tool needs to implement
manual starting localcmd anyway, don't use localcmd= run_service's
argument at all to unify calling methods.
There is slight behavior change: previously localcmd was started only
after establishing service connection (for example only if qrexec policy
allows), now it is started in all the cases.
FixesQubesOS/qubes-issues#4040
Add note in QubesBase docstring it shouldn't be used directly.
Additionally add base qubesd_call and run_service methods raising
NotImplementedError with helpful message. Lack of qubesd_call in
QubesBase leads to infinite recursion, because one in PropertyHolder
calls itself then.
FixesQubesOS/qubes-issues#4568
The qubesd daemon have no information about clone source - from that
side it looks like a new VM. This means application menu is created as
for a new VM.
To fix this re-initialize menu with --source option as part of the clone
operation. It will copy both list of available applications (if
applicable) and selected applications.
This fixes both qvm-clone case and rename.
FixesQubesOS/qubes-issues#3902FixesQubesOS/qubes-issues#4124
By definition StandaloneVM is not linked to the template. Creating one
from a template is a clone operation. It's already possible using
qvm-clone tool, but it's logical to do that using qvm-create tool too.
This was the case in R3.2 too.
While adding this special case, skip cloning private volume, to preserve
behaviour of TemplateBaseVMs which do not inherit private volume either.
FixesQubesOS/qubes-issues#3793
* devices-api:
devices: include devclass when comparing devices
events: deserialize DeviceInfo class in device-* events
devices: drop DeviceInfo.options
Since now event listener reports proper QubesDaemonCommunicationError
exception instead of some form of IOError. Include it for automatic
reconnect logic.
Fixes a481490 "app: fix error reporting when connection to qubesd fails"
Port 5a39e777089d8bde6d0a620830a898c1cf3dd924 ("events: add support for
wildcard event handlers") from qubes-core-admin:
Support registering handlers for more flexible wildcard events: not only
'*', but also 'something*'. This allows to register handlers for
'property-set:*' and such.
If file to be imported is larger than the default root volume, resize
the volume first. It might be also a good idea to shrink it when needed,
but currently the backend refuse it.
FixesQubesOS/qubes-issues#3422
When both threads and processes are used, logging module can deadlock.
Workaround this by re-initializating locks in logging module.
See https://bugs.python.org/issue6721 for details.
Revert this commit when the python issue is fixed (in the python version
used for backup restore).
* windows-tools:
doc: add info what properties are inherited from template
Add 'gui-emulated' feature
qvm-start-gui: fix handlign rpc-clipboard feature
Add an explicit method for forcing emulated VGA output. Previously it
was possible only by removing `gui` feature (setting it to false had a
different effect), or enabling debug mode.
Using lack of a feature as a third state was a bad idea.
QubesOS/qubes-issues#3585
Do not crash if qubesd restart exactly the moment event handler was
called.
Event listener properly handle qubesd restarts (reconnects), but exists
if any handler raise an exception. Avoid this by logging such exceptions
but not propagating them.
This is especially the case for domain-shutdown event for DispVM - when
handled, DispVM can be already removed. Give the handled VM object even
if actual VM is already removed. For this, avoid VM existence check by
using domains.get_blind() method.
And actually implement domains.get_blind() method - it was present only
in generic collection, but not VMCollection.
This changes to accept the string none as the value None,
as if an empty string was entered. This allows setting the
netvm to "None" as described in QubesOS/qubes-issues#3942
There was such option on Qubes 3.2, so add it here too. This is
especially useful for kernel package - preun script use it to verify if
given kernel isn't needed anymore.
* improved-tools-messages:
tools: suppress full traceback in console tools
tools: add SubParsersHelpAction, which include subcommands details in --help
Right now Admin API backend will refuse to shrink volume anyway, but
we're planning to relax this restriction. Make sure the client side
(qvm-volume tool here, GUI VM settings already have this in place) will
employ appropriate safety check.
QubesOS/qubes-issues#3725
If needs to be extended - do it before import. If needs to be reduced -
after. This way, if data import fails for any reason, previous data
won't be destroyed (truncated).
Also, convert error on shrinking volume to a warning, as it doesn't break
the template (just leave it with bigger disk than needed). Currently all
storage pool implementations refuse to shrink a volume (but it may
change in the future).
QubesOS/qubes-issues#3169
QubesException class is used with meaningful messages and should be ok
to use it directly as error message. For other exceptions, still use
full traceback (most likely a bug somewhere, not user error).
FixesQubesOS/qubes-issues#3610
Don't print None value as 'None' string, but as empty one (same as at
API level). Otherwise it is indistinguishable from VM named 'None', or
same string property value.
This is especially important on LVM thin pool, where space after
removing the file needs to be given back to the pool, to be reused for
other volumes (for example this template).
qvm-start-gui lifecycle should be bound to X server lifecycle. It should
be restarted when user logoff and login again, at least to start
gui-daemons again.
Do that by opening a connection to X server and reacting to breaking
that socket.
FixesQubesOS/qubes-issues#3147
* qubesos/pr/52:
backup: don't crash when no 'qubes' group is present
tests: dom0 backup restore, both v3 and v4
backup: add support for openssl 1.1.0 options
backup: skip dom0's properties while restoring core2 backup
unused variable
style issues
Fix dom0 restore
Fix dom0 handling
Fix AdminVm class name
This is specifically the case on Travis-CI. But since dom0 backup is
restored into separate directory now, instead of directly overriding
user home, this check is much less relevant now.
I don't know if any template currently hits this code path, even the
fedora-26-minimal root.img is large enough to be split into multiple
parts. Maybe Arch Linux?
Related to https://github.com/QubesOS/qubes-core-admin/pull/188
It's already available in config dict, but lets provide uniform API. And
also it's a bit weird to look for usage data in configuration...
QubesOS/qubes-issues#3240
'qvm-run --dispvm' cannot easily make a separate qubes.WaitForSession
call. Instead, if --gui is active, pass the new WaitForSession argument
to qubes.VMShell, which will do the equivalent.
The unit tests have been copied (in slightly adapted form) from commit
a620f02e2aFixesQubesOS/qubes-issues#3012ClosesQubesOS/qubes-core-admin-client#49
When calling a service in DispVM, the connection is established only
after session is ready (if required for given service). qrexec-client by
default use 5s here, which is too low depending on hardware. Use
'qrexec_timeout' property here for DispVM case.
FixesQubesOS/qubes-issues#3012
In core-admin matching collections are real dicts, so clone this API
behaviour here too. Specific changes:
- iteration yields keys, not values
- implement values and items methods
Additionally fix keys method, it was broken on python2 (list have no
copy method).
It wasn't possible to use QubesArgumentParser(vmname_nargs=...) for
optional domain list - the option forced usage of either --all or
explicit domain list.
When starting a VM with --cdrom=some-vm:/some/path/to.iso, it can be
started only when loop device matching the path is available. For now,
add naive waiting (while ... sleep(1)) for it. Later it might worth
converting it to events handling.
This is to match core-admin API. Logically it could be a property not a
method (as other things like usage or size), but it is already defined
(and used in various places) as a method in core-admin, so lets don't
change the API right now.
* qubesos/pr/44:
Fix style else-return
tests: update qvm-template-process and qvm-remove tests
Add --force to manpage.
Avoid cloning installed_by_rpm
Print vm list before prompt
Use --force instead of --yes
Toggle installed_by_rpm in template tool
Fix error message grammar
Add --yes option and confirm prompt.
Make use of better security of Qubes 4.x by using HVM by default. If
some VMs are incompatible with it (like MirageOS based), user can always
switch it to PV manually later.
This restores Qubes R3.2 behavior
Before this patch, the following:
qvm-run -p sys-firewall 'echo -e "\e[0;46mcyan!" >&2' | wc -l
leaks the escape sequences through to the dom0 terminal via stderr,
in this case demonstrated by the ability to change the text color while
it should be fixed to red.
This can also be abused with xterm reporting sequences to cause input
to be sent to the dom0 terminal. This is potentially a security issue.
The main process sometimes sets fd 1 to O_NONBLOCK, and since in the
terminal case fd 0 and 1 are the same fd, this also results in fd 0
being non-blocking, causing qvm-run to crash with EAGAIN.
So just make the code work for both blocking and non-blocking stdin.
When some VM timeout on shutdown, the tool will try to kill all of them,
but at this point some of them may be already powered off (not all
hanged during shutdown, but only some). Handle this
situation instead of crashing. And add appropriate test.
Do not crash if the other object is completely different type. Return
False ("unequal") instead.
This crashed preparing list of devices in qubes-vm-boot-from-device.
FixesQubesOS/qubes-issues#3182
If EOF is reached on tar's stderr, stop reading it, even if didn't found
expected data. Log this event.
This may happen when tar output some fatal error, instead of filelist.
os.path.splitext fails on path without proper file base name, like
'/something/..000'. Use plain string methods (rsplit).
FixesQubesOS/qubes-issues#3167
Packages had missing dependency on python-dbus. Since DBusHandler isn't
used anywhere, drop it, instead of introducing more dependencies.
Reported by @pietrushnic
QubesOS/qubes-issues#3179
vm.get_power_state() have specifically documented 'NA' state for cases
when it's unable to get VM's power state. Use this when qrexec policy
forbid checking it.
Reported by @pietrushnic
FixesQubesOS/qubes-issues#3179
1. Output of `losetup` command contains `\n` - strip it.
2. Provide read-only option - if device info hasn't propagated to qubesd
yet, it will not be set automatically.
FixesQubesOS/qubes-issues#3146
First, TemplateVM is not used anymore (see previous commit). Second,
don't harcode on client side that "only TemplateVM can be a template for
any VM" (which actually isn't true: AppVM can be a template for DispVM).
Very few calls at client side really needs VM class name. So, even in
non-blind mode use just QubesVM class, to avoid strange cases depending
on blind mode being enabled or not. Then, have VM class name in 'klass'
property. If known at object creation time, cache it, otherwise query
qubesd at first access.
It may happen that when client handle the event, domain no longer
exists. This is for example common for DispVMs, which get removed just
after shutdown.
This will cause some events to be dropped, but one can enable blind
mode, to get them anyway (because it will not cause KeyError, even if
domain is already removed).
QubesOS/qubes-issues#3100
This allows to perform actions on objects (VM, storage etc), without
listing them. This is useful when calling VM have minimal permissions
and only selected actions are allowed.
This means that app.domains['some-name'] will not raise KeyError, even
when domain do not exists. But performing actual action (like
vm.start()) will fail in that case.
Booting a VM from cdrom require attaching the device before VM startup,
which is possible only in persistent mode. But for qvm-start --cdrom
adding a cdrom only temporarily, use new update_persistence() function
to convert the assignment to temporary one.
FixesQubesOS/qubes-issues#3055
Abort tar process after extracting requested files - do not parse the
archive until the end (possibly tens of GB later).
FixesQubesOS/qubes-issues#2986
This may be confiusing, for example one may think that
`qvm-prefs --unset vmname netvm` will make vmname network-disconnected.
This type of mistakes may have severe security consequence, so better
drop those option names.
QubesOS/qubes-issues#3002
cc @rootkovska
Add option to uniformly start new DispVM from either VM or Dom0. This
use DispVMWrapper, which translate it to either qrexec call to $dispvm,
or (in dom0) to appropriate Admin API call to create fresh DispVM
first.
This require abandoning registering --all and --exclude by
QubesArgumentParser, because we need to add --dispvm mutually exclusive
with those two. But actually handling those two options is still done by
QubesArgumentParser.
This also updates man page and tests.
FixesQubesOS/qubes-issues#2974
This is a wrapper to use `$dispvm` target of qrexec call, just like any
other service call in qubesadmin module - using vm.run_service().
When running in dom0, qrexec-client-vm is not available, so DispVM needs
to be created "manually", using appropriate Admin API call
(admin.vm.CreateDisposable).
QubesOS/qubes-issues#2974
When data block is smaller than 4096 (and no EOF is reached), python's
io.read() will call read(2) again to get more data. This may deadlock if
the other end of connection will write anything only after receiveing
data (which is the case for qubes.Filecopy).
Disable this buffering by using syscall wrappers directly. To not affect
performance that much, increase buffer size to 64k.
FixesQubesOS/qubes-issues#2948
* devel-6:
qvm-ls: fix total VM size reporting
doc: update manpage of qvm-service
tools: qvm-service tool
tests: too much copy&paste
features: serialize True as '1'
tools/qvm-start-gui: add --force-stubdomain options
tools/qvm-shutdown: fix help message
tools/qvm-shutdown: drop --force option, it isn't supported anymore
New qvm-backup tool can either use pre-existing backup profile
(--profile), or - when running in dom0 - can create new one based on
used options (--save-profile).
This commit add a tool itself, update its man page, and add tests for
it.
FixesQubesOS/qubes-issues#2931
Do not check for qubesd socket (at module import time), because if not
running at this precise time, it will lead to wrong choice. And a weird
error message in consequence (looking for qrexec-client-vm in dom0).
FixesQubesOS/qubes-issues#2917
* devel-2-qvm-run-1:
Make pylint happy
tools/qvm-run: fix handling EOF
tests: mark qvm-run tests with "expected failure"
tools/qvm-run: fix handling copying stdin to the process
since qvm-run use multiprocessing.Process now, stdin sent to it is
processed in separate process and doesn't come back to
TestApp.actual_calls (self.app). Annotate tests for now, to be fixed
later.
The most important part is fixing resize handling - call size_func
before data_func, but after tar gets initial data (and output file
size).
But other than that, it makes the process a little faster.
QubesOS/qubes-issues#1214
Opt for a simple one-liner error messages, instead of meaningless stack
trace (it's most of the time about qubesd responding with error, so the
stack trace of actual problem is elsewhere).
This will be less realistic (private.img of 2MB?!), but makes tests much
quicker. And since tar is used to make files sparse, we don't really
test multi-part archives anyway.
Based on "backup compatibility" tests, which manually assemble the
backup. This is because we don't have access to actual backup creation
code here.
QubesOS/qubes-issues#1214
Launch stdin copy loop in a separate process (multiprocessing.Process)
and terminate it when target process is terminated.
Another idea here was threads, but there is no API to kill a thread
waiting on read().
VMs can have runtime dependencies - for example it isn't possible to
shutdown netvm used by some other running VM(s). Since client-side tools
may not have full knowledge about rules enforcing those dependencies
(for example may not have access to 'netvm' property), implement
best-effort approach:
1. Try to shutdown all requested VMs
2. For those where shutdown request succeed, wait for actual shutdown
3. For others - go back to step 1
And loop until all VMs are shutdown, or all shutdown requests fails.
Allow programming different responses for the same request when called
multiple times. This is useful for example for shutdown tests - first
domain is running, but after issuing shutdown request is is not.
Do not close event loop in utility function - handle it only in main().
For this reason, change appropriate functions to coroutines.
FixesQubesOS/qubes-issues#2865
Don't fail the whole process when "just" appmenus import fails.
But if data import fails, remove the VM
Also update for vm.run_service_for_stdio raising CalledProcessError.
Don't start GUI daemon for given VM twice when qvm-start-gui was started
during VM startup (while waiting for qrexec startup). This is especially
common while running tests.
Report failed qubes.SetMonitorLayout as warning (instead of unhandled
exception).
Clear VM cache on qubesd reconnect.
Fix logging.
qubesd may be restarted during different stages of connection - either
while attempting to connect, or while already listening on events.
Adjust exception list accordingly.
This is especially important for qvm-start-gui - otherwise it crashes on
qubesd restart.
This way we don't need separate admin.vm.Clone call, which is tricky to
handler properly with policy.
A VM may not have access to all the properties and other metadata, so
add ignore_errors argument, for best-effort approach (copy what is
possible). In any case, failure of cloning VM data fails the whole
operation.
When operation fails, VM is removed.
While at it, allow to specify alternative VM class - this allows
morphing one VM into another (for example AppVM -> StandaloneVM).
Adjust qvm-clone tool and tests accordingly.
QubesOS/qubes-issues#2622
Enable 'qrexec' VM feature to wait for qrexec initialization - it is
required to call qubes.PostInstall service. If VM start fails, assume
there is no qrexec and drop that feature.
Since Admin API, qvm-ls takes a long time to complete. Therefore,
Corporate Headquarters commanded that a Enterprise Spinner is to be
implemented and mandated it's use unto us.
We take amusement from its endless gyrations.
Empty parameter value is encoded as b'parameter\0\0', so we can't simply
read the data until b'\0\0', because it isn't necessary event end.
Instead, read event parts separately, according to specification.
