Do not abort VM restarts due to inconsistent info

Prior to this commit, there have been occasional issues with the usage
of the Qubes Manager's VM restart button where the restart procedure
is interrupted due to an exception thrown after the VM in question is
shut down. The exception has the following backtrace:

  ----
  line: assert not vm.is_running()
  func: start_vm
  line no.: 1198
  file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py
  ----
  line: self.caller.start_vm(vm)
  func: check_if_vm_has_shutdown
  line no.: 308
  file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py

Upon investigation, the root cause of the issue appears to be
inconsistent information provided by Xen regarding a recently-shut-down
VM's start-up timestamp and its state (i.e., running or shut down).

In some cases Xen would report that the VM is running whereas the
start-up timestamp would be returned as None, due to unknown reasons.
This inconsistency would then cause the code modified by this commit to
call the Qubes Manager's "start_vm" method, which would attempt to
assert that a VM is shut down, which would raise the aforementioned
exception.

This commit aims to resolve this issue by checking whether the VM has
fully shut down according to Xen and by calling "start_vm" only if the
VM has fully shut down.

This commit also slightly refactors the affected code.

Fixes: QubesOS/qubes-issues#2438

qubesmanager/main.py

@@ -273,13 +273,26 @@ class VmShutdownMonitor(QObject):
         self.shutdown_started = datetime.now()
         self.caller = caller
 
+    def restart_vm_if_needed(self):
+        if self.and_restart and self.caller:
+            self.caller.start_vm(self.vm)
+
+    def check_again_later(self):
+        # noinspection PyTypeChecker,PyCallByClass
+        QTimer.singleShot(self.check_time, self.check_if_vm_has_shutdown)
+
+    def timeout_reached(self):
+        actual = datetime.now() - self.shutdown_started
+        allowed = timedelta(milliseconds=self.shutdown_time)
+
+        return actual > allowed
+
     def check_if_vm_has_shutdown(self):
         vm = self.vm
+        vm_is_running = vm.is_running()
         vm_start_time = vm.get_start_time()
-        if vm.is_running() and vm_start_time and \
-                vm_start_time < self.shutdown_started:
-            if (datetime.now() - self.shutdown_started) > \
-                    timedelta(milliseconds=self.shutdown_time):
+        if vm_is_running and vm_start_time and vm_start_time < self.shutdown_started:
+            if self.timeout_reached():
                 reply = QMessageBox.question(
                     None, self.tr("VM Shutdown"),
                     unicode(self.tr("The VM <b>'{0}'</b> hasn't shutdown within the last "
@@ -290,22 +303,22 @@ class VmShutdownMonitor(QObject):
                         self.shutdown_time / 1000))
                 if reply == 0:
                     vm.force_shutdown()
-                    if self.and_restart:
-                        if self.caller:
-                            self.caller.start_vm(vm)
+                    self.restart_vm_if_needed()
                 else:
-                    # noinspection PyTypeChecker,PyCallByClass
                     self.shutdown_started = datetime.now()
-                    QTimer.singleShot(self.check_time,
-                        self.check_if_vm_has_shutdown)
+                    self.check_again_later()
             else:
-                QTimer.singleShot(self.check_time,
-                    self.check_if_vm_has_shutdown)
+                self.check_again_later()
         else:
+            if vm_is_running:
+                # Due to unknown reasons, Xen sometimes reports that a domain
+                # is running even though its start-up timestamp is not valid.
+                # Make sure that "restart_vm_if_needed" is not called until
+                # the domain has been completely shut down according to Xen.
+                self.check_again_later()
+                return
 
-            if self.and_restart:
-                if self.caller:
-                    self.caller.start_vm(vm)
+            self.restart_vm_if_needed()
 
 class VmManagerWindow(Ui_VmManagerWindow, QMainWindow):
     row_height = 30