storage: add API documentation

QubesOS/qubes-issues#2256

doc/index.rst

@@ -16,6 +16,7 @@ manpages and API documentation. For primary user documentation, see
    qubes
    qubes-vm/index
    qubes-events
+   qubes-storage
    qubes-exc
    qubes-ext
    qubes-log

doc/qubes-storage.rst

@@ -0,0 +1,145 @@
+:py:mod:`qubes.storage` -- Qubes data storage
+=============================================
+
+Qubes provide extensible API for domains data storage. Each domain have
+multiple storage volumes, for different purposes. Each volume is provided by
+some storage pool. Qubes support different storage pool drivers, and it's
+possible to register additional 3rd-party drivers.
+
+Domain's storage volumes:
+
+ - `root` - this is where operating system is installed. The volume is
+   available read-write to :py:class:`~qubes.vm.templatevm.TemplateVM` and
+   :py:class:`~qubes.vm.standalonevm.StandaloneVM`, and read-only to others
+   (:py:class:`~qubes.vm.appvm.AppVM` and :py:class:`~qubes.vm.dispvm.DispVM`).
+ - `private` - this is where domain's data live. The volume is available
+   read-write to all domain classes (including :py:class:`~qubes.vm.dispvm.DispVM`,
+   but data written there is discarded on domain shutdown).
+ - `volatile` - this is used for any data that do not to persist. This include
+   swap, copy-on-write layer for `root` volume etc.
+ - `kernel` - domain boot files - operating system kernel, initial ramdisk,
+   kernel modules etc. This volume is provided read-only and should be provided by
+   a storage pool respecting :py:attr:`qubes.vm.qubesvm.QubesVM.kernel` property.
+
+Storage pool concept
+--------------------
+
+Storage pool is responsible for managing its volumes. Qubes have defined
+storage pool driver API, allowing to put domains storage in various places. By
+default two drivers are provided: :py:class:`qubes.storage.file.FilePool`
+(named `file`) and :py:class:`qubes.storage.lvm.ThinPool` (named `lvm_thin`).
+But the API allow to implement variety of other drivers (like additionally
+encrypted storage, external disk, drivers using special features of some
+filesystems like btrfs, etc).
+
+Most of storage API focus on storage volumes. Each volume have at least those
+properties:
+ - :py:attr:`~qubes.storage.Volume.rw` - should the volume be available
+   read-only or read-write to the domain
+ - :py:attr:`~qubes.storage.Volume.snap_on_start` - should the domain start
+   with its own state of the volume, or rather a snapshot of its template volume
+   (pointed by a :py:attr:`~qubes.storage.Volume.source` property). This can be
+   set to `True` only if a domain do have `template` property (AppVM and DispVM).
+   If the domain's template is running already, the snapshot should be made out of
+   the template's before its startup.
+ - :py:attr:`~qubes.storage.Volume.save_on_stop` - should the volume state be
+   saved or discarded on domain
+   stop. In either case, while the domain is running, volume's current state
+   should not be committed immediately. This is to allow creating snapshots of the
+   volume's state from before domain start (see
+   :py:attr:`~qubes.storage.Volume.snap_on_start`).
+ - :py:attr:`~qubes.storage.Volume.revisions_to_keep` - number of volume
+   revisions to keep. If greater than zero, at each domain stop (and if
+   :py:attr:`~qubes.storage.Volume.save_on_stop` is `True`) new revision is saved
+   and old ones exceeding :py:attr:`~qubes.storage.Volume.revisions_to_keep` limit
+   are removed.
+ - :py:attr:`~qubes.storage.Volume.source` - source volume for
+   :py:attr:`~qubes.storage.Volume.snap_on_start` volumes
+ - :py:attr:`~qubes.storage.Volume.vid` - pool specific volume identifier, must
+   be unique inside given pool
+ - :py:attr:`~qubes.storage.Volume.pool` - storage pool object owning this volume
+ - :py:attr:`~qubes.storage.Volume.name` - name of the volume inside owning
+   domain (like `root`, or `private`)
+ - :py:attr:`~qubes.storage.Volume.size` - size of the volume, in bytes
+
+Storage pool driver may define additional properties.
+
+Storage pool driver API
+-----------------------
+
+Storage pool driver need to implement two classes:
+ - pool class - inheriting from :py:class:`qubes.storage.Pool`
+ - volume class - inheriting from :py:class:`qubes.storage.Volume`
+
+Pool class should be registered with `qubes.storage` entry_point, under the
+name of storage pool driver. Volume class instances should be returned by
+:py:meth:`qubes.storage.Pool.init_volume` method of pool class instance.
+
+Methods required to be implemented by the pool class:
+ - :py:meth:`~qubes.storage.Pool.init_volume` - return instance of appropriate
+   volume class; this method should not alter any persistent disk state, it is
+   used to instantiate both existing volumes and create new ones
+ - :py:meth:`~qubes.storage.Pool.setup` - setup new storage pool
+ - :py:meth:`~qubes.storage.Pool.destroy` - destroy storage pool
+
+Methods and properties required to be implemented by the volume class:
+ - :py:meth:`~qubes.storage.Volume.create` - create volume on disk
+ - :py:meth:`~qubes.storage.Volume.remove` - remove volume from disk
+ - :py:meth:`~qubes.storage.Volume.start` - prepare the volume for domain start;
+   this include making a snapshot if
+   :py:attr:`~qubes.storage.Volume.snap_on_start` is `True`
+ - :py:meth:`~qubes.storage.Volume.stop` - cleanup after domain shutdown; this
+   include committing changes to the volume if
+   :py:attr:`~qubes.storage.Volume.save_on_stop` is `True`
+ - :py:meth:`~qubes.storage.Volume.export` - return a path to be read to extract
+   volume data; for complex formats, this can be a pipe (connected to some
+   data-extracting process)
+ - :py:meth:`~qubes.storage.Volume.import_data` - return a path the data should
+   be written to, to import volume data; for complex formats, this can be pipe
+   (connected to some data-importing process)
+ - :py:meth:`~qubes.storage.Volume.import_data_end` - finish data import
+   operation (cleanup temporary files etc); this methods is called always after
+   :py:meth:`~qubes.storage.Volume.import_data` regardless if operation was
+   successful or not
+ - :py:meth:`~qubes.storage.Volume.import_volume` - import data from another volume
+ - :py:meth:`~qubes.storage.Volume.resize` - resize volume
+ - :py:meth:`~qubes.storage.Volume.revert` - revert volume state to a given revision
+ - :py:attr:`~qubes.storage.Volume.revisions` - collection of volume revisions (to use
+   with :py:meth:`qubes.storage.Volume.revert`)
+ - :py:meth:`~qubes.storage.Volume.is_dirty` - is volume properly committed
+   after domain shutdown? Applies only to volumes with
+   :py:attr:`~qubes.storage.Volume.save_on_stop` set to `True`
+ - :py:meth:`~qubes.storage.Volume.is_outdated` - have the source volume started
+   since domain startup? applies only to volumes with
+   :py:attr:`~qubes.storage.Volume.snap_on_start` set to `True`
+ - :py:attr:`~qubes.storage.Volume.config` - volume configuration, this should
+   be enough to later reinstantiate the same volume object
+ - :py:meth:`~qubes.storage.Volume.block_device` - return
+   :py:class:`qubes.storage.BlockDevice` instance required to configure volume in
+   libvirt
+
+Some storage pool drivers can provide limited functionality only - for example
+support only `volatile` volumes (those with
+:py:attr:`~qubes.storage.Volume.snap_on_start` is `False`,
+:py:attr:`~qubes.storage.Volume.save_on_stop` is `False`, and
+:py:attr:`~qubes.storage.Volume.rw` is `True`). In that case, it should raise
+:py:exc:`NotImplementedError` in :py:meth:`qubes.storage.Pool.init_volume` when
+trying to instantiate unsupported volume.
+
+Note that pool driver should be prepared to recover from power loss before
+stopping a domain - so, if volume have
+:py:attr:`~qubes.storage.Volume.save_on_stop` is `True`, and
+:py:meth:`qubes.storage.Volume.stop` wasn't called, next
+:py:meth:`~qubes.storage.Volume.start` should pick up previous (not committed)
+state.
+
+See specific methods documentation for details.
+
+Module contents
+---------------
+
+.. automodule:: qubes.storage
+   :members:
+   :show-inheritance:
+
+.. vim: ts=3 sw=3 et

qubes/storage/__init__.py

@@ -78,6 +78,8 @@ class Volume(object):
     domain = None
     path = None
     script = None
+    #: disk space used by this volume, can be smaller than :py:attr:`size`
+    #: for sparse volumes
     usage = 0
 
     def __init__(self, name, pool, vid, internal=False, removable=False,
@@ -85,7 +87,7 @@ class Volume(object):
             snap_on_start=False, source=None, **kwargs):
         ''' Initialize a volume.
 
-            :param str name: The domain name
+            :param str name: The name of the volume inside owning domain
             :param Pool pool: The pool object
             :param str vid:  Volume identifier needs to be unique in pool
             :param bool internal: If `True` volume is hidden when qvm-block ls
@@ -94,9 +96,12 @@ class Volume(object):
                 run time
             :param int revisions_to_keep: Amount of revisions to keep around
             :param bool rw: If true volume will be mounted read-write
-            :param bool snap_on_start: Create a snapshot from source on start
-            :param bool save_on_stop: Write changes to disk in vm.stop()
-            :param Volume source: other volume in same pool, or None
+            :param bool snap_on_start: Create a snapshot from source on
+                start, instead of using volume own data
+            :param bool save_on_stop: Write changes to the volume in
+                vm.stop(), otherwise - discard
+            :param Volume source: other volume in same pool to make snapshot
+                from, required if *snap_on_start*=`True`
             :param str/int size: Size of the volume
 
         '''
@@ -106,16 +111,27 @@ class Volume(object):
         assert source is None or (isinstance(source, Volume)
                                   and source.pool == pool)
 
+        #: Name of the volume in a domain it's attached to (like `root` or
+        #: `private`).
         self.name = str(name)
+        #: :py:class:`Pool` instance owning this volume
         self.pool = pool
         self.internal = internal
         self.removable = removable
+        #: How many revisions of the volume to keep. Each revision is created
+        #  at :py:meth:`stop`, if :py:attr:`save_on_stop` is True
         self.revisions_to_keep = int(revisions_to_keep)
+        #: Should this volume be writable by domain.
         self.rw = rw
+        #: Should volume state be saved or discarded at :py:meth:`stop`
         self.save_on_stop = save_on_stop
         self._size = int(size)
+        #: Should the volume state be initialized with a snapshot of
+        #: same-named volume of domain's template.
         self.snap_on_start = snap_on_start
+        #: source volume for :py:attr:`snap_on_start` volumes
         self.source = source
+        #: Volume unique (inside given pool) identifier
         self.vid = vid
 
     def __eq__(self, other):
@@ -142,6 +158,10 @@ class Volume(object):
     def create(self):
         ''' Create the given volume on disk.
 
+            This method is called only once in the volume lifetime. Before
+            calling this method, no data on disk should be touched (in
+            context of this volume).
+
             This can be implemented as a coroutine.
         '''
         raise self._not_implemented("create")
@@ -153,17 +173,37 @@ class Volume(object):
         raise self._not_implemented("remove")
 
     def export(self):
-        ''' Returns an object that can be `open()`. '''
+        ''' Returns a path to read the volume data from.
+
+            Reading from this path when domain owning this volume is
+            running (i.e. when :py:meth:`is_dirty` is True) should return the
+            data from before domain startup.
+
+            Reading from the path returned by this method should return the
+            volume data. If extracting volume data require something more
+            than just reading from file (for example connecting to some other
+            domain, or decompressing the data), the returned path may be a pipe.
+        '''
         raise self._not_implemented("export")
 
     def import_data(self):
-        ''' Returns an object that can be `open()`. '''
+        ''' Returns a path to overwrite volume data.
+
+            This method is called after volume was already :py:meth:`create`-ed.
+
+            Writing to this path should overwrite volume data. If importing
+            volume data require something more than just writing to a file (
+            for example connecting to some other domain, or converting data
+            on the fly), the returned path may be a pipe.
+        '''
         raise self._not_implemented("import")
 
     def import_data_end(self, success):
-        ''' End data import operation. This may be used by pool
+        ''' End the data import operation. This may be used by pool
         implementation to commit changes, cleanup temporary files etc.
 
+        This method is called regardless the operation was successful or not.
+
         :param success: True if data import was successful, otherwise False
         '''
         # by default do nothing
@@ -173,19 +213,24 @@ class Volume(object):
         ''' Imports data from a different volume (possibly in a different
         pool.
 
-        The needs to be create()d first.
+        The volume needs to be create()d first.
 
         This can be implemented as a coroutine. '''
         # pylint: disable=unused-argument
         raise self._not_implemented("import_volume")
 
     def is_dirty(self):
-        ''' Return `True` if volume was not properly shutdown and commited '''
+        ''' Return `True` if volume was not properly shutdown and committed.
+
+            This include the situation when domain owning the volume is still
+            running.
+
+        '''
         raise self._not_implemented("is_dirty")
 
     def is_outdated(self):
-        ''' Returns `True` if the currently used `volume.source` of a snapshot
-            volume is outdated.
+        ''' Returns `True` if this snapshot of a source volume (for
+        `snap_on_start`=True) is outdated.
         '''
         raise self._not_implemented("is_outdated")
 
@@ -195,23 +240,33 @@ class Volume(object):
             given size is less than current_size
 
             This can be implemented as a coroutine.
+
+            :param int size: new size in bytes
         '''
         # pylint: disable=unused-argument
         raise self._not_implemented("resize")
 
     def revert(self, revision=None):
-        ''' Revert volume to previous revision  '''
+        ''' Revert volume to previous revision
+
+        :param revision: revision to revert volume to, see :py:attr:`revisions`
+        '''
         # pylint: disable=unused-argument
         raise self._not_implemented("revert")
 
     def start(self):
-        ''' Do what ever is needed on start
+        ''' Do what ever is needed on start.
+
+        This include making a snapshot of template's volume if
+        :py:attr:`snap_on_start` is set.
 
         This can be implemented as a coroutine.'''
         raise self._not_implemented("start")
 
     def stop(self):
-        ''' Do what ever is needed on stop
+        ''' Do what ever is needed on stop.
+
+        This include committing data if :py:attr:`save_on_stop` is set.
 
         This can be implemented as a coroutine.'''
 
@@ -230,12 +285,14 @@ class Volume(object):
 
     @property
     def revisions(self):
-        ''' Returns a `dict` containing revision identifiers and paths '''
+        ''' Returns a dict containing revision identifiers and time of their
+        creation '''
         msg = "{!s} has revisions not implemented".format(self.__class__)
         raise NotImplementedError(msg)
 
     @property
     def size(self):
+        ''' Volume size in bytes '''
         return self._size
 
     @size.setter