Notes and attachments for GSoC 2021 for Qubes OS implementing easy port forwarding.

Giulio 459b9800ea Fixes rules		3 years ago
assets	da33365f07 Added note in chart	3 years ago
Readme.md	459b9800ea Fixes rules	3 years ago

QubesOS Port Forwarding GSoC 2021

Final Summary

The Qubes codebase is complex but well organized and written. Simple tasks, such as the basic port forwarding do require to edit and commit to multiple different components of the ecosystem. As a new entry, a lot has to be learned before being able to understand the whole picture and thus being able to plan new fetures and write useful code. Furthermore, setting up a testing environment has proven to be somewhat hard and testing is anyway is currently a manual process and is a bit time consuming. In this page. The original goals of this GSOC had to be scaled down in impelementing simple and straightforward port forwarding of two types via CLI only. This page also contains the notes i took during the planning and development process.

Proposal text

Introduction

Forwarding ports to Qubes VM is currently possible only though a multi step, error prone, manual process that also requires writing custom configuration in order to be persistent between reboots. Things as simple as starting a webserver or netcat for LAN file sharing canbe eventually a troublesome and time-wasting process[1][2]. Furthermore, applications thatrely on NAT traversal protocols such as those for audio and video communications do not workin direct P2P mode with STUN and always use TURN instead[3].

Project Goals

Implement a GUI for automatic and persistent, eventually with a predefined timespan (ie: untilreboot), port forwarding. The idea is to split horizontally the "Firewall Rules" tab in the"Qubes Settings" window and add another area below it. It is already possible to forward TCP streams, however there is no GUI nor a clear dashboard and furthermore its versatility is limited. In addition, discuss and verify the possibility to implement a secure NAT traversal systemand GUI. A basic proposal could be a checkbox to enable NAT traversal requests. When the checkbox is selected, the FirwallVM will redirect NAT traversal requests to a local pythondaemon or a dedicated VM that will negotiate the NAT traversal and configure the networkaccordingly. In this case, prompt the user in Dom0 about the NAT traversal request. Of coursethe qvm-* set of tools must e able to achieve the same tasks via CLI.

Implementation

First develop and document the part related to manual port forwarding since it is both a more frequent use case and is less complicated. Depending on the problems encountered, evaluate the feasibility of secure NAT traversal.

Notes

Development

Background

Dev Repositories

All changes are in the gsoc-port-forwarding branch of each repo.

Main components involved

Current Status

How does the GUI and `qvm-firewall` configuration work?

The Qubes Manager GUI and the qvm-firewall both use the code imlemented in the Admi API Client library. The Client Library sends specific messages to the qubesd daemon. The currently supported operatins are:

admin.vm.firewall.Get
admin.vm.firewall.Set
admin.vm.firewall.Reload

These actions can be tested by using the qvm-firewall utility. It is important to note that both the client and the daemon are more flexibile compared to the settings available via the GUI.

Configuration files

If any non default configuration is set by the user, an AppVM will have a firewall.xml configuration file in the var/lib/qubes/<appvm>/ path. Deleting the file will reset the firewall to the default state and any customization will be lost.

The firewall.xml is clearly human readable and contains rules in the form:

<firewall version="2">
	<rules>
		<rule>
			<properties>
				<!-- accept outgoing to lsd.cat porto tcp port 443 -->
				<property name="action">accept</property>
				<property name="dsthost">lsd.cat</property>
				<property name="proto">tcp</property>
				<property name="dstports">443</property>
			</properties>
		</rule>
		<rule>
			<properties>
				<!-- accept outgoing to 10.132.11.1/24 proto any -->
				<property name="action">accept</property>
				<property name="dsthost">10.132.11.1/24</property>
			</properties>
		</rule>
		<rule>
			<properties>
				<!-- allow outgoing dns queries. needed for domain based rules -->
				<property name="action">accept</property>
				<property name="specialtarget">dns</property>
			</properties>
		</rule>
		<rule>
			<properties>
				<!-- drop everything else -->
				<property name="action">drop</property>
			</properties>
		</rule>
	</rules>
</firewall>

Commands

The following command will return the firewall rules for <vmnname>.

qvm-firewall <vmname>

As can be seen, the output will show more colums that the GUI, specifically an EXPIRE, COMMENT, and SPECIAL TARGET columns will be displayed.

The following command will reload the persistent rules stored in firewall.xml of <vmname>

qvm-firewall <vmname> --reload

The following command can be used to add a rule. Not that if the GUI detects that the firewall has been edited from CLI, since it does not support all CLI settings, it will refuse to allow management again from the GUI.

qvm-firewall <vmname> add action=accept dsthost=1.1.1.1 proto=tcp dstports=80-80 expire=+5000 comment="cloudflare http test rule"

Proposal

Currently, all firewall rules have an action properties which can be either accept or drop. The plan is to add a third option forward specifically for implementing automatic port forwarding. Such options must be supported both in the configuration file and in the Admin API (client-server). Lastly, it shall be implemented in the agent daemon. The main issue however is the fact that currenly, the firewall client library is designated to operate only on the AppVM configured Firewall NetVM. However, in order to forward ports from the outside world, specific rules needs to be applied to the Firewall NetVM Networking NetVM. (ie: both is sys-firewall and sys-net, as currently done for manual port forwarding).

action=forward

Since in the case of port forwarding the target ip address would always be the <vmname> IP address, users should not be asked for a dsthost field. Adding a forward rule could look like this:

qvm-firewall <vmname> add action=forward proto=tcp forwardtype=external srcports=443-443 dstports=80443-80443 srchost=0.0.0.0/0 expire=+500000 comment="example https server rule"
qvm-firewall <vmname> add action=forward proto=tcp forwardtype=internal srcports=80-80 dstports=8000-8000 srchost=10.137.0.13 expire=+500000 comment="example internal simplehttpserver file sharing rule"

Of course expire= and comment= are optional fields.

		<rule>
			<properties>
				<!-- sample syntax for port forwarding -->
				<property name="action">forward</property>
				<property name="proto">tcp</property>
				<property name="forwardtype">external</property>
				<property name="srcports">443-443</property>
				<property name="dstports">80443-80443</property>
				<property name="srchost">0.0.0.0/0</property>
				<property name="comment">example https server rule</property>
			</properties>
		</rule>

Proposal chart

The main distinction between internal and external port forwarding is:

Internal resolves only <appvm>'s <ntvm>
External recursively resolves all upstream networking vm and sets forwarding rules on all of them

This should cover multiple scenarios:

Standard external forwarding when <appvm> service needs to be exposed on a physical interface
Standard internal forwarding when <appvm> service needs to be exposed to other <appvm2/3/4> connected to the same <netvm>
VPN internal port forwarding when <appvm> service needs to be exposed through a VPN

It is important to note that in the last case, it is just a standard case of internal forwarding.

Implementation Roadmap

✔️ In core-admin-client/qubesadmin/firewall.py firewall.py -> The code needs to support the new options for the rule (action=forward frowardtype= srcports=443-443 srchosts=0.0.0.0/0
✔️ In core-admin/qubes/firewall.py -> The code needs to support the same options as the point above
✔️ In core-admin/qubes/vm/mix/net.py -> The most important logic goes here. Here there is the need to resolve the full network chain for external port forwarding. From here it is possible to add the respective rules to the QubesDB of each NetVM in he chain and trigger a reload event.
✔️ In core-agent-linux/qubesagent/firewall.py -> Here goes the logic for building the correct syntax for iptables or nft and the actual execution
❌ Tests
❌ GUI

Steps 1-3 are completed and needs the automated test. Step 4 has still some issues but it is in its final stages. 5 will be worked on in the following weeks, since it is mandatory before merging anything. 6 can come at a later stage.

Known Issues

Currently, in the destination Qube, such as the personal or work, or any other qube that does not provide networking, the systemd unit qubes-firewall is not started by default. Currently, each domain of this kind has a set of predefined iptables rules that will be deprecated as soon as the full switch to nft is completed. In the meantime, in order to use the port forwarding succesfully, it is necessary to drop such rules and thus stop the service with:

sudo systemctl stop qubes-iptables

Required rules

External

The iptables backend in the firewall worker is being deprecated. If the nft binary is available on the target Qubes, iptables will be never involved. Thus, only nft rules are relevant in this context.

Sample setup:

sys-net - 10.137.0.5 (ens6 phy with 192.168.10.20)
sys-firewall - 10.137.0.6
personal - 10.137.0.7

All of them are running fedora-32.

And assume the following rule added via qvm-firewall:

$ qvm-firewall personal add action=forward forwardtype=external scrports=22-22 proto=tcp dstports=2222-2222 srchost=192.168.10.0/24

First, a table for the forwarding rules is created:

table {family} qubes-firewall-forward {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
    }
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        }
}

Then, if the qube is marked as 'last', meaning that it is the external qube with the physical interface the following rules are added:

table {family} qubes-firewall-forward {
    chain prerouting {
        meta iifname "ens6" {family} saddr 192.168.10.0/24 tcp dport {{ 22 }} dnat to 10.137.0.6:2222
    }
}

table {family} qubes-firewall {
    chain forward {
        meta iifname "eth0" {family} daddr 10.137.0.6 tcp dport 2222 ct state new counter accept
    }
}

And that is all for sys-net.

In sys-firewall, since it is an 'internal' qube, the following rules are added instead:

table {family} qubes-firewall-forward {
    chain prerouting {
        meta iifname "eth0" {family} saddr 120.137.0.5 tcp dport {{ 2222 }} dnat to 10.137.0.7:2222
    }
}

table {family} qubes-firewall {
    chain forward {
        meta iifname "eth0" {family} daddr 10.137.0.7 tcp dport 2222 ct state new counter accept
    }
}

Lastly, some rules need to be added in the target Qube in order to accept the incoming connections. Since the target Qube does not have a running firewall worker, the method for doing this has yet to be determined.

Extra

QubesDB Debugging

Since all firewall rules are written to the respective domains QubesDB by the qubesd it is essential dor debugging purposes to be able to easily read QubesDB entries. The QubesOS Project provide some useful utilities to interact with each DB. Such utilities have self explicative names and works like the respective functions used in the source code. The most useful are:

qubesdb-list
qubesdb-read
qubesdb-write

Useful example:

 # qubesdb-list -fr /qubes-firewall/ -d sys-firewall
 # qubesdb-read -fr /qubes-firewall/10.137.0.2/0001 -d sys-firewall

Flags

Flags explanation as produced from the qvm-ls utility:

        Type of domain (When it is HVM, the letter is capital).

        0   AdminVM (AKA Dom0)
        aA  AppVM
        dD  DisposableVM
        sS  StandaloneVM
        tT  TemplateVM

        Current power state.

        r   running
        t   transient
        p   paused
        s   suspended
        h   halting
        d   dying
        c   crashed
        ?   unknown

        Extra

     	U   updateable
     	N   provides_network
     	R   installed_by_rpm
     	i   internal
     	D   debug
     	A   autostart

Dev Environment

Currently developing on VMWare Workstation on Windows due to issues in virtualizing on linux on my home hardware. QubesOS is virtualized behind NAT and can reach Windows Host via SSH.

In order to test the code, I wrote some helper scripts. The required setup involves:

Clone the tools on the Windows Host
Generate an SSH keypair on sys-net
Add sys-net SSH pubkey on Windows for non interactive authentication (sshd is easier using Windows Subsystem for Linux)
Via scp/sftp, copy all the bash script in the sys-net VM. Leve pull.sh in /home/user/pull.sh
Using qvm-run copy backup.sh, restore.sh and updte.sh in Dom0
First, run once backup.sh and pay attention to never run it again in order to recover from broken states (breaking qubesd, qvm-run will stop working and it will be hard to recover)
Run update.sh to automatically pull changes from the Windows host. qubesd is restarted within the same script.
In case of issues, run restore.sh and investigate the previous errors

Nft Debugging

To debug rules with nft, it is necessary to add a trace rule to each relevant table-chain:

nft add rule qubes-firewall forward meta nftrace 1
nft add rule qubes-firewall prerouting meta nftrace 1
nft add rule qubes-firewall postrouting meta nftrace 1
nft add rule qubes-firewall-forward postrouting meta nftrace 1
nft add rule qubes-firewall-forward postrouting meta nftrace 1

Then, the rule processing log can be monitored running:

nft monitor trace

Readme.md