Oggetto: GSoC Port Forwarding |
Mittente: Giulio |
Data: 11/06/2021, 08:24 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>, Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 11/06/2021, 09:16 |
A: Giulio <giulio@gmx.com>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hello, ++
Thank you for accepting my proposal and volunteering for mentoring me. I +
spent the last weeks reading Qubes sources, documentation and mailing +
lists, as well as setting up a virtual machine and attempting to prepare +
a comfy development environment. I Hope by the end of the next week to +
be ready to propose you a draft of the plan for the development of the +
static port forwardign feature. Does that sound ok? +
Cheers, ++
Giulio +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 20/06/2021, 22:50 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hello, +
+
Le 6/11/21 à 8:24 AM, Giulio a écrit : +Hello, ++
Thank you for accepting my proposal and volunteering for mentoring me. I +
spent the last weeks reading Qubes sources, documentation and mailing +
lists, as well as setting up a virtual machine and attempting to prepare +
a comfy development environment. I Hope by the end of the next week to +
be ready to propose you a draft of the plan for the development of the +
static port forwardign feature. Does that sound ok? +
Sure. +
+Cheers, ++
Giulio +
Best, +
Frédéric +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 22/06/2021, 01:49 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hello, +
sorry for the late reply. +
+
I read a lot of code and I have to admit that I did not grasp the +complexity of the Admin API and networking stack before looking so much +into it. I think I've got the overall picture, but it will take a little +more to fully be confident moving there. +
+
Here is the summary of the notes I've taken from my understanding in +this week of digging. +
+
Main references: +
https://www.qubes-os.org/doc/admin-api/ +
https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x +
https://www.qubes-os.org/doc/firewall/ +
+
_core-admin/qubes/firewall.py_ +
Contains the classes with the parsing and formatting rules for firewall +information in the firewall XML file. It already checks proper format +for ports and ip addresses/netmask. It already support an expiry date +for a rule. +
+
_core-admin-client/qubesadmin/tools/qvm_firewall.py_ +
Class for the qvm-firewall cli tool. It is able to view, add, delete and +reload firewall rules. +
+
+
_core-admin-client/qubesadmin/firewall.py_ +
The file responsible for calling Admin API (qubesd). Currently has its +own rule syntax for setting rules. +
+
_core-agent-linux/qubesagent/firewall.py_ +
Is the actual file responsible for running nftables and thus +adding/deleting/reloading firewall ruless in the target firewall vm. It +also resolves DNS names for domain rules. It is run by Admin API (qubesd). +
+
_manager/qubesmanager/firewall.py_ +
Contains the code for the "Firewall" tab of the "Qube Manager" window. +
+
_manager/ui/qubemanager.ui_ +
String and properties for the "Qube Manager" UI. +
+
+
Questions: +
+
1) Should we both support internal port forwarding and external port +forwarding? Such as exposing a port for another domain or exposing a +port through the public network interface? I would say yes. +
2) Should it be possible to add rules with an 'any' clause (both tcp and +udp). I would say no because since port forwarding brings a higher +attack surface all rules should be as precise as possible. +
3) Since the expire= feature seems to be already implemented (and +limited for the expiring full outgoing access) would it be useful to be +implemented in gui and cli for every rule? I would say yes since the +admin and agent code seems to be already there. The same goes for the +"comment=" field. +
4) How would you implement the management of forwarding rules in the +network providing domain (sys-net)? Shall the user add a rule both in +the target domain (ie the one with webserver and another one in sys-net) +or should it be fully automatic from the first? +
5) Users should be able to set forward rules using domain names and not +static ip addresses. In this case, the actual ip addresses of the dst +domains should be collected in a similr way as currently DNS are +resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good? +
+
+
Proposed XML Syntax: +
<rule> +
<properties> +
<property name="action">forward</property> +
<property name="proto">udp</property> +
<property name="dstports">443-8080-5555</property> +
</properties> +
<rule> +
+
Proposed Admin API Syntax: +
action=forward proto=udp dstports=443-8080-5555 [expire=<unix +timestamp>] [comment=random text] +
+
I also plan to document, at least partially, the journey into this. +
As a last question, I'm curious what is your setup in order to test +modifications in the aforementioned repos while developing. +
+
Thank you for your time. +
Cheers +
Giulio +
+
+
+
Il 11/06/2021 09:16, Frédéric Pierret ha scritto: +Hello, +
+
Le 6/11/21 à 8:24 AM, Giulio a écrit : +Hello, ++
Thank you for accepting my proposal and volunteering for mentoring me. I +
spent the last weeks reading Qubes sources, documentation and mailing +
lists, as well as setting up a virtual machine and attempting to prepare +
a comfy development environment. I Hope by the end of the next week to +
be ready to propose you a draft of the plan for the development of the +
static port forwardign feature. Does that sound ok? +
Sure. +
+Cheers, ++
Giulio +
Best, +
Frédéric +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 22/06/2021, 14:28 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Hi, I'm replying to both emails at once: ++
+
On Sun, Jun 20, 2021 at 10:50:04PM +0200, Giulio wrote: +Questions: ++
+
1) Should we both support internal port forwarding and external port +
forwarding? Such as exposing a port for another domain or exposing a +
port through the public network interface? I would say yes. +
Yes, I think so. Technically, those two cases should be quite similar. +
See also the case of sys-vpn much lower in the email. +
+
+3) Since the expire= feature seems to be already implemented (and ++
limited for the expiring full outgoing access) would it be useful to be +
implemented in gui and cli for every rule? I would say yes since the +
admin and agent code seems to be already there. The same goes for the +
"comment=" field. +
Per-rule expire may be tricky to handle at the GUI level, I have no idea +
how to make the UI for this not very confusing... +
But the comment field is definitely useful to use. +
+
+4) How would you implement the management of forwarding rules in the ++
network providing domain (sys-net)? Shall the user add a rule both in +
the target domain (ie the one with webserver and another one in sys-net) +
or should it be fully automatic from the first? +
From the user point of view, I think it should be automated as much as +
possible. Like, let the user choose which port in which VM redirect to +
where. There may be cases when such redirection won't be possible - if +
there is no network path between the two points. +
+
+5) Users should be able to set forward rules using domain names and not ++
static ip addresses. In this case, the actual ip addresses of the dst +
domains should be collected in a similr way as currently DNS are +
resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good? +
But here we are mostly talking about IP addresses of different VMs, +
right? Those can (and should) be resolved at core-admin side, so the VM +
applying the rules will have all the IP given. In fact VM may not be +
able to resolve IP of another VM at all. +
+
+Proposed XML Syntax: ++
<rule> +
<properties> +
<property name="action">forward</property> +
<property name="proto">udp</property> +
<property name="dstports">443-8080-5555</property> +
</properties> +
<rule> +
I don't see an important information here: forward to _where_. +
+Proposed Admin API Syntax: ++
action=forward proto=udp dstports=443-8080-5555 [expire=<unix +
timestamp>] [comment=random text] +
Similar here, there needs to be a forward target (IP, and possibly a +
port) +
+
On Tue, Jun 22, 2021 at 01:49:15AM +0200, Giulio wrote: +Since in the case of port forwarding the target ip address would always be the <vmname> IP address, ++
This is very true. But there needs to be an information where to forward +
the traffic to (as noted earlier). Plus, possibly a second set of ports +
(if you want to redirect to a different port). +
+
+I think my main concern now is the question 4 from the aforementioned ++
email. Shall the rules be automatically implemented in all 3 involved +
vms? (<netvm,firewallvm,appvm>). I think yes, because otherwise it would +
be counterintuitive to be a partially manual and partially automatic +
operation. But since it actually 'automatically' exposes more attack +
surface, by loosening up all 3 vms network rules, I guess that maybe +
more reasoning on it would be helpful. +
Yes, but you need to pass the traffic somehow. The direct connection can +
be achieved with qvm-connnect-tcp (connecting to the target directly +
using qrexec, bypassing intermediate VMs), but it has its limits (low +
performance, TCP only). To keep it as actual IP traffic, you need to +
change firewall rules at all intermediate VMs too. +
+
Lets have a specific example: in default setup, redirect TCP port 80 +
from the outside, to 'work' VM port 8080. +
+
The setup looks like this: +
+ sys-net -> sys-firewall -> work +
+
For this, you will need those rules: +
+
1. In sys-net: forward TCP port 80 to sys-firewall +
2. In sys-firewall: forward TCP port 80 to work, port 8080 +
3. In work: allow TCP port 8080 +
+
Now is the important design question: how to store those rules? If you +
store them at all three places separately, it +
will be easier to apply them at runtime, but it will be harder to +
correlate them in UI. Plus, if any of them get modified/removed, it may +
be non-trivial to troubleshoot the issue. +
The other approach is to store the forward rules only in one place (the +
target, 'work' in this example? or the source, 'sys-net' here?). This +
way, it's harder to mess thing up. But when applying the rules (building +
rule sets for qubes-firewall service in all the involved VMs), you need +
to check several places. +
Plus, the UI should clearly show such redirected ports at all involved +
places, because it does affect system security - it must be easy to spot +
if any redirects are enabled. +
+
+
To make things more complex (sorry...), there may be a VPN or other +
proxy service (Tor?) involved. For example: +
+
sys-net -> sys-firewall -> sys-vpn -> work +
+
In such a case, the "external" VM for 'work' is not really sys-net, but +
rather sys-vpn. And actually you need to be careful to not accidentally +
bypass VPN either by allowing 'work' to communicate outside of the VPN, +
or (maybe even worse) systems on the LAN (via sys-net) reach inside VPN. +
+
This case is not easy to solve, because currently core-admin has no idea +
whether sys-vpn (or other such VM) do any of such tunnelling. Maybe we +
need to (finally) introduce some flag to mark such VMs? +
+
+
And another question: what should happen if you change netvm of 'work'. +
For example switch to something like: +
+
sys-net -> sys-firewall -> (other VMs, but not 'work') +
+
sys-wifi -> work +
+
Should the redirection stay active via sys-wifi? I think it should not, +
at least not automatically (maybe have an option for that?). +
+
And finally, don't forget IPv6 exists. Which means you can have the same ++
port on IPv4 and IPv6. And theoretically they could be redirected to +
different places (but I'm not sure if that's a good idea...). +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 22/06/2021, 04:43 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+Hi, I'm replying to both emails at once: + +On Sun, Jun 20, 2021 at 10:50:04PM +0200, Giulio wrote: +
+Hello, +sorry for the late reply. + +I read a lot of code and I have to admit that I did not grasp the +complexity of the Admin API and networking stack before looking so much +into it. I think I've got the overall picture, but it will take a little +more to fully be confident moving there. +
+ +(...) + +
+_core-agent-linux/qubesagent/firewall.py_ +Is the actual file responsible for running nftables and thus +adding/deleting/reloading firewall ruless in the target firewall vm. It +also resolves DNS names for domain rules. It is run by Admin API (qubesd). +
+ +This file is run as part of the qubes-firewall service in relevant VMs +(especially sys-firewall). + +
+_manager/qubesmanager/firewall.py_ +Contains the code for the "Firewall" tab of the "Qube Manager" window. + +_manager/ui/qubemanager.ui_ +String and properties for the "Qube Manager" UI. + + +Questions: + +1) Should we both support internal port forwarding and external port +forwarding? Such as exposing a port for another domain or exposing a +port through the public network interface? I would say yes. +
+ +Yes, I think so. Technically, those two cases should be quite similar. +See also the case of sys-vpn much lower in the email. + +
+2) Should it be possible to add rules with an 'any' clause (both tcp and +udp). I would say no because since port forwarding brings a higher +attack surface all rules should be as precise as possible. +
+ +Indeed. Furthermore, most services are either TCP or UDP, very few are +both (and for those, it's ok to require two rules). + +
+3) Since the expire= feature seems to be already implemented (and +limited for the expiring full outgoing access) would it be useful to be +implemented in gui and cli for every rule? I would say yes since the +admin and agent code seems to be already there. The same goes for the +"comment=" field. +
+ +Per-rule expire may be tricky to handle at the GUI level, I have no idea +how to make the UI for this not very confusing... +But the comment field is definitely useful to use. + +
+4) How would you implement the management of forwarding rules in the +network providing domain (sys-net)? Shall the user add a rule both in +the target domain (ie the one with webserver and another one in sys-net) +or should it be fully automatic from the first? +
+ +From the user point of view, I think it should be automated as much as +possible. Like, let the user choose which port in which VM redirect to +where. There may be cases when such redirection won't be possible - if +there is no network path between the two points. + +
+5) Users should be able to set forward rules using domain names and not +static ip addresses. In this case, the actual ip addresses of the dst +domains should be collected in a similr way as currently DNS are +resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good? +
+ +But here we are mostly talking about IP addresses of different VMs, +right? Those can (and should) be resolved at core-admin side, so the VM +applying the rules will have all the IP given. In fact VM may not be +able to resolve IP of another VM at all. + +
+Proposed XML Syntax: +<rule> + <properties> + <property name="action">forward</property> + <property name="proto">udp</property> + <property name="dstports">443-8080-5555</property> + </properties> +<rule> +
+
+I don't see an important information here: forward to _where_.
+
++Proposed Admin API Syntax: +action=forward proto=udp dstports=443-8080-5555 [expire=<unix +timestamp>] [comment=random text] +
+ +Similar here, there needs to be a forward target (IP, and possibly a +port) + +
+ +I also plan to document, at least partially, the journey into this. +As a last question, I'm curious what is your setup in order to test +modifications in the aforementioned repos while developing. +
+ +In practice, I have a separate (physical) computer to run various tests. +It can be a system installed on an external disk. But there are also +other options: +1. Test on the system you are developing on - this may be the easiest +option, but is also a risky one (it is easy to break things - may be +undesirable if you need that system for other stuff too). +2. Test in a virtual machine - it's possible to run Qubes OS inside KVM +with nested virtualization enabled (included emulated IOMMU - sadly +works only Intel only). This requires non-Qubes host. While it is +possible to run Qubes inside Qubes, the setup is quite challenging and +fragile, so I don't recommend this path. +3. As a last resort, I have a bunch test systems and I can give you +access to one of them via SSH. But developing network-related changes, +accessing the system through network, may not be the wisest thing to +do... + +You can also find some helpful info here: +https://www.qubes-os.org/doc/#debugging + +On Tue, Jun 22, 2021 at 01:49:15AM +0200, Giulio wrote: +
+Hi, +sorry for yesterday long and a bit confusing message. I started writing +down my documentation and progress here https://git.lsd.cat/Qubes/gsoc +so it should me more readable and easier to follow. +
+ +No worries. Some comments for this page: + +
+qvm-firewall <vmname> add action=accept dsthost=1.1.1.1 proto=tcp dstports=80-80 command="cloudflare http test rule" expire=+5000
++ +It's "comment", not "command". And also, if you ever need to write Admin +API level line manually, the comment field must be last. + +(but on qvm-firewall cmdline it is fine as is) + +
+Since in the case of port forwarding the target ip address would always be the <vmname> IP address,
++ +This is very true. But there needs to be an information where to forward +the traffic to (as noted earlier). Plus, possibly a second set of ports +(if you want to redirect to a different port). + +
+I think my main concern now is the question 4 from the aforementioned +email. Shall the rules be automatically implemented in all 3 involved +vms? (<netvm,firewallvm,appvm>). I think yes, because otherwise it would +be counterintuitive to be a partially manual and partially automatic +operation. But since it actually 'automatically' exposes more attack +surface, by loosening up all 3 vms network rules, I guess that maybe +more reasoning on it would be helpful. +
+ +Yes, but you need to pass the traffic somehow. The direct connection can +be achieved with qvm-connnect-tcp (connecting to the target directly +using qrexec, bypassing intermediate VMs), but it has its limits (low +performance, TCP only). To keep it as actual IP traffic, you need to +change firewall rules at all intermediate VMs too. + +Lets have a specific example: in default setup, redirect TCP port 80 +from the outside, to 'work' VM port 8080. + +The setup looks like this: + + sys-net -> sys-firewall -> work + +For this, you will need those rules: + +1. In sys-net: forward TCP port 80 to sys-firewall +2. In sys-firewall: forward TCP port 80 to work, port 8080 +3. In work: allow TCP port 8080 + +Now is the important design question: how to store those rules? If you +store them at all three places separately, it +will be easier to apply them at runtime, but it will be harder to +correlate them in UI. Plus, if any of them get modified/removed, it may +be non-trivial to troubleshoot the issue. +The other approach is to store the forward rules only in one place (the +target, 'work' in this example? or the source, 'sys-net' here?). This +way, it's harder to mess thing up. But when applying the rules (building +rule sets for qubes-firewall service in all the involved VMs), you need +to check several places. +Plus, the UI should clearly show such redirected ports at all involved +places, because it does affect system security - it must be easy to spot +if any redirects are enabled. + + +To make things more complex (sorry...), there may be a VPN or other +proxy service (Tor?) involved. For example: + +sys-net -> sys-firewall -> sys-vpn -> work + +In such a case, the "external" VM for 'work' is not really sys-net, but +rather sys-vpn. And actually you need to be careful to not accidentally +bypass VPN either by allowing 'work' to communicate outside of the VPN, +or (maybe even worse) systems on the LAN (via sys-net) reach inside VPN. + +This case is not easy to solve, because currently core-admin has no idea +whether sys-vpn (or other such VM) do any of such tunnelling. Maybe we +need to (finally) introduce some flag to mark such VMs? + + +And another question: what should happen if you change netvm of 'work'. +For example switch to something like: + + sys-net -> sys-firewall -> (other VMs, but not 'work') + + sys-wifi -> work + +Should the redirection stay active via sys-wifi? I think it should not, +at least not automatically (maybe have an option for that?). + + +And finally, don't forget IPv6 exists. Which means you can have the same +port on IPv4 and IPv6. And theoretically they could be redirected to +different places (but I'm not sure if that's a good idea...). + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 22/06/2021, 16:04 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Tue, Jun 22, 2021 at 02:28:26PM +0200, Giulio wrote: +
+Hello, +thank you for the detailed response. + +Il 22/06/2021 04:43, Marek Marczykowski-Górecki ha scritto: ++Hi, I'm replying to both emails at once: + +On Sun, Jun 20, 2021 at 10:50:04PM +0200, Giulio wrote: ++Questions: + +1) Should we both support internal port forwarding and external port +forwarding? Such as exposing a port for another domain or exposing a +port through the public network interface? I would say yes. ++ +Yes, I think so. Technically, those two cases should be quite similar. +See also the case of sys-vpn much lower in the email. + ++ +I think that I'm actually failing to picture all the possible internal +scenarios. + +1) In the case of external port forwarding <sys-net> should forward to +<sys-firewall> and <sys-firewall> then to the <appvm>. +In this case the port gets forwarded on the external interface ie: a LAN +or a public ip address depending on the network environment. +
+ +Yes. + +
+2) In the case of internal port forwarding, the port is forwarded only +from <sys-firewall> to <appvm>. In that case, another <appvm2> can visit +the <appvm> service using <sys-firewall> ip address and the chosen port. +
+ +Yes. + +
+In this case, the ports get exposed on <sys-firewall> and thus depending +on how the rules are implemented, may be available to all the AppVMs +that share the same <sys-firewall>. +
+ +Yes. + +
+In both cases may be important to allow to specify access rules for the +forwarded port, such as the lan/public ip addresses ranges allowed for +case 1 and the appvm name for case 2. +
+ +Yes, indeed. + +
+3) Since the expire= feature seems to be already implemented (and +limited for the expiring full outgoing access) would it be useful to be +implemented in gui and cli for every rule? I would say yes since the +admin and agent code seems to be already there. The same goes for the +"comment=" field. ++ +Per-rule expire may be tricky to handle at the GUI level, I have no idea +how to make the UI for this not very confusing... +But the comment field is definitely useful to use. + ++ +How do you see the same checkbox that actually allows full internet +access with the 5 minutes expiration time, displayed also on the window +for adding a rule? +
+ +This may be more relevant to longer times. With times like 5min, just +setting the rules up (if you want more than one of them) may already eat +up significant portion of the expiration time... + +
+However I think there is time to think more through this as the UI will +be the last component. + ++4) How would you implement the management of forwarding rules in the +network providing domain (sys-net)? Shall the user add a rule both in +the target domain (ie the one with webserver and another one in sys-net) +or should it be fully automatic from the first? ++ +From the user point of view, I think it should be automated as much as +possible. Like, let the user choose which port in which VM redirect to +where. There may be cases when such redirection won't be possible - if +there is no network path between the two points. + ++ +I agree with you. We might just check when the user adds an internal +forwarding rule if both the source and the destination shares the same +<firewallvm>, don't we? +
+ +I think we can simplify it even further: allow forwarding ports only +from (any of) upstream VMs. For example in this case: + + appvm1 -> sys-firewall -> sys-net + | | + appvm2 ------+ | + | + appvm3 -> some-other-firewall -+ + +Allow forwarding to appvm1 only from sys-net (external case) or +sys-firwall, but not appvm3 or some-other-firewall. +Then, within the forward rule configuration you can restrict access +rules (like you propose below, with default 0.0.0.0/0). This restriction +will work for VMs directly connected to sys-firewall only, because there +is NAT (sys-net does not know whether its appvm1 or appvm2 - it only +sees sys-firewall IP in those cases). But I think it's ok to make this +limitation and require VMs to be connected to the same <firewallvm> if +you want to forward traffic between them. + +I think you did it right with the internal/external type. + +Allowing forwarding from others (like some-other-firewall in the picture +above) may be tricky (and unreliable), as it will be hard to restrict +who can really connect (sys-net have no idea which VM behind +sys-firewall/some-other-firewall really connects). + +
+5) Users should be able to set forward rules using domain names and not +static ip addresses. In this case, the actual ip addresses of the dst +domains should be collected in a similr way as currently DNS are +resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good? ++ +But here we are mostly talking about IP addresses of different VMs, +right? Those can (and should) be resolved at core-admin side, so the VM +applying the rules will have all the IP given. In fact VM may not be +able to resolve IP of another VM at all. + ++ +Thanks for the insight, it totally makes sense. + ++Proposed XML Syntax: +<rule> + <properties> + <property name="action">forward</property> + <property name="proto">udp</property> + <property name="dstports">443-8080-5555</property> + </properties> +<rule> ++ +I don't see an important information here: forward to _where_. + ++Proposed Admin API Syntax: +action=forward proto=udp dstports=443-8080-5555 [expire=<unix +timestamp>] [comment=random text] ++ +Similar here, there needs to be a forward target (IP, and possibly a +port) + +On Tue, Jun 22, 2021 at 01:49:15AM +0200, Giulio wrote: ++Since in the case of port forwarding the target ip address would always be the <vmname> IP address, ++ +This is very true. But there needs to be an information where to forward +the traffic to (as noted earlier). Plus, possibly a second set of ports +(if you want to redirect to a different port). + ++ +I am still failing to understand something here, could you give me a an +example on when the dsthosts would different rather than the <appvm> or +<firewallvm> ip? +
+ +I'm talking about the other end of the connection. Let me summarize +this for different rules: + +1. For allow/deny rules, it is about where the VM can connect to - +outgoing connection. The source IP is always the VM's IP (implicitly), +and the rule can specify destination IP, protocol, port. + +2. For forward rules, it is about incoming connection - the destination +IP is VM's IP (implicitly), but then the connection is redirected to +somewhere else (like some appvm) - and the rule needs to point out to +where it should redirect. + +If you have simplified case like this: + + sys-net -> appvm + +Then, the rule in sys-net not only needs to know what to redirect (like, +TCP port 80), but also needs to know to redirect it to appvm, not +anywhere else. +Similarly, if the rule is stored with appvm, it needs to know to +install the redirect in sys-net and not some intermediate other VM (as +talked about internal, or VPN or Tor cases). + +Ok, reading your response further, I think you covered it with +external/internal type, so it should be good. + +
+I think my main concern now is the question 4 from the aforementioned +email. Shall the rules be automatically implemented in all 3 involved +vms? (<netvm,firewallvm,appvm>). I think yes, because otherwise it would +be counterintuitive to be a partially manual and partially automatic +operation. But since it actually 'automatically' exposes more attack +surface, by loosening up all 3 vms network rules, I guess that maybe +more reasoning on it would be helpful. ++ +Yes, but you need to pass the traffic somehow. The direct connection can +be achieved with qvm-connnect-tcp (connecting to the target directly +using qrexec, bypassing intermediate VMs), but it has its limits (low +performance, TCP only). To keep it as actual IP traffic, you need to +change firewall rules at all intermediate VMs too. + +Lets have a specific example: in default setup, redirect TCP port 80 +from the outside, to 'work' VM port 8080. + +The setup looks like this: + + sys-net -> sys-firewall -> work + +For this, you will need those rules: + +1. In sys-net: forward TCP port 80 to sys-firewall +2. In sys-firewall: forward TCP port 80 to work, port 8080 +3. In work: allow TCP port 8080 + +Now is the important design question: how to store those rules? If you +store them at all three places separately, it +will be easier to apply them at runtime, but it will be harder to +correlate them in UI. Plus, if any of them get modified/removed, it may +be non-trivial to troubleshoot the issue. +The other approach is to store the forward rules only in one place (the +target, 'work' in this example? or the source, 'sys-net' here?). This +way, it's harder to mess thing up. But when applying the rules (building +rule sets for qubes-firewall service in all the involved VMs), you need +to check several places. +Plus, the UI should clearly show such redirected ports at all involved +places, because it does affect system security - it must be easy to spot +if any redirects are enabled. + + +To make things more complex (sorry...), there may be a VPN or other +proxy service (Tor?) involved. For example: + +sys-net -> sys-firewall -> sys-vpn -> work + +In such a case, the "external" VM for 'work' is not really sys-net, but +rather sys-vpn. And actually you need to be careful to not accidentally +bypass VPN either by allowing 'work' to communicate outside of the VPN, +or (maybe even worse) systems on the LAN (via sys-net) reach inside VPN. + +This case is not easy to solve, because currently core-admin has no idea +whether sys-vpn (or other such VM) do any of such tunnelling. Maybe we +need to (finally) introduce some flag to mark such VMs? + + +And another question: what should happen if you change netvm of 'work'. +For example switch to something like: + + sys-net -> sys-firewall -> (other VMs, but not 'work') + + sys-wifi -> work + +Should the redirection stay active via sys-wifi? I think it should not, +at least not automatically (maybe have an option for that?). + ++ +I understand all of your points and consequently it is hard to figure +out a catch-all solution. + +I tried charting the flow of the possible solution. +https://git.lsd.cat/Qubes/gsoc/src/master/assets/implementation.png + +As a sum up: +1) Rules are stored only in <appvm>/firewall.xml +2) Rules can either be internal or exteranl (ie: they are applied only +to <firewallvm> or both to <firewallvm> and its <netvm>) +
+ +Please note there may be more VMs in between, some examples at: +https://www.qubes-os.org/doc/firewall/ + +So, I think it's better to define it as: +1. internal: redirect from immediate parent only (VM's 'netvm' property) +2. external: redirect on the whole chain up to the top (follow 'netvm' +property until you get a VM with it set to None). + +My idea of redirect source was to explicitly point where the redirection +starts, instead of this automatic internal/external. But indeed the +automatic may be easier to use. + +
+3) Forwarding rules should be purged if <appvm> changes <firewall> +(maybe also if <firewallvm> changes <netvm>? But that would be harde to +detect I guess) +
+ +Or maybe it should be configurable? I'd hate to loose the configuration just +because I temporarily switched to another netvm (and then switched it +back)... + +Or, allow both automatic internal/external and explicit redirect source. +Then, setting redirect type to 'internal' would set the redirect point to +the VM's direct upstream (and would automatically adjust if you change +netvm), setting to 'external' would follow the whole chain. But setting +to explicit sys-net for example would always try to apply rule there, if +reachable (and add no rules, if not reachable). + +Does it make sense? + +Maybe let me explain it on a diagram (see attachment). Especially see +how redirections to appvm3 has changed after switching it from sys-vpn +to sys-firewall: the "internal" redirection (in green) remained there, +but the "sys-vpn" one (in blue) did not. + +
+4) Users should be able to specify both the forwarded port and +destination port as you were saying +5) Users should be able to eventually restrict forwarding to designated +networks (with 0.0.0.0/0 being the wildcard instead of being a wildcard +by default) +
+ +I wonder if this should be combined in the same rule, or maybe separate +rule for limiting? But maybe indeed putting it into the same rule may be +easier (avoids duplicating port numbers etc). + +
+However, in this case it will surely be harder to display the rules in +all the affected vms. +The other approach, as you were suggesting, of adding each specific rule +in each vm conf does make sense, but I think then it would necessary +something to keep track of the rule dependencies (such as a unique +identifier). Furthermore there is a higher risk of having orphaned rules +or a inconsistent state. + +Furthermore, in the "internal" vpn case that I have in mind, the idea is +to forward the local port via the VPN interface or Tor (but in the Tor +case users should just stick to Whonix). Some providers, such as +Mullvad, AirVPN, PIA etc allows port forwarding this way and I think +that's the most relevant case since it allows exposing a service on the +internet while maintaining a bit of privacy/anonimity and whithout +needing to bypass the local network NAT. Is this the same case you are +referring to? +
+ +Yes, exactly. + +
+And finally, don't forget IPv6 exists. Which means you can have the same +port on IPv4 and IPv6. And theoretically they could be redirected to +different places (but I'm not sure if that's a good idea...). + ++ +I think that once we have figured out the overall logic to implement, it +should not be hard to duplicate it for ipv4/ipv6. I think the main +problem to think about is to insert proper checks to prevent users from +adding mixed rules. +
+ +Yes, that sounds about right. + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab ++ +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 23/06/2021, 16:37 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
On Tue, Jun 22, 2021 at 02:28:26PM +0200, Giulio wrote: ++++3) Since the expire= feature seems to be already implemented (and ++
limited for the expiring full outgoing access) would it be useful to be +
implemented in gui and cli for every rule? I would say yes since the +
admin and agent code seems to be already there. The same goes for the +
"comment=" field. +
Per-rule expire may be tricky to handle at the GUI level, I have no idea +
how to make the UI for this not very confusing... +
But the comment field is definitely useful to use. +
+
How do you see the same checkbox that actually allows full internet +
access with the 5 minutes expiration time, displayed also on the window +
for adding a rule? +
This may be more relevant to longer times. With times like 5min, just +
setting the rules up (if you want more than one of them) may already eat +
up significant portion of the expiration time... +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 23/06/2021, 23:11 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Wed, Jun 23, 2021 at 04:37:20PM +0200, Giulio wrote: +
+Hello, +thank you again for your time and the explanations, as well as the +network graph. I have now a better understanding of the overall design +and I am moving myself trhough the source code in order to think what to +place where. + +So, in order to translate what we discussed in practice and also check +my understanding of the code so far: + +1) In core-admin-client/qubesadmin/firewall.py firewall.py > The code +needs to support the new options for the rule (action=forward +frowardtype=<internal/external> srcports=443-443 srchosts=0.0.0.0/0 +2) In core-admin/qubes/firewall.py -> The code needs to support the same +options as the point above +3) In core-admin/qubes/vm/mix/net.py -> The most important logic goes +here. Here there is the need to resolve the full network chain for +external port forwarding. From here it is possible to add the respective +rules to the QubesDB of each netvm in he chain and trigger a reload event. +4) in core-agent-linux/qubesagent/firewall.py -> Here goes the logic for +building the correct syntax for iptables or nft and the actual execution + +Does it makes sense for you? +
+ +Yes, I think you got this perfectly correct. + +
+I now totally understand your doubts, and I think the simplest solution +then would be a time/date picker, so if the user is planning anything +specific he can configure all the set of rules to the same expiration +timewithout incurring in the synchronization issues you mentioned. +
+ +Yes, that could work, with some easy way to use the same time for +multiple rules (for example default to the last choice). + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 28/06/2021, 22:46 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
On Wed, Jun 23, 2021 at 04:37:20PM +0200, Giulio wrote: ++Hello, ++
thank you again for your time and the explanations, as well as the +
network graph. I have now a better understanding of the overall design +
and I am moving myself trhough the source code in order to think what to +
place where. +
+
So, in order to translate what we discussed in practice and also check +
my understanding of the code so far: +
+
1) In core-admin-client/qubesadmin/firewall.py firewall.py > The code +
needs to support the new options for the rule (action=forward +
frowardtype=<internal/external> srcports=443-443 srchosts=0.0.0.0/0 +
2) In core-admin/qubes/firewall.py -> The code needs to support the same +
options as the point above +
3) In core-admin/qubes/vm/mix/net.py -> The most important logic goes +
here. Here there is the need to resolve the full network chain for +
external port forwarding. From here it is possible to add the respective +
rules to the QubesDB of each netvm in he chain and trigger a reload event. +
4) in core-agent-linux/qubesagent/firewall.py -> Here goes the logic for +
building the correct syntax for iptables or nft and the actual execution +
+
Does it makes sense for you? +
Yes, I think you got this perfectly correct. +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 29/06/2021, 03:31 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Mon, Jun 28, 2021 at 10:46:59PM +0200, Giulio wrote: +
+On 6/23/21 11:11 PM, Marek Marczykowski-Górecki wrote: ++On Wed, Jun 23, 2021 at 04:37:20PM +0200, Giulio wrote: ++Hello, +thank you again for your time and the explanations, as well as the +network graph. I have now a better understanding of the overall design +and I am moving myself trhough the source code in order to think what to +place where. + +So, in order to translate what we discussed in practice and also check +my understanding of the code so far: + +1) In core-admin-client/qubesadmin/firewall.py firewall.py > The code +needs to support the new options for the rule (action=forward +frowardtype=<internal/external> srcports=443-443 srchosts=0.0.0.0/0 +2) In core-admin/qubes/firewall.py -> The code needs to support the same +options as the point above +3) In core-admin/qubes/vm/mix/net.py -> The most important logic goes +here. Here there is the need to resolve the full network chain for +external port forwarding. From here it is possible to add the respective +rules to the QubesDB of each netvm in he chain and trigger a reload event. +4) in core-agent-linux/qubesagent/firewall.py -> Here goes the logic for +building the correct syntax for iptables or nft and the actual execution + +Does it makes sense for you? ++ +Yes, I think you got this perfectly correct. + ++ +I am at a good stage with 1 and 2. In 3, I am still thinking about some +design choices. I have written the function to resolve the network +'path', however I am trying to figure out which one is the most +appropriate way of inserting the forward rule(s) in each vm in the +chain. I feel like no parsing of the rules should be done in net.py +since it would be out of place and not fit well within the rest of the +code. Thus the rules should be provided to net.py already separated and +sorted in some way. My idea as of now is to add a 'qdb_forward_entries' +function, returning a dict of lists for 'internal' and 'external' rules +in firewall.py. It would be the trivial to process the information in +net.py. What do you think about that? +
+ +Yes, preparing rules in firewall.py sounds like a good idea. A new +function is a good idea too. But note that for 'external' rules you need +to apply them at several places (sys-net, sys-firewall etc). They aren't +necessarily will be the same. +I'd recommend getting an example, and writing down all the rules that +should be applied, in all related VMs (specific iptables/nft commands). +You have mostly done this part already. +This part you can also test manually - really add those rules +manually and check if everything works as it should. This way you ensure +the rule set is sufficient. + +Then, write down QubesDB entries that describe them - carefully matching +which information in the rule is built from which information in qdb +entry. +With that information, you know what qdb entries you need to produce for +each VM, and should be easier to design this extra function/functions - +especially, you'll see what input data such function needs and how many +different rules it needs to return. + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Auto: Re: GSoC Port Forwarding |
Mittente: |
Data: 13/07/2021, 15:56 |
A: |
+Hi, + +I'm out of office until July 25. I'll read your message after coming back. + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab + +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 13/07/2021, 15:56 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Yes, preparing rules in firewall.py sounds like a good idea. A new ++
function is a good idea too. But note that for 'external' rules you need +
to apply them at several places (sys-net, sys-firewall etc). They aren't +
necessarily will be the same. +
I'd recommend getting an example, and writing down all the rules that +
should be applied, in all related VMs (specific iptables/nft commands). +
You have mostly done this part already. +
This part you can also test manually - really add those rules +
manually and check if everything works as it should. This way you ensure +
the rule set is sufficient. +
+
Then, write down QubesDB entries that describe them - carefully matching +
which information in the rule is built from which information in qdb +
entry. +
With that information, you know what qdb entries you need to produce for +
each VM, and should be easier to design this extra function/functions - +
especially, you'll see what input data such function needs and how many +
different rules it needs to return. +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 14/07/2021, 12:16 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hi, +Thank you. +
+
I will have a look to your work probably in the afternoon. +
+
Just a question, any reason for hosting your work elsewhere than on +GitHub? For example, that would be easier for us to review your code +directly on GitHub (adding comments, tracking easily progression etc). ++
Also, I've briefly checked, a good practice is to not work on master +branch directly. I encourage you to use a separate branch named as you +want. ++
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 14/07/2021, 17:30 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 14/07/2021, 18:27 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Giulio, ++
+
Generally looks good. Do you have already some testing and working case? +If yes, can you please provide few steps here (that would be also good +for doc later). +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 14/07/2021, 12:10 |
A: Giulio <giulio@gmx.com>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hi, ++
+
Il 29/06/2021 03:31, Marek Marczykowski-Górecki ha scritto: +Yes, preparing rules in firewall.py sounds like a good idea. A new ++
function is a good idea too. But note that for 'external' rules you need +
to apply them at several places (sys-net, sys-firewall etc). They aren't +
necessarily will be the same. +
I'd recommend getting an example, and writing down all the rules that +
should be applied, in all related VMs (specific iptables/nft commands). +
You have mostly done this part already. +
This part you can also test manually - really add those rules +
manually and check if everything works as it should. This way you ensure +
the rule set is sufficient. +
+
Then, write down QubesDB entries that describe them - carefully matching +
which information in the rule is built from which information in qdb +
entry. +
With that information, you know what qdb entries you need to produce for +
each VM, and should be easier to design this extra function/functions - +
especially, you'll see what input data such function needs and how many +
different rules it needs to return. +
+
I tried writing a possible implementation to see how it could work and +
also to get an initial feedback. Since in the past week I had no access +
to my test machine, I just fixed the last things today and seems that +
overall the implemented parts are working (up to writing the rules with +
the correctly IPs in the appropriate agent databases). +
+
Here are the repositories https://git.lsd.cat/Qubes +
+
Here is a list of what has yet to be done: +
1) Lot of testing and writing tests +
2) Any modification to the agent (such as applying the rules) +
3) "srchost" parameter support +
4) GUI +
5) Find a way to display the chain of rules in the qvm-firewall of every +
VM involved since as of now it is displayed only in the VM for which the +
rule was set +
+
Here is a list of what should work: +
1) Adding and deleting forward rules, both internal and external, via +
qvm-firewall. Also basic checks of the consistency of rules and required +
options should be in place +
2) Display of forward rules via qvm-firewall +
3) Persistence and resume of forward rules in firewall.xml +
4) Correct distribution of the required rules in the network chain in net.py +
+
+
Overall I tried getting the most possible from already existing code in +
order not to change the style and introduce as few changes as possible. +
Without having you correct the code step by step, before going forward +
with the agent I would like to have a feedback if the coding style seems +
consistent enough with yours and especially if the implementation in +
net.py of the distributions of the rules matches your expectations. +
+
My changes are only in core-admin and core-admin-client for now. +
+
Cheers +
Giulio +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 14/07/2021, 14:57 |
A: Giulio <giulio@gmx.com>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hi, ++
+
Il 14/07/2021 12:10, Frédéric Pierret ha scritto: +Hi, +Thank you. +
+
I will have a look to your work probably in the afternoon. +
+
+Just a question, any reason for hosting your work elsewhere than on ++
GitHub? For example, that would be easier for us to review your code +
directly on GitHub (adding comments, tracking easily progression etc). +
The main reason is that I manage the server, and so I manage the backups +
and have control over everything because I like keeping the web +
decentralized somehow, but I understand that working on GitHub is +
easier. I already have an account so I can move there. +
+Also, I've briefly checked, a good practice is to not work on master ++
branch directly. I encourage you to use a separate branch named as you +
want. +
I will do a separate branch moving the code to GitHub. Will also merging +
all the commits in a single one improve readability for you? +
Cheers ++
Giulio +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 14/07/2021, 17:40 |
A: Giulio <giulio@gmx.com>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hello, ++
I have forked the latest repositories available on GitHub and I have +
manually diffed and inserted all my changes. I have added some more code +
that I have not tested yet in order to refactor a couple of things and +
add a basic 'srchost' support, but I think that overall the design +
choices already discussed with Marek should be clear. +
+
The code until now is in a single commit, in the "gsoc-port-forwarding" +
branch in the affected repositories: +
+
- +
https://github.com/lsd-cat/qubes-core-admin/commit/6f78a5fcf707140a6401c64a8ed9c8a47ce6e3a1 +
- +
https://github.com/lsd-cat/qubes-core-admin-client/commit/3b02e93cba17303598db0f0ed031793c6f79dec1 +
+
Thank you +
Cheers +
Giulio +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 17/07/2021, 17:31 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 17/07/2021, 21:52 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org> |
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
I've not an alternative idea yet but, I'm wondering if leaking appvm +names in "higher" untrusted appvms is reasonable, especially for +confidentiality. Maybe simply use the destination appvm ip, here in your +example that would be personal ip. dom0/GuiVM has access to the info so +getting appvm name from ip should be simple. ++
Here too, I'm not sure adding such info is a good idea for security. +What exactly do you have in mind for the last needs additional rules? ++
+One more thing, maybe between internal hops it makes sense to randomize ++
the forwarded ports? This way we can prevent forwarding from different +
appvm which shares the same network path or even just one hop from +
overlapping, at least internally. Does it makes sense for you? +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 17/07/2021, 21:07 |
A: Giulio |
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hi, ++
thank you for the positive feedback, I really appreciate it. +
I spent the past couple of days digging into the "rules distribution" +
mechanism in the various QubesDB. It really took some time to find a +
better way to handle and separate the rules for each domain at every +
'step' of their network path. +
+
https://github.com/lsd-cat/qubes-core-admin/commit/0cf04fb290469340a59a013531bba6e06e8a0169 +
+
The idea here is that each appvm will have a separate QubesDB folder on +
the untrusted qdb of each netvm. This is for easier cleaning and +
reloading, and can be used in the future when displaying the forwarding +
chain at each step like we were discussing with Marek in the first +
emails. A practical example follows. +
+
Assume the following network path: +
+
*internet/lan* <-> sys-net (10.137.0.5) <-> sys-firewall (10.137.0.6) +
<-> sys-vpn (10.137.0.13) <-> sys-tor (10.137.0.14) <-> personal +
(10.137.0.10) +
+
And the following rule: +
# qvm-firewall personal add action=forward forwardtype=external +
proto=tcp srcports=443-443 dstports=8443-8443 srchost=192.168.0.1/24 +
+
Here are the QubesDB entries that are automatically added, to be +
consumed from the core-agent-linux: +
+
In sys-net: +
key = /qubes-firewall-forward/personal/10.137.0.6/0000 +
value = action=forward forwardtype=external proto=tcp srcports=443-443 +
dstports=8443-8443 srchost=192.168.0.1/24 +
+
In sys-firewall: +
key = /qubes-firewall-forward/personal/10.137.0.13/0000 +
value = action=forward forwardtype=external proto=tcp srcports=443-443 +
dstports=8443-8443 srchost=192.168.0.1/24 +
+
In sys-vpn: +
key = /qubes-firewall-forward/personal/10.137.0.14/0000 +
value = action=forward forwardtype=external proto=tcp srcports=443-443 +
dstports=8443-8443 srchost=192.168.0.1/24 +
+
In sys-tor: +
key = /qubes-firewall-forward/personal/10.137.0.10/0000 +
value = action=forward forwardtype=external proto=tcp srcports=443-443 +
dstports=8443-8443 srchost=192.168.0.1/24 +
+
Although this mechanism seems complex, I was not able to think of a +
simpler solution. Furthermore I think it is important to know which +
appvm is requesting the forwarding at every step, both for debugging and +
auditing purposes. Lastly, the next hop ip address has to be determined +
automatically anyway and writtem somewhere so there it is. +
I am also thinking about adding a couple of flags to let the nodes know ++
which one is the first and which one is the last since especially the +
last needs additional rules for the external forwarding. +
One more thing, maybe between internal hops it makes sense to randomize ++
the forwarded ports? This way we can prevent forwarding from different +
appvm which shares the same network path or even just one hop from +
overlapping, at least internally. Does it makes sense for you? +
Cheers ++
Giulio +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 01/08/2021, 23:50 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org> |
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 05/08/2021, 23:31 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+Sorry for late response... + +On Sun, Aug 01, 2021 at 11:50:18PM +0200, Giulio wrote: +
+Hi, +I am still working on the implementation of the rules in the +core-agent-linux package. I have a couple of additional questions: + +1) Currently, I fail to understand and the inner workings the purpose of +the 'connected_ips' part. Could you give me an overall idea of it or any +useful additional details that you think may help me understand? +
+ +This is to inform what IPs belong to some VM, even powered off. This +way, firewall can prevent someone spoofing IP of a not running VM +(because it knows that IP cannot come from anywhere else). + +
+2) Since, as we talked in the previous emails, the last node needs an +additional rule in order to forward the port from the external interface +I am wondering how the correct interface is to be determined. I would +automatically choose the device on which there is the route with the +default gateway/destination. But, is it a good idea? Or would be better +to let the user choose? +
+ +This is a very good question. I think the most user-friendly thing to +do, is to include all the external interfaces (network manager will +add several default gateways, just with different priorities). Maybe +later it can be made configurable, but I wouldn't worry about it right +now. + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 14/08/2021, 18:33 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 14/08/2021, 23:43 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Sat, Aug 14, 2021 at 06:33:18PM +0200, Giulio wrote: +
+Hello, +Sorry for the late reply. +While everything cli related is almost ready, I am having some issues on +the actual implementation of the iptables/nft rules. I see that in the +current state, it seems like Qubes is using both as also stated in [1]. +
+ +Yes, there are two backends, depending on nft availability. This is +mostly because older distros (Debian 8...) did not have nft at all. + +
+However, in the core-agent-linux source code, if the 'nft' binary is +available that is the only one that gets invoked. Furthermore, there +are differences on the iptables backend depending on templates as +reported in [2]. +
+ +Yes, and since basically all the distributions have nft now, iptables +backend may be soon removed. I don't think we have any case where +iptables backend is used in practice in Qubes 4.1. RPM package has +strict dependency on nft, and Debian package has it as Suggests only, +but it is in practice installed. + +
+I am a bit stuck in understanding which rule to put where in order to +have consistency across templates and between iptables/nft, also because +if I blindly implement the rules suggested in [1] they will not actually +work since most of the time iptables is not invoked at all. +
+ +Indeed adding rules to the IptablesWorker class will make no effect if +nft is in use. +Theoretically, iptables rules and nft rules can coexist, but we should +really focus on nft with new features. + +
+I also checked [3], however it is very similar to the instructions in +[1] which leads to the same problems. +Are we able to write working nft forwarding rules without invoking +iptables at all? If yes, could you heml me determine which ones? +
+ +As for the nft syntax, I think iptables-translate tool can help you +(part of the iptables-nft package). +See https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_iptables_to_nftables + +
+You can see in [4] how I organized the forwarding mechanism. All the +necessary information, as well as ipv4/ipv6 support should already be in +the 'prepare_forward_rules' function meaning that only the actual +building syntax is left. +
+ +Yes, this layout looks ok. + +
+
+For simplicity you can look at the other changes at [5] and at [6].
++ +This too looks fine (although I haven't don't detailed review). + +In both cases, the code will need some tests of course. + +
+[1] - https://www.qubes-os.org/doc/firewall/ +[2] - https://github.com/QubesOS/qubes-issues/issues/5031 +[3] - https://gist.github.com/fepitre/941d7161ae1150d90e15f778027e3248 +[4] - +https://github.com/lsd-cat/qubes-core-agent-linux/compare/f24ca2c..3e944af +[5] - https://github.com/lsd-cat/qubes-core-admin/compare/6a1570b..6e3e250 +[6] - +https://github.com/lsd-cat/qubes-core-admin-client/compare/1a2ce72..a17853b +
+ +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 15/08/2021, 17:35 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
++
As for the nft syntax, I think iptables-translate tool can help you +
(part of the iptables-nft package). +
See https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_iptables_to_nftables +
+You can see in [4] how I organized the forwarding mechanism. All the ++
necessary information, as well as ipv4/ipv6 support should already be in +
the 'prepare_forward_rules' function meaning that only the actual +
building syntax is left. +
In both cases, the code will need some tests of course. ++
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 17/08/2021, 01:50 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Sun, Aug 15, 2021 at 05:35:37PM +0200, Giulio wrote: +
+Hello, +Thank you for you fast reply. + +Il 14/08/2021 23:43, Marek Marczykowski-Górecki ha scritto: ++ +As for the nft syntax, I think iptables-translate tool can help you +(part of the iptables-nft package). +See https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_iptables_to_nftables + ++You can see in [4] how I organized the forwarding mechanism. All the +necessary information, as well as ipv4/ipv6 support should already be in +the 'prepare_forward_rules' function meaning that only the actual +building syntax is left. ++ ++ +I have tried to write the external nft rules as well the extarnal ones, +with the exception of the destination domain. + +Assume the following setup: +sys-net - 10.137.0.5 (ens6 phy with 192.168.10.20) +sys-firewall - 10.137.0.6 +personal - 10.137.0.7 + +All of them are running fedora-32. + +And assume the following rule added via qvm-firewall: +# qvm-firewall personal add action=forward forwardtype=external +scrports=22-22 proto=tcp dstports=2222-2222 srchost=192.168.10.0/24 +. +First, a table for the forwarding rules is created: + +flush chain {family} qubes-firewall-forward prerouting +flush chain {family} qubes-firewall-forward postrouting +table {family} qubes-firewall-forward { + chain postrouting { + type nat hook postrouting priority srcnat; policy accept; + masquerade +
+ +I think this is too broad - this will hide the source address of all +incoming connections - something that shouldn't be needed. +masquerade is necessary for outgoing traffic only, but it's there +already in default setup (via iptables...) + +
+ } + chain prerouting { + type nat hook prerouting priority dstnat; policy accept; + } +} + +Then, if the qube is marked as 'last', meaning that it is the external +qube with the physical interface the following rules are added: + +table {family} qubes-firewall-forward { + chain prerouting { + meta iifname "ens6" {family} saddr 192.168.10.0/24 tcp dport {{ 22 }} +dnat to 10.137.0.6:2222 + } +} + +table {family} qubes-firewall { + chain forward { + meta iifname "eth0" {family} daddr 10.137.0.6 tcp dport 2222 ct state +new counter accept +
+ +iifname "eth0" ? Should be rather ens6. + +
+ } +} + +And that is all for sys-net. + +In sys-firewall, since it is an 'internal' qube, the following rules are +added instead: + +table {family} qubes-firewall-forward { + chain prerouting { + meta iifname "eth0" {family} saddr 120.137.0.5 tcp dport {{ 2222 }} +dnat to 10.137.0.7:2222 +
+ +And here, if there wouldn't be masquerade for everything, you could keep +the original source addr (192.168.10.0/24) + +
+ } +} + +table {family} qubes-firewall { + chain forward { + meta iifname "eth0" {family} daddr 10.137.0.7 tcp dport 2222 ct state +new counter accept + } +} + +Lastly, the appropriate rules allowing incoming traffic on the selected +port from the previous hop should be added directly yo the 'personal' +domain. However I see that there the nft ruleset is empty, while +iptables seems indeed to be in use. I guess that those rules are the +ones specified in qubes-core-agent-linux/network/iptables, however I am +wondering how we should proceed on this one? +
+ +Ok, this indeed is an issue with mixed iptables / nft usage. For the +purpose of this project, since there isn't much time left, you can +simply stop 'iptables' service in the 'personal' VM - and document +this as a manual step needed. This will become unnecessary when iptables +rules will be migrated to nft. + +But there is also another issue: the qubes-firewall daemon is currently +not started if a VM doesn't provide network. So, it isn't started in +'personal' VM here. + +
+Also are you able to spot errors or something missing in the +aforedescribed rule flow? When testing I can see the incoming connection +on port 22 of the physical interface of sys-net, but then I am losing +track of the connection after that... +
+ +See above - the interface name. You may also like to see this: +https://wiki.nftables.org/wiki-nftables/index.php/Ruleset_debug/tracing + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 20/08/2021, 03:20 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 21/08/2021, 00:08 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 22/08/2021, 00:30 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Sat, Aug 21, 2021 at 12:08:55AM +0200, Giulio wrote: +
+Hi, +as an addendum to the previous email, the problema was the fact that the +first rule to match in the qubes-firewall table, forward chain was: +iifname !="*vif" accept +By moving that to the end of the chain, the attached one is the new +trace which makes a lot more sense and increase the counters. +However, I still cannot see any traffic reaching the next hop. +
+ +Check if that isn't iptables blocking it. By default it does block new +connections coming from outside. I initially thought it would interfere +only at the final hop, but maybe at an earlier too... + + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 02/09/2021, 13:26 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 02/09/2021, 13:59 |
A: Giulio |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
+On Thu, Sep 02, 2021 at 01:26:34PM +0200, Giulio wrote: +
+Hello, +Thank you both for the support during the previous phase and your +patience. Thank you also for the positive evaluation, I really +appreciate it. +
+ +As said in the evaluation form - thanks for working on this, and +persistence in solving challenges! + +
+Although the project is not production ready, I would like to continue +working on it and eventually get it merged if it reaches a stable and +fully working condition. + +I am little bit offline now because I just moved to a new city to pursue +my master degree. However, I will continue to work on it again as soon +as I can. +
+ +When you have time, maybe open a (draft) pull requests to relevant +repositories? This would ease collecting feedback, I think. + +
+Since we are now out of the GSoC scope I would like to introduce the +project to the community so I can eventually get feedbacks and support, +especially for the network layer and rules. Is it ok if I open a GitHub +issue and maybe a discussion on the forum? +
+ +Yes, definitely! + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
Oggetto: Re: GSoC Port Forwarding |
Mittente: Giulio |
Data: 19/10/2021, 14:48 |
A: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
CC: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Oggetto: Re: GSoC Port Forwarding |
Mittente: Frédéric Pierret <frederic.pierret@qubes-os.org> |
Data: 19/10/2021, 14:59 |
A: Giulio <giulio@gmx.com>, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Hello, ++
+
Il 02/09/2021 13:59, Marek Marczykowski-Górecki ha scritto: +
> On Thu, Sep 02, 2021 at 01:26:34PM +0200, Giulio wrote: +
> +
> When you have time, maybe open a (draft) pull requests to relevant +
> repositories? This would ease collecting feedback, I think. +
> +
>> Since we are now out of the GSoC scope I would like to introduce the +
>> project to the community so I can eventually get feedbacks and support, +
>> especially for the network layer and rules. Is it ok if I open a GitHub +
>> issue and maybe a discussion on the forum? +
> +
> Yes, definitely! +
> +
+
I am now drafting the pull requests and the message to send to the +
community. May I publish our correspondance? I feel like it could be a +
useful addendum to the documentation and help people understand the +
design choices and the development process. Otherwise if you prefer to +
keep them private no problem at all :) +
+
Cheers +
Giulio +
+
Oggetto: Re: GSoC Port Forwarding |
Mittente: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> |
Data: 19/10/2021, 15:10 |
A: Frédéric Pierret <frederic.pierret@qubes-os.org> |
CC: Giulio <giulio@gmx.com> |
+On Tue, Oct 19, 2021 at 02:59:45PM +0200, Frédéric Pierret wrote: +
+Hello, + +Le 10/19/21 à 14:48, Giulio a écrit : ++Hello, + +Il 02/09/2021 13:59, Marek Marczykowski-Górecki ha scritto: + > On Thu, Sep 02, 2021 at 01:26:34PM +0200, Giulio wrote: + > + > When you have time, maybe open a (draft) pull requests to relevant + > repositories? This would ease collecting feedback, I think. + > + >> Since we are now out of the GSoC scope I would like to introduce the + >> project to the community so I can eventually get feedbacks and support, + >> especially for the network layer and rules. Is it ok if I open a GitHub + >> issue and maybe a discussion on the forum? + > + > Yes, definitely! + > + +I am now drafting the pull requests and the message to send to the +community. May I publish our correspondance? I feel like it could be a +useful addendum to the documentation and help people understand the +design choices and the development process. Otherwise if you prefer to +keep them private no problem at all :) + +Cheers +Giulio + ++ +Thank you for that. I'm ok with publishing private correspondence. +
+ +Fine with me too :) + +-- +Best Regards, +Marek Marczykowski-Górecki +Invisible Things Lab +
| Oggetto | Mittente | A | Data | Allegato |
|---|---|---|---|---|
| GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +6/11/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com>, Marek Marczykowski-Górec | +6/11/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +6/20/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +6/22/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +6/22/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +6/22/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +6/22/2021 | +* |
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +6/23/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +6/23/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +6/28/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +6/29/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +7/13/2021 | +|
| Auto: Re: GSoC Port Forwarding | +<marmarek@invisiblethingslab.com> | +<giulio@gmx.com> | +7/13/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com>, Marek Marczykowski-Górec | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com>, Marek Marczykowski-Górec | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com>, Marek Marczykowski-Górec | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +7/14/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org>, | +7/17/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com> | +7/17/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +7/17/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +8/01/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +8/05/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +8/14/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +8/14/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +8/15/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +8/17/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +8/20/2021 | +* |
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +8/21/2021 | +* |
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +8/22/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +9/02/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Giulio <giulio@gmx.com> | +9/02/2021 | +|
| Re: GSoC Port Forwarding | +Giulio <giulio@gmx.com> | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +10/19/2021 | +|
| Re: GSoC Port Forwarding | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +Giulio <giulio@gmx.com>, Marek Marczykowski-Górec | +10/19/2021 | +|
| Re: GSoC Port Forwarding | +Marek Marczykowski-Górecki <marmarek@invisiblethi | +Frédéric Pierret <frederic.pierret@qubes-os.org> | +10/19/2021 | +
