On Tue, Jun 22, 2021 at 02:28:26PM +0200, Giulio wrote:

> Hello,
> thank you for the detailed response.
> 
> Il 22/06/2021 04:43, Marek Marczykowski-Górecki ha scritto:

> > Hi, I'm replying to both emails at once:
> > 
> > On Sun, Jun 20, 2021 at 10:50:04PM +0200, Giulio wrote:

> > > Questions:
> > > 
> > > 1) Should we both support internal port forwarding and external port
> > > forwarding? Such as exposing a port for another domain or exposing a
> > > port through the public network interface? I would say yes.

> > 
> > Yes, I think so. Technically, those two cases should be quite similar.
> > See also the case of sys-vpn much lower in the email.
> > 

> 
> I think that I'm actually failing to picture all the possible internal
> scenarios.
> 
> 1) In the case of external port forwarding <sys-net> should forward to
> <sys-firewall> and <sys-firewall> then to the <appvm>.
> In this case the port gets forwarded on the external interface ie: a LAN
> or a public ip address depending on the network environment.

Yes.

> 2) In the case of internal port forwarding, the port is forwarded only
> from <sys-firewall> to <appvm>. In that case, another <appvm2> can visit
> the <appvm> service using <sys-firewall> ip address and the chosen port.

Yes.

> In this case, the ports get exposed on <sys-firewall> and thus depending
> on how the rules are implemented, may be available to all the AppVMs
> that share the same <sys-firewall>.

Yes.

> In both cases may be important to allow to specify access rules for the
> forwarded port, such as the lan/public ip addresses ranges allowed for
> case 1 and the appvm name for case 2.

Yes, indeed.

> > > 3) Since the expire= feature seems to be already implemented (and
> > > limited for the expiring full outgoing access) would it be useful to be
> > > implemented in gui and cli for every rule? I would say yes since the
> > > admin and agent code seems to be already there. The same goes for the
> > > "comment=" field.

> > 
> > Per-rule expire may be tricky to handle at the GUI level, I have no idea
> > how to make the UI for this not very confusing...
> > But the comment field is definitely useful to use.
> > 

> 
> How do you see the same checkbox that actually allows full internet
> access with the 5 minutes expiration time, displayed also on the window
> for adding a rule?

This may be more relevant to longer times. With times like 5min, just
setting the rules up (if you want more than one of them) may already eat
up significant portion of the expiration time...

> However I think there is time to think more through this as the UI will
> be the last component.
> 

> > > 4) How would you implement the management of forwarding rules in the
> > > network providing domain (sys-net)? Shall the user add a rule both in
> > > the target domain (ie the one with webserver and another one in sys-net)
> > > or should it be fully automatic from the first?

> > 
> > From the user point of view, I think it should be automated as much as
> > possible. Like, let the user choose which port in which VM redirect to
> > where. There may be cases when such redirection won't be possible - if
> > there is no network path between the two points.
> > 

> 
> I agree with you. We might just check when the user adds an internal
> forwarding rule if both the source and the destination shares the same
> <firewallvm>, don't we?

I think we can simplify it even further: allow forwarding ports only
from (any of) upstream VMs. For example in this case:

    appvm1 -> sys-firewall -> sys-net
                 |                 |
    appvm2 ------+                 |
                                   |
    appvm3 -> some-other-firewall -+

Allow forwarding to appvm1 only from sys-net (external case) or
sys-firwall, but not appvm3 or some-other-firewall.
Then, within the forward rule configuration you can restrict access
rules (like you propose below, with default 0.0.0.0/0). This restriction
will work for VMs directly connected to sys-firewall only, because there
is NAT (sys-net does not know whether its appvm1 or appvm2 - it only
sees sys-firewall IP in those cases). But I think it's ok to make this
limitation and require VMs to be connected to the same <firewallvm> if
you want to forward traffic between them.

I think you did it right with the internal/external type.

Allowing forwarding from others (like some-other-firewall in the picture
above) may be tricky (and unreliable), as it will be hard to restrict
who can really connect (sys-net have no idea which VM behind
sys-firewall/some-other-firewall really connects).

> > > 5) Users should be able to set forward rules using domain names and not
> > > static ip addresses. In this case, the actual ip addresses of the dst
> > > domains should be collected in a similr way as currently DNS are
> > > resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good?

> > 
> > But here we are mostly talking about IP addresses of different VMs,
> > right? Those can (and should) be resolved at core-admin side, so the VM
> > applying the rules will have all the IP given. In fact VM may not be
> > able to resolve IP of another VM at all.
> > 

> 
> Thanks for the insight, it totally makes sense.
> 

> > > Proposed XML Syntax:
> > > <rule>
> > > 	<properties>
> > > 		<property name="action">forward</property>
> > > 		<property name="proto">udp</property>
> > > 		<property name="dstports">443-8080-5555</property>
> > > 	</properties>
> > > <rule>

> > 
> > I don't see an important information here: forward to _where_.
> > 

> > > Proposed Admin API Syntax:
> > > action=forward proto=udp dstports=443-8080-5555 [expire=<unix
> > > timestamp>] [comment=random text]

> > 
> > Similar here, there needs to be a forward target (IP, and possibly a
> > port)
> > 
> > On Tue, Jun 22, 2021 at 01:49:15AM +0200, Giulio wrote:

> > > Since in the case of port forwarding the target ip address would always be the <vmname> IP address,

> > 
> > This is very true. But there needs to be an information where to forward
> > the traffic to (as noted earlier). Plus, possibly a second set of ports
> > (if you want to redirect to a different port).
> > 

> 
> I am still failing to understand something here, could you give me a an
> example on when the dsthosts would different rather than the <appvm> or
> <firewallvm> ip?

I'm talking about the other end of the connection. Let me summarize
this for different rules:

1. For allow/deny rules, it is about where the VM can connect to -
outgoing connection. The source IP is always the VM's IP (implicitly),
and the rule can specify destination IP, protocol, port.

2. For forward rules, it is about incoming connection - the destination
IP is VM's IP (implicitly), but then the connection is redirected to
somewhere else (like some appvm) - and the rule needs to point out to
where it should redirect.

If you have simplified case like this:

  sys-net -> appvm

Then, the rule in sys-net not only needs to know what to redirect (like,
TCP port 80), but also needs to know to redirect it to appvm, not
anywhere else.
Similarly, if the rule is stored with appvm, it needs to know to
install the redirect in sys-net and not some intermediate other VM (as
talked about internal, or VPN or Tor cases).

Ok, reading your response further, I think you covered it with
external/internal type, so it should be good.

> > > I think my main concern now is the question 4 from the aforementioned
> > > email. Shall the rules be automatically implemented in all 3 involved
> > > vms? (<netvm,firewallvm,appvm>). I think yes, because otherwise it would
> > > be counterintuitive to be a partially manual and partially automatic
> > > operation. But since it actually 'automatically' exposes more attack
> > > surface, by loosening up all 3 vms network rules, I guess that maybe
> > > more reasoning on it would be helpful.

> > 
> > Yes, but you need to pass the traffic somehow. The direct connection can
> > be achieved with qvm-connnect-tcp (connecting to the target directly
> > using qrexec, bypassing intermediate VMs), but it has its limits (low
> > performance, TCP only). To keep it as actual IP traffic, you need to
> > change firewall rules at all intermediate VMs too.
> > 
> > Lets have a specific example: in default setup, redirect TCP port 80
> > from the outside, to 'work' VM port 8080.
> > 
> > The setup looks like this:
> > 
> >    sys-net -> sys-firewall -> work
> > 
> > For this, you will need those rules:
> > 
> > 1. In sys-net: forward TCP port 80 to sys-firewall
> > 2. In sys-firewall: forward TCP port 80 to work, port 8080
> > 3. In work: allow TCP port 8080
> > 
> > Now is the important design question: how to store those rules? If you
> > store them at all three places separately, it
> > will be easier to apply them at runtime, but it will be harder to
> > correlate them in UI. Plus, if any of them get modified/removed, it may
> > be non-trivial to troubleshoot the issue.
> > The other approach is to store the forward rules only in one place (the
> > target, 'work' in this example? or the source, 'sys-net' here?). This
> > way, it's harder to mess thing up. But when applying the rules (building
> > rule sets for qubes-firewall service in all the involved VMs), you need
> > to check several places.
> > Plus, the UI should clearly show such redirected ports at all involved
> > places, because it does affect system security - it must be easy to spot
> > if any redirects are enabled.
> > 
> > 
> > To make things more complex (sorry...), there may be a VPN or other
> > proxy service (Tor?) involved. For example:
> > 
> > sys-net -> sys-firewall -> sys-vpn -> work
> > 
> > In such a case, the "external" VM for 'work' is not really sys-net, but
> > rather sys-vpn. And actually you need to be careful to not accidentally
> > bypass VPN either by allowing 'work' to communicate outside of the VPN,
> > or (maybe even worse) systems on the LAN (via sys-net) reach inside VPN.
> > 
> > This case is not easy to solve, because currently core-admin has no idea
> > whether sys-vpn (or other such VM) do any of such tunnelling. Maybe we
> > need to (finally) introduce some flag to mark such VMs?
> > 
> > 
> > And another question: what should happen if you change netvm of 'work'.
> > For example switch to something like:
> > 
> >      sys-net -> sys-firewall -> (other VMs, but not 'work')
> > 
> >      sys-wifi -> work
> > 
> > Should the redirection stay active via sys-wifi? I think it should not,
> > at least not automatically (maybe have an option for that?).
> > 

> 
> I understand all of your points and consequently it is hard to figure
> out a catch-all solution.
> 
> I tried charting the flow of the possible solution.
> https://git.lsd.cat/Qubes/gsoc/src/master/assets/implementation.png
> 
> As a sum up:
> 1) Rules are stored only in <appvm>/firewall.xml
> 2) Rules can either be internal or exteranl (ie: they are applied only
> to <firewallvm> or both to <firewallvm> and its <netvm>)

Please note there may be more VMs in between, some examples at:
https://www.qubes-os.org/doc/firewall/

So, I think it's better to define it as:
1. internal: redirect from immediate parent only (VM's 'netvm' property)
2. external: redirect on the whole chain up to the top (follow 'netvm'
property until you get a VM with it set to None).

My idea of redirect source was to explicitly point where the redirection
starts, instead of this automatic internal/external. But indeed the
automatic may be easier to use.

> 3) Forwarding rules should be purged if <appvm> changes <firewall>
> (maybe also if <firewallvm> changes <netvm>? But that would be harde to
> detect I guess)

Or maybe it should be configurable? I'd hate to loose the configuration just
because I temporarily switched to another netvm (and then switched it
back)...

Or, allow both automatic internal/external and explicit redirect source.
Then, setting redirect type to 'internal' would set the redirect point to
the VM's direct upstream (and would automatically adjust if you change
netvm), setting to 'external' would follow the whole chain. But setting
to explicit sys-net for example would always try to apply rule there, if
reachable (and add no rules, if not reachable).

Does it make sense?

Maybe let me explain it on a diagram (see attachment). Especially see
how redirections to appvm3 has changed after switching it from sys-vpn
to sys-firewall: the "internal" redirection (in green) remained there,
but the "sys-vpn" one (in blue) did not.

> 4) Users should be able to specify both the forwarded port and
> destination port as you were saying
> 5) Users should be able to eventually restrict forwarding to designated
> networks (with 0.0.0.0/0 being the wildcard instead of being a wildcard
> by default)

I wonder if this should be combined in the same rule, or maybe separate
rule for limiting? But maybe indeed putting it into the same rule may be
easier (avoids duplicating port numbers etc).

> However, in this case it will surely be harder to display the rules in
> all the affected vms.
> The other approach, as you were suggesting, of adding each specific rule
> in each vm conf does make sense, but I think then it would necessary
> something to keep track of the rule dependencies (such as a unique
> identifier). Furthermore there is a higher risk of having orphaned rules
> or a inconsistent state.
> 
> Furthermore, in the "internal" vpn case that I have in mind, the idea is
> to forward the local port via the VPN interface or Tor (but in the Tor
> case users should just stick to Whonix). Some providers, such as
> Mullvad, AirVPN, PIA etc allows port forwarding this way and I think
> that's the most relevant case since it allows exposing a service on the
> internet while maintaining a bit of privacy/anonimity and whithout
> needing to bypass the local network NAT. Is this the same case you are
> referring to?

Yes, exactly.

> > And finally, don't forget IPv6 exists. Which means you can have the same
> > port on IPv4 and IPv6. And theoretically they could be redirected to
> > different places (but I'm not sure if that's a good idea...).
> > 

> 
> I think that once we have figured out the overall logic to implement, it
> should not be hard to duplicate it for ipv4/ipv6. I think the main
> problem to think about is to insert proper checks to prevent users from
> adding mixed rules.

Yes, that sounds about right.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab