20210622-Re_GSoC Port Forwarding-13692.html 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253
  1. <html>
  2. <head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  3. <title>Re: GSoC Port Forwarding</title>
  4. <link rel="important stylesheet" href="">
  5. <style>div.headerdisplayname {font-weight:bold;}
  6. </style></head>
  7. <body>
  8. <table border=0 cellspacing=0 cellpadding=0 width="100%" class="header-part1"><tr><td><div class="headerdisplayname" style="display:inline;">Oggetto: </div>Re: GSoC Port Forwarding</td></tr><tr><td><div class="headerdisplayname" style="display:inline;">Mittente: </div>Marek Marczykowski-Górecki &lt;marmarek@invisiblethingslab.com&gt;</td></tr><tr><td><div class="headerdisplayname" style="display:inline;">Data: </div>22/06/2021, 04:43</td></tr></table><table border=0 cellspacing=0 cellpadding=0 width="100%" class="header-part2"><tr><td><div class="headerdisplayname" style="display:inline;">A: </div>Giulio <giulio@gmx.com></td></tr><tr><td><div class="headerdisplayname" style="display:inline;">CC: </div>Frédéric Pierret &lt;frederic.pierret@qubes-os.org&gt;</td></tr></table><br>
  9. <div class="moz-text-plain" wrap=true graphical-quote=true style="font-family: -moz-fixed; font-size: 14px;" lang="x-unicode"><pre wrap class="moz-quote-pre">
  10. Hi, I'm replying to both emails at once:
  11. On Sun, Jun 20, 2021 at 10:50:04PM +0200, Giulio wrote:
  12. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  13. <span class="moz-txt-citetags">&gt; </span>Hello,
  14. <span class="moz-txt-citetags">&gt; </span>sorry for the late reply.
  15. <span class="moz-txt-citetags">&gt; </span>
  16. <span class="moz-txt-citetags">&gt; </span>I read a lot of code and I have to admit that I did not grasp the
  17. <span class="moz-txt-citetags">&gt; </span>complexity of the Admin API and networking stack before looking so much
  18. <span class="moz-txt-citetags">&gt; </span>into it. I think I've got the overall picture, but it will take a little
  19. <span class="moz-txt-citetags">&gt; </span>more to fully be confident moving there.
  20. </pre></blockquote><pre wrap class="moz-quote-pre">
  21. (...)
  22. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  23. <span class="moz-txt-citetags">&gt; </span><span class="moz-txt-underscore"><span class="moz-txt-tag">_</span>core-agent-linux/qubesagent/firewall.py<span class="moz-txt-tag">_</span></span>
  24. <span class="moz-txt-citetags">&gt; </span>Is the actual file responsible for running nftables and thus
  25. <span class="moz-txt-citetags">&gt; </span>adding/deleting/reloading firewall ruless in the target firewall vm. It
  26. <span class="moz-txt-citetags">&gt; </span>also resolves DNS names for domain rules. It is run by Admin API (qubesd).
  27. </pre></blockquote><pre wrap class="moz-quote-pre">
  28. This file is run as part of the qubes-firewall service in relevant VMs
  29. (especially sys-firewall).
  30. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  31. <span class="moz-txt-citetags">&gt; </span><span class="moz-txt-underscore"><span class="moz-txt-tag">_</span>manager/qubesmanager/firewall.py<span class="moz-txt-tag">_</span></span>
  32. <span class="moz-txt-citetags">&gt; </span>Contains the code for the "Firewall" tab of the "Qube Manager" window.
  33. <span class="moz-txt-citetags">&gt; </span>
  34. <span class="moz-txt-citetags">&gt; </span><span class="moz-txt-underscore"><span class="moz-txt-tag">_</span>manager/ui/qubemanager.ui<span class="moz-txt-tag">_</span></span>
  35. <span class="moz-txt-citetags">&gt; </span>String and properties for the "Qube Manager" UI.
  36. <span class="moz-txt-citetags">&gt; </span>
  37. <span class="moz-txt-citetags">&gt; </span>
  38. <span class="moz-txt-citetags">&gt; </span>Questions:
  39. <span class="moz-txt-citetags">&gt; </span>
  40. <span class="moz-txt-citetags">&gt; </span>1) Should we both support internal port forwarding and external port
  41. <span class="moz-txt-citetags">&gt; </span>forwarding? Such as exposing a port for another domain or exposing a
  42. <span class="moz-txt-citetags">&gt; </span>port through the public network interface? I would say yes.
  43. </pre></blockquote><pre wrap class="moz-quote-pre">
  44. Yes, I think so. Technically, those two cases should be quite similar.
  45. See also the case of sys-vpn much lower in the email.
  46. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  47. <span class="moz-txt-citetags">&gt; </span>2) Should it be possible to add rules with an 'any' clause (both tcp and
  48. <span class="moz-txt-citetags">&gt; </span>udp). I would say no because since port forwarding brings a higher
  49. <span class="moz-txt-citetags">&gt; </span>attack surface all rules should be as precise as possible.
  50. </pre></blockquote><pre wrap class="moz-quote-pre">
  51. Indeed. Furthermore, most services are either TCP or UDP, very few are
  52. both (and for those, it's ok to require two rules).
  53. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  54. <span class="moz-txt-citetags">&gt; </span>3) Since the expire= feature seems to be already implemented (and
  55. <span class="moz-txt-citetags">&gt; </span>limited for the expiring full outgoing access) would it be useful to be
  56. <span class="moz-txt-citetags">&gt; </span>implemented in gui and cli for every rule? I would say yes since the
  57. <span class="moz-txt-citetags">&gt; </span>admin and agent code seems to be already there. The same goes for the
  58. <span class="moz-txt-citetags">&gt; </span>"comment=" field.
  59. </pre></blockquote><pre wrap class="moz-quote-pre">
  60. Per-rule expire may be tricky to handle at the GUI level, I have no idea
  61. how to make the UI for this not very confusing...
  62. But the comment field is definitely useful to use.
  63. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  64. <span class="moz-txt-citetags">&gt; </span>4) How would you implement the management of forwarding rules in the
  65. <span class="moz-txt-citetags">&gt; </span>network providing domain (sys-net)? Shall the user add a rule both in
  66. <span class="moz-txt-citetags">&gt; </span>the target domain (ie the one with webserver and another one in sys-net)
  67. <span class="moz-txt-citetags">&gt; </span>or should it be fully automatic from the first?
  68. </pre></blockquote><pre wrap class="moz-quote-pre">
  69. From the user point of view, I think it should be automated as much as
  70. possible. Like, let the user choose which port in which VM redirect to
  71. where. There may be cases when such redirection won't be possible - if
  72. there is no network path between the two points.
  73. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  74. <span class="moz-txt-citetags">&gt; </span>5) Users should be able to set forward rules using domain names and not
  75. <span class="moz-txt-citetags">&gt; </span>static ip addresses. In this case, the actual ip addresses of the dst
  76. <span class="moz-txt-citetags">&gt; </span>domains should be collected in a similr way as currently DNS are
  77. <span class="moz-txt-citetags">&gt; </span>resolved in `/core-agent-linux/qubesagent/firewall.py`, would this be good?
  78. </pre></blockquote><pre wrap class="moz-quote-pre">
  79. But here we are mostly talking about IP addresses of different VMs,
  80. right? Those can (and should) be resolved at core-admin side, so the VM
  81. applying the rules will have all the IP given. In fact VM may not be
  82. able to resolve IP of another VM at all.
  83. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  84. <span class="moz-txt-citetags">&gt; </span>Proposed XML Syntax:
  85. <span class="moz-txt-citetags">&gt; </span>&lt;rule&gt;
  86. <span class="moz-txt-citetags">&gt; </span> &lt;properties&gt;
  87. <span class="moz-txt-citetags">&gt; </span> &lt;property name="action"&gt;forward&lt;/property&gt;
  88. <span class="moz-txt-citetags">&gt; </span> &lt;property name="proto"&gt;udp&lt;/property&gt;
  89. <span class="moz-txt-citetags">&gt; </span> &lt;property name="dstports"&gt;443-8080-5555&lt;/property&gt;
  90. <span class="moz-txt-citetags">&gt; </span> &lt;/properties&gt;
  91. <span class="moz-txt-citetags">&gt; </span>&lt;rule&gt;
  92. </pre></blockquote><pre wrap class="moz-quote-pre">
  93. I don't see an important information here: forward to <span class="moz-txt-underscore"><span class="moz-txt-tag">_</span>where<span class="moz-txt-tag">_</span></span>.
  94. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  95. <span class="moz-txt-citetags">&gt; </span>Proposed Admin API Syntax:
  96. <span class="moz-txt-citetags">&gt; </span>action=forward proto=udp dstports=443-8080-5555 [expire=&lt;unix
  97. <span class="moz-txt-citetags">&gt; </span>timestamp&gt;] [comment=random text]
  98. </pre></blockquote><pre wrap class="moz-quote-pre">
  99. Similar here, there needs to be a forward target (IP, and possibly a
  100. port)
  101. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  102. <span class="moz-txt-citetags">&gt; </span>
  103. <span class="moz-txt-citetags">&gt; </span>I also plan to document, at least partially, the journey into this.
  104. <span class="moz-txt-citetags">&gt; </span>As a last question, I'm curious what is your setup in order to test
  105. <span class="moz-txt-citetags">&gt; </span>modifications in the aforementioned repos while developing.
  106. </pre></blockquote><pre wrap class="moz-quote-pre">
  107. In practice, I have a separate (physical) computer to run various tests.
  108. It can be a system installed on an external disk. But there are also
  109. other options:
  110. 1. Test on the system you are developing on - this may be the easiest
  111. option, but is also a risky one (it is easy to break things - may be
  112. undesirable if you need that system for other stuff too).
  113. 2. Test in a virtual machine - it's possible to run Qubes OS inside KVM
  114. with nested virtualization enabled (included emulated IOMMU - sadly
  115. works only Intel only). This requires non-Qubes host. While it is
  116. possible to run Qubes inside Qubes, the setup is quite challenging and
  117. fragile, so I don't recommend this path.
  118. 3. As a last resort, I have a bunch test systems and I can give you
  119. access to one of them via SSH. But developing network-related changes,
  120. accessing the system through network, may not be the wisest thing to
  121. do...
  122. You can also find some helpful info here:
  123. <a class="moz-txt-link-freetext" href="https://www.qubes-os.org/doc/#debugging">https://www.qubes-os.org/doc/#debugging</a>
  124. On Tue, Jun 22, 2021 at 01:49:15AM +0200, Giulio wrote:
  125. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  126. <span class="moz-txt-citetags">&gt; </span>Hi,
  127. <span class="moz-txt-citetags">&gt; </span>sorry for yesterday long and a bit confusing message. I started writing
  128. <span class="moz-txt-citetags">&gt; </span>down my documentation and progress here <a class="moz-txt-link-freetext" href="https://git.lsd.cat/Qubes/gsoc">https://git.lsd.cat/Qubes/gsoc</a>
  129. <span class="moz-txt-citetags">&gt; </span>so it should me more readable and easier to follow.
  130. </pre></blockquote><pre wrap class="moz-quote-pre">
  131. No worries. Some comments for this page:
  132. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  133. <span class="moz-txt-citetags">&gt; </span>qvm-firewall &lt;vmname&gt; add action=accept dsthost=1.1.1.1 proto=tcp dstports=80-80 command="cloudflare http test rule" expire=+5000
  134. </pre></blockquote><pre wrap class="moz-quote-pre">
  135. It's "comment", not "command". And also, if you ever need to write Admin
  136. API level line manually, the comment field must be last.
  137. (but on qvm-firewall cmdline it is fine as is)
  138. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  139. <span class="moz-txt-citetags">&gt; </span>Since in the case of port forwarding the target ip address would always be the &lt;vmname&gt; IP address,
  140. </pre></blockquote><pre wrap class="moz-quote-pre">
  141. This is very true. But there needs to be an information where to forward
  142. the traffic to (as noted earlier). Plus, possibly a second set of ports
  143. (if you want to redirect to a different port).
  144. </pre><blockquote type=cite style="color: #007cff;"><pre wrap class="moz-quote-pre">
  145. <span class="moz-txt-citetags">&gt; </span>I think my main concern now is the question 4 from the aforementioned
  146. <span class="moz-txt-citetags">&gt; </span>email. Shall the rules be automatically implemented in all 3 involved
  147. <span class="moz-txt-citetags">&gt; </span>vms? (&lt;netvm,firewallvm,appvm&gt;). I think yes, because otherwise it would
  148. <span class="moz-txt-citetags">&gt; </span>be counterintuitive to be a partially manual and partially automatic
  149. <span class="moz-txt-citetags">&gt; </span>operation. But since it actually 'automatically' exposes more attack
  150. <span class="moz-txt-citetags">&gt; </span>surface, by loosening up all 3 vms network rules, I guess that maybe
  151. <span class="moz-txt-citetags">&gt; </span>more reasoning on it would be helpful.
  152. </pre></blockquote><pre wrap class="moz-quote-pre">
  153. Yes, but you need to pass the traffic somehow. The direct connection can
  154. be achieved with qvm-connnect-tcp (connecting to the target directly
  155. using qrexec, bypassing intermediate VMs), but it has its limits (low
  156. performance, TCP only). To keep it as actual IP traffic, you need to
  157. change firewall rules at all intermediate VMs too.
  158. Lets have a specific example: in default setup, redirect TCP port 80
  159. from the outside, to 'work' VM port 8080.
  160. The setup looks like this:
  161. sys-net -&gt; sys-firewall -&gt; work
  162. For this, you will need those rules:
  163. 1. In sys-net: forward TCP port 80 to sys-firewall
  164. 2. In sys-firewall: forward TCP port 80 to work, port 8080
  165. 3. In work: allow TCP port 8080
  166. Now is the important design question: how to store those rules? If you
  167. store them at all three places separately, it
  168. will be easier to apply them at runtime, but it will be harder to
  169. correlate them in UI. Plus, if any of them get modified/removed, it may
  170. be non-trivial to troubleshoot the issue.
  171. The other approach is to store the forward rules only in one place (the
  172. target, 'work' in this example? or the source, 'sys-net' here?). This
  173. way, it's harder to mess thing up. But when applying the rules (building
  174. rule sets for qubes-firewall service in all the involved VMs), you need
  175. to check several places.
  176. Plus, the UI should clearly show such redirected ports at all involved
  177. places, because it does affect system security - it must be easy to spot
  178. if any redirects are enabled.
  179. To make things more complex (sorry...), there may be a VPN or other
  180. proxy service (Tor?) involved. For example:
  181. sys-net -&gt; sys-firewall -&gt; sys-vpn -&gt; work
  182. In such a case, the "external" VM for 'work' is not really sys-net, but
  183. rather sys-vpn. And actually you need to be careful to not accidentally
  184. bypass VPN either by allowing 'work' to communicate outside of the VPN,
  185. or (maybe even worse) systems on the LAN (via sys-net) reach inside VPN.
  186. This case is not easy to solve, because currently core-admin has no idea
  187. whether sys-vpn (or other such VM) do any of such tunnelling. Maybe we
  188. need to (finally) introduce some flag to mark such VMs?
  189. And another question: what should happen if you change netvm of 'work'.
  190. For example switch to something like:
  191. sys-net -&gt; sys-firewall -&gt; (other VMs, but not 'work')
  192. sys-wifi -&gt; work
  193. Should the redirection stay active via sys-wifi? I think it should not,
  194. at least not automatically (maybe have an option for that?).
  195. And finally, don't forget IPv6 exists. Which means you can have the same
  196. port on IPv4 and IPv6. And theoretically they could be redirected to
  197. different places (but I'm not sure if that's a good idea...).
  198. <div class="moz-txt-sig">--
  199. Best Regards,
  200. Marek Marczykowski-Górecki
  201. Invisible Things Lab
  202. </div></pre></div></body>
  203. </html>
  204. </table></div>