Create Thunderbolt_Networking.md

Thunderbolt_Networking.md

@@ -0,0 +1,103 @@
+# Thunderbolt Networking
+
+[this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
+
+#### NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve  OR HIGER
+This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol) 
+
+## Install LLDP - this is great to see what nodes can see which.
+-  install lldpctl with `apt install lldpd`
+
+## Load Kernel Modules
+- add `thunderbolt` and `thunderbolt-net` kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
+    1. `nano /etc/modules` add modules at bottom of file, one on each line	
+    2. save using `x` then `y` then `enter`
+
+## Prepare /etc/network/interfaces
+doing this means we don't have to give each thunderbolt a manual IPv6 addrees and that these addresses stay constant no matter what
+Add the following to each node using `nano /etc/network/interfaces`
+
+If you see any sections called thunderbolt0 or thunderbol1 delete them at this point.
+
+Now add the following (note we will set IP addresses in the UI):
+
+```
+allow-hotplug en05
+iface en05 inet manual
+       mtu 65520
+
+iface en05 inet6 manual
+        mtu 65520
+
+allow-hotplug en06
+iface en06 inet manual
+        mtu 65520
+
+iface en06 inet6 manual
+        mtu 65520
+
+```
+
+If you see any thunderbol sections delete them from the file before you save it.
+
+## Rename Thunderbolt Connections
+This is needed as proxmox doesn't recognize the thunderbolt interface name.  There are various methods to do this. This method was selected after trial and error because:
+- the thunderboltX naming is not fixed to a port (it seems to be based on sequence you plug the cables in)
+- the MAC address of the interfaces changes with most cable insertion and removale events
+
+
+1. use `udevadm monitor` command to find your device IDs when you insert and remove each TB4 cable. Yes you can use other ways to do this, i recommend this one as it is great way to understand what udev does - the command proved more useful to me than `the syslog` or `lspci command` for troublehsooting thunderbolt issues and behavious.  In my case my two pci paths are `0000:00:0d.2`and `0000:00:0d.3` if you bought the same hardware this will be the same on all 3 units. Don't assume your PCI device paths will be the same as mine.
+
+3. create a link file using `nano /etc/systemd/network/00-thunderbolt0.link` and enter the following content:
+
+```
+[Match]
+Path=pci-0000:00:0d.2
+Driver=thunderbolt-net
+[Link]
+MACAddressPolicy=none
+Name=en05
+```
+3. create a second link file using `nano /etc/systemd/network/00-thunderbolt1.link` and enter the following content:
+```
+[Match]
+Path=pci-0000:00:0d.3
+Driver=thunderbolt-net
+[Link]
+MACAddressPolicy=none
+Name=en06
+```
+
+## Set Interfaces to UP on reboots and cable insertions
+This section en sure that the interfaces will be brought up at boot or cable insertion with whatever settings are in /etc/network/interfaces  - this shouldn't need to be done, it seems like a bug in the way thunderbolt networking is handled (i assume this is debian wide but haven't checked).
+
+1. create a udev rule to detect for cable insertion using `nano /etc/udev/rules.d/10-tb-en.rules` with the following content:
+```
+ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
+ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"
+```
+2. save the file
+
+3. create the first script referenced above using `nano /usr/local/bin/pve-en05.sh` and with the follwing content:
+```
+#!/bin/bash
+
+# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
+/usr/sbin/ifup en05
+```
+save the file and then
+
+3. create the second script referenced above using `nano /usr/local/bin/pve-en06.sh` and with the follwing content:
+```
+#!/bin/bash
+
+# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
+/usr/sbin/ifup en06
+```
+and save the file
+
+4. make both scripts executable with `chmod +x /usr/local/bin/*.sh`
+5. Reboot (restarting networking, init 1 and init 3 are not good enough, so reboot)
+
+## Enabling IP Connectivity
+[proceed to the next gist](/4c664734535da122f4ab2951b22b2085)