From 8ccadea8423c228d0cf229ba25882d0e0b4c5da7 Mon Sep 17 00:00:00 2001 From: weyne85 Date: Wed, 22 May 2024 08:30:09 +0200 Subject: [PATCH] Create Thunderbolt_Networking.md --- Thunderbolt_Networking.md | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 Thunderbolt_Networking.md diff --git a/Thunderbolt_Networking.md b/Thunderbolt_Networking.md new file mode 100644 index 0000000..ca6015e --- /dev/null +++ b/Thunderbolt_Networking.md @@ -0,0 +1,103 @@ +# Thunderbolt Networking + +[this gist is part of this series](/76e94832927a89d977ea989da157e9dc) + +#### NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve OR HIGER +This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol) + +## Install LLDP - this is great to see what nodes can see which. +- install lldpctl with `apt install lldpd` + +## Load Kernel Modules +- add `thunderbolt` and `thunderbolt-net` kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-) + 1. `nano /etc/modules` add modules at bottom of file, one on each line + 2. save using `x` then `y` then `enter` + +## Prepare /etc/network/interfaces +doing this means we don't have to give each thunderbolt a manual IPv6 addrees and that these addresses stay constant no matter what +Add the following to each node using `nano /etc/network/interfaces` + +If you see any sections called thunderbolt0 or thunderbol1 delete them at this point. + +Now add the following (note we will set IP addresses in the UI): + +``` +allow-hotplug en05 +iface en05 inet manual + mtu 65520 + +iface en05 inet6 manual + mtu 65520 + +allow-hotplug en06 +iface en06 inet manual + mtu 65520 + +iface en06 inet6 manual + mtu 65520 + +``` + +If you see any thunderbol sections delete them from the file before you save it. + +## Rename Thunderbolt Connections +This is needed as proxmox doesn't recognize the thunderbolt interface name. There are various methods to do this. This method was selected after trial and error because: +- the thunderboltX naming is not fixed to a port (it seems to be based on sequence you plug the cables in) +- the MAC address of the interfaces changes with most cable insertion and removale events + + +1. use `udevadm monitor` command to find your device IDs when you insert and remove each TB4 cable. Yes you can use other ways to do this, i recommend this one as it is great way to understand what udev does - the command proved more useful to me than `the syslog` or `lspci command` for troublehsooting thunderbolt issues and behavious. In my case my two pci paths are `0000:00:0d.2`and `0000:00:0d.3` if you bought the same hardware this will be the same on all 3 units. Don't assume your PCI device paths will be the same as mine. + +3. create a link file using `nano /etc/systemd/network/00-thunderbolt0.link` and enter the following content: + +``` +[Match] +Path=pci-0000:00:0d.2 +Driver=thunderbolt-net +[Link] +MACAddressPolicy=none +Name=en05 +``` +3. create a second link file using `nano /etc/systemd/network/00-thunderbolt1.link` and enter the following content: +``` +[Match] +Path=pci-0000:00:0d.3 +Driver=thunderbolt-net +[Link] +MACAddressPolicy=none +Name=en06 +``` + +## Set Interfaces to UP on reboots and cable insertions +This section en sure that the interfaces will be brought up at boot or cable insertion with whatever settings are in /etc/network/interfaces - this shouldn't need to be done, it seems like a bug in the way thunderbolt networking is handled (i assume this is debian wide but haven't checked). + +1. create a udev rule to detect for cable insertion using `nano /etc/udev/rules.d/10-tb-en.rules` with the following content: +``` +ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh" +ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh" +``` +2. save the file + +3. create the first script referenced above using `nano /usr/local/bin/pve-en05.sh` and with the follwing content: +``` +#!/bin/bash + +# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface +/usr/sbin/ifup en05 +``` +save the file and then + +3. create the second script referenced above using `nano /usr/local/bin/pve-en06.sh` and with the follwing content: +``` +#!/bin/bash + +# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface +/usr/sbin/ifup en06 +``` +and save the file + +4. make both scripts executable with `chmod +x /usr/local/bin/*.sh` +5. Reboot (restarting networking, init 1 and init 3 are not good enough, so reboot) + +## Enabling IP Connectivity +[proceed to the next gist](/4c664734535da122f4ab2951b22b2085)