Add caral documentation

nixos-configurations/cyprianspitz/README.md

@@ -1,4 +1,4 @@
-# Host: Cyprianspitz
+# Host: Cyprianspitz (+Router: Caral)
 
 ## Operations {#_operations}
 
@@ -25,6 +25,38 @@ TODO
 - [Mainboard Manual](docs/z790m-itx-wifi.pdf)
 
 
+### Networking: Caral Internet Router
+
+A [MikroTik `CCR2004-1G-2XS-PCIe`](https://mikrotik.com/product/ccr2004_1g_2xs_pcie#fndtn-downloads) is used for internet access.
+It's a fiber card with build in router, supporting 2x 25Gbit SFP28 cages and 1Gbit RJ45 eth.
+
+- [RouterOS Docs](https://help.mikrotik.com/docs/spaces/ROS/pages/328059/RouterOS)
+
+[The manual](docs/CCR2004-1G-2XS-PCIe_241138.pdf) states:
+
+> This form-factor does come with certain limitations that you should keep in mind.
+> The CCR NIC card needs some time to boot up compared to ASIC-based setups.
+> If the host system is up before the CCR card, it will not appear among the available devices.
+> You should add a PCIe device initialization delay after power-up in the BIOS.
+> Or you will need to re-initialize the PCIe devices from the HOST system.
+
+In our case, since networking is reinitialized after the LUKS password promt, this should not be a issue in practice. However, if networking would not be available, contact someone for a physical reboot and wait longer before entering the HDD password.
+
+To reload the card's virtual interfaces on a running system:
+
+```bash
+echo "1" > /sys/bus/pci/devices/0000\:01\:00.0/remove
+sleep 2
+echo "1" > /sys/bus/pci/rescan
+```
+
+To restart the card on a running system:
+
+```bash
+echo "1" > /sys/bus/pci/devices/0000\:01\:00.0/reset
+sleep 2m # Wait for reboot
+echo "1" > /sys/bus/pci/rescan
+```
 
 ### Top Overview
 

nixos-configurations/cyprianspitz/networking.nix

@@ -14,9 +14,25 @@ in
 
   networking.nameservers = [ calandaIp ];
   networking.useDHCP = false;
-  networking.interfaces.enp0s31f6.ipv4.addresses = [
+  networking.interfaces = {
-    (getNetV4Ip meta.network.physical.plessur-lan)
+    # enp0s31f6: 1 Gbit mainboard interface
-  ];
+    enp0s31f6.ipv4.addresses = [
+      (getNetV4Ip meta.network.physical.plessur-lan)
+    ];
+
+    # wlp0s20f3: Mainboard Wireless interface
+    # enp3s0: 2.5 Gbit mainboard interface: Connected to ether1
+    #enp3s0.useDHCP = true;
+
+    # enp1s0f0: mikrotik sfp28-1: ether-pcie1 passthrough
+    enp1s0f0.useDHCP = true;
+    # enp1s0f1: mikrotik sfp28-2: ether-pcie2 passthrough
+    enp1s0f1.useDHCP = true;
+    # enp1s0f2: mikrotik ether1/bridge1: ether-pcie3 bridge \
+    enp1s0f2.useDHCP = true;
+    # enp1s0f3: mikrotik ether1/bridge1: ether-pcie4 bridge  > connected to enp3s0
+    enp1s0f3.useDHCP = true;
+  };
 
   networking.defaultGateway = {
     address = calandaIp;

private

@@ -1 +1 @@
-Subproject commit 18d3b3b703a6139b9ebd5ec64311717cf2a6f9bc
+Subproject commit 90543f538fc13c3d291196a9769483c6c51ac84d