Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add openziti network extension #454

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ TARGETS += realtek-firmware
TARGETS += spin
TARGETS += stargz-snapshotter
TARGETS += tailscale
TARGETS += openziti
TARGETS += thunderbolt
TARGETS += usb-modem-drivers
TARGETS += util-linux-tools
Expand Down
50 changes: 50 additions & 0 deletions network/openziti/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# OpenZiti

Runs OpenZiti's edge tunneler in run-host mode allowing to access talos nodes resources on the overlay network

https://openziti.io

https://openziti.io/docs/reference/tunnelers/docker/#use-case-hosting-openziti-services

## Installation

See [Installing Extensions](https://github.com/siderolabs/extensions#installing-extensions).

## Usage

Create an identity for the machine.
Manually enroll it using(for example using ziti edge enroll ...) and output the enrolled identity into a file or stdout
Copy the contents of the enrolled identity
Create `ExtensionServiceConfig` as the example below with replacing `JSON_IDENTITY` with the JSON of the identity you created and in mountPath replace `IDENTITY_FILE_NAME` with the filename of the identity(preserve the json in the filename)

```yaml
---
apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: openziti
configFiles:
- content: 'JSON_IDENTITY'
mountPath: /var/lib/ziti/etc/identities/IDENTITY_FILE_NAME.json
```

Then apply the patch to your node's MachineConfigs
```bash
talosctl patch mc -p @openziti.talos.yaml
```

You will then be able to verify that it is in place with the following command
```bash
talosctl get extensionserviceconfigs

NODE NAMESPACE TYPE ID VERSION
192.168.10.10 runtime ExtensionServiceConfig openziti 1
```

Example of creation of ziti service to serve talos api on the overlay network(note that 192.168.10.10 is the address of the node):
```
ziti edge create config talosctl-controlplane.intercept.v1 intercept.v1 '{"protocols":["tcp"],"addresses": ["talosctl-controlplane.ziti.internal"], "portRanges":[{"low": 50000, "high":50000}]}'
ziti edge create config talosctl-controlplane.host.v1 host.v1 '{"protocol": "tcp","address":"'"192.168.10.10"'", "port": 50000}'
ziti edge create service talosctrl-controlplane.svc --configs talosctl-controlplane.intercept.v1,talosctl-controlplane.host.v1
ziti edge create service-policy talosctl-controlplane.policy.dial Dial --service-roles "@talosctrl-controlplane.svc" --identity-roles "@macos"
ziti edge create service-policy talosctl-controlplane.policy.bind Bind --service-roles "@talosctrl-controlplane.svc" --identity-roles "@talos-cluster-test-identity"
```
12 changes: 12 additions & 0 deletions network/openziti/manifest.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
version: v1alpha1
metadata:
name: openziti
version: "$VERSION"
author: Ruslan Chernenko
description: |
OpenZiti is the world’s most used and widely integrated open source secure networking platform.
This extension deploys a ziti-edge-tunneler in run-host mode which allows to access node's net resources
through the overlay network.
compatibility:
talos:
version: ">= v1.7.0"
50 changes: 50 additions & 0 deletions network/openziti/openziti.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: openziti
depends:
- service: cri
- network:
- addresses
- connectivity
- etcfiles
- configuration: true
container:
entrypoint: /usr/local/bin/ziti-edge-tunnel
args:
- run-host
- --verbose=4
- --identity-dir=/var/lib/ziti/etc/identities
security:
writeableRootfs: false
writeableSysfs: true
mounts:
# libssl lib
- source: /usr/lib
destination: /usr/lib
type: bind
options:
- bind
- ro
- source: /sbin
destination: /sbin
type: bind
options:
- bind
- ro
- source: /dev/net/tun
destination: /dev/net/tun
type: bind
options:
- bind
- rw
- source: /var/lib/ziti/etc/identities/
destination: /var/lib/ziti/etc/identities/
type: bind
options:
- bind
- rw
- source: /var/lib/ziti/
destination: /var/lib/ziti/
type: bind
options:
- bind
- rw
restart: always
65 changes: 65 additions & 0 deletions network/openziti/pkg.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: openziti
variant: alpine
shell: /toolchain/bin/bash
install:
- unzip
- zip
- ninja-build
- ninja
- zlib-static
dependencies:
- image: "ghcr.io/siderolabs/tools:v1.7.0-4-gc844dc3"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks a bit strange... we usually build from base layer which contains toolchain (tools) and don't use alpine for the build. is there anything missing?

Copy link
Author

@nenkoru nenkoru Aug 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no zlib-static, zip, unzip when using scratch and its impossible to use install directive as apk manager doesnt exist in `scratch' variant.
Vcpkg relies on having zip, unzip as it downloads dependencies in that format.
Zlib-static is required to link an executable against it as there is no zlib on the host /usr/lib folder
the build process doesn't build zlib from sources and it has to be manually retrieved from the alpine packages repo in this case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zlib should be available though, zip and unzip, there will be no apk of course.

vcpkg downloads dependencies? does it pin them in a secure way?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I could just change to using 'base' stage instead of explicitly binding to a tools image here.
Build seems to work fine with that. But idk what to do with those zip, unzip, zlib-static, ninja deps

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zlib should be available though, zip and unzip, there will be no apk of course.

vcpkg downloads dependencies? does it pin them in a secure way?

Yes it does by verifying their sha512sum as I skimmed the code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, then this extension needs to add it's build time dependencies as like qemu-guest-agent for example

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, then this extension needs to add it's build time dependencies as like qemu-guest-agent for example

I guess would be hard to do as it heavily depends on vcpkg as the dependency manager.
Maybe I could try working on creating static builds and then pulling them within the build process. Would this be okay?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean pulling them from the releases page, as you suggested above.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, let's try static builds from the releases page. how big are those?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, let's try static builds from the releases page. how big are those?

Well, around 5MB, I am currently working on buildin those using gh workflows, just having a linker issue with CMAKE trying to link libpthread.so instead of libpthread.a.
openziti/ziti-tunnel-sdk-c#938

steps:
- sources:
- url: https://github.com/openziti/ziti-tunnel-sdk-c/archive/refs/tags/{{ .OPENZITI_TUNNELER_VERSION }}.tar.gz
destination: edge-tunnel-source.tar.gz
sha256: d455672bf3b6ff28fd0ff864f868f7d6e3de99e6c666a120227fa9dab2d91f78
sha512: 5aeb05347381124e9d75693f12e0d234499a8c6482d322019576a24c622d986d43f159454a43c844623f171a456062a5e1afffcee7f7cbb03740216dbd3bb74c
env:
VCPKG_ROOT: /vcpkg
VCPKG_FORCE_SYSTEM_BINARIES: 1
CC: /toolchain/bin/gcc
CXX: /toolchain/bin/g++
prepare:
- |
# vcpkg depends on git as a delivery tool
# if to download .tar.gz from releases it would fail saying it cannot retrieve versions/baseline.json version using git show
# tried my best to find any CMAKE variable within vcpkg which disables this behaviour
# we fetch here commit to which tag 2024.07.12 is assigned
git clone https://github.com/microsoft/vcpkg.git /vcpkg
git --git-dir=/vcpkg/.git --work-tree=/vcpkg checkout 1de2026f28ead93ff1773e6e680387643e914ea1

- |
sed -i 's#$VERSION#{{ .VERSION }}#' /pkg/manifest.yaml

- |
mkdir -p /vcpkg-git /vcpkg /ziti-tunnel-sdk-c/build
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we building an sdk? doesn't make sense, shouldn't it be using this https://github.com/openziti/ziti/?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we building an sdk? doesn't make sense, shouldn't it be using this https://github.com/openziti/ziti/?

No we shouldn't. OpenZiti tunneler is exactly for hosting or tunneling. Main ziti repo is for the other part of the OpenZiti project which provides a controller, router and etc.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tunneler is just a daemon that runs on the host machines and allows to put the network communication on or from the overlay network it creates. For Talos its impossible to change routing from the extension, so the tunneler runs in a 'run-host' mode which allows offloading of a overlay traffic on the underlay network.

tar -xzvf edge-tunnel-source.tar.gz --strip-components=1 -C /ziti-tunnel-sdk-c

build:
- |
export PATH=${PATH}:/toolchain/bin
/vcpkg/bootstrap-vcpkg.sh

- |
export PATH=${PATH}:/toolchain/bin
PRESET="ci-linux-x64"
if [[ "$(uname -m)" == "arm64" || "$(uname -m)" == "aarch64" ]]; then
PRESET="ci-linux-arm64"
ln -s /toolchain/bin/g++ /toolchain/bin/aarch64-linux-gnu-g++
ln -s /toolchain/bin/gcc /toolchain/bin/aarch64-linux-gnu-gcc
fi
cmake -DBUILD_SHARED_LIBS=OFF -DCMAKE_EXE_LINKER_FLAGS="-static" -DDISABLE_SEMVER_VERIFICATION=ON -DDISABLE_LIBSYSTEMD_FEATURE=ON -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja --preset $PRESET -S /ziti-tunnel-sdk-c -B /ziti-tunnel-sdk-c/build
- |
cmake --build /ziti-tunnel-sdk-c/build --config Release
install:
- |
mkdir -p /rootfs/usr/local/lib/containers/openziti/usr/local/bin/
mv /ziti-tunnel-sdk-c/build/programs/ziti-edge-tunnel/Release/ziti-edge-tunnel /rootfs/usr/local/lib/containers/openziti/usr/local/bin/
finalize:
- from: /rootfs
to: /rootfs
- from: /pkg/manifest.yaml
to: /
- from: /pkg/openziti.yaml
to: /rootfs/usr/local/etc/containers/
1 change: 1 addition & 0 deletions network/openziti/vars.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
VERSION: "{{ .OPENZITI_TUNNELER_VERSION }}"
1 change: 1 addition & 0 deletions network/vars.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
# renovate: datasource=github-releases extractVersion=^v(?<version>.*)$ depName=tailscale/tailscale
TAILSCALE_VERSION: 1.70.0
OPENZITI_TUNNELER_VERSION: v1.1.2