Skip to content

Circuit breaker plugin #688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
73edcab
circuit breaker plugin readme
Camillarhi Feb 27, 2025
63d41bb
circuit breaker plugin skeleton setup
Camillarhi Feb 27, 2025
83f9f41
Circuit breaker plugin integration
Camillarhi Feb 27, 2025
caeb5ad
circuit breaker setup
Camillarhi Mar 2, 2025
9d42a45
circuit breaker setup
Camillarhi Mar 3, 2025
bccc111
connect to the local lnd node and access circuit breaker from the UI
Camillarhi Mar 3, 2025
9bcadb6
use nodeport to expose circuit breaker UI
Camillarhi Mar 4, 2025
370a830
update lnd pod to run circuit breaker
Camillarhi Mar 9, 2025
01c04f1
plugin: remove circuit-breaker from the plugin section
Camillarhi Mar 9, 2025
a1c8645
plugin: remove circuit-breaker from the plugin section
Camillarhi Mar 10, 2025
51e2f61
template: add extracontainers for circuitbreaker
Camillarhi Mar 12, 2025
00ce51c
pod: update pod for extra containers
Camillarhi Mar 12, 2025
8cecc14
update volume mounts
Camillarhi Mar 17, 2025
067d98e
Circuitbreaker Update (#4)
Camillarhi Mar 12, 2025
f2f23d7
update init containers
Camillarhi Mar 18, 2025
a2f25aa
update shared volume from `values.yaml`
Camillarhi Mar 19, 2025
aef4f9c
circuit breaker plugin readme
Camillarhi Feb 27, 2025
0418553
circuit breaker plugin skeleton setup
Camillarhi Feb 27, 2025
a1bbd23
Merge pull request #7 from Camillarhi/circuitbreaker4
Camillarhi Mar 19, 2025
91da2a1
remove plugin implementation
Camillarhi Mar 19, 2025
6745906
Merge branch 'circuit-breaker-plugin' of https://github.com/Camillarh…
Camillarhi Mar 19, 2025
61da106
Define new values in `network.yaml` file
Camillarhi Mar 21, 2025
8572824
update line endings
Camillarhi Mar 22, 2025
58257a9
update macaroon path to use hex value
Camillarhi Mar 26, 2025
acf25a1
removed init containers and shared volumes
Camillarhi Mar 26, 2025
8e7b135
add circuit breaker to the extra containers
Camillarhi Mar 27, 2025
8c541b1
update network and port to be pulled as global helm variables
Camillarhi Apr 28, 2025
b775b6b
remove extracontainers from test file
Camillarhi Apr 28, 2025
bb663f8
Include unit tests for circuitbreaker
Camillarhi Apr 29, 2025
746b9f7
update file linting and formating
Camillarhi Apr 29, 2025
914ba77
fix failing tests
Camillarhi Apr 29, 2025
f0b0b9b
increase timeout on port-forwarding
Camillarhi Apr 29, 2025
074192e
increase timeout on port-forwarding
Camillarhi Apr 29, 2025
dc56158
Set up Kubernetes Service access to the Circuit Breaker API
Camillarhi Apr 29, 2025
03954ca
update file linting and formating
Camillarhi Apr 29, 2025
b634e1e
update file linting and formating
Camillarhi Apr 29, 2025
64c1f1f
Set up Kubernetes Service access to the Circuit Breaker API
Camillarhi Apr 29, 2025
dc82fa4
remove interactive flags
Camillarhi Apr 29, 2025
c265bb1
remove interactive flags
Camillarhi Apr 29, 2025
9035bd6
remove interactive flags
Camillarhi Apr 29, 2025
46bb5b6
update circuitbreaker request method
Camillarhi Apr 29, 2025
c4289d1
update docker image tag
Camillarhi Apr 29, 2025
0e440f5
remove change
Camillarhi Apr 29, 2025
b4d66e7
fix lint
Camillarhi Apr 29, 2025
08903ec
remove change
Camillarhi Apr 29, 2025
c044f17
Set up Kubernetes Service access to the Circuit Breaker API
Camillarhi Apr 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions resources/charts/bitcoincore/charts/lnd/templates/pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,37 @@ spec:
- mountPath: /root/.lnd/tls.cert
name: config
subPath: tls.cert
- name: shared-volume
mountPath: /root/.lnd/
Comment on lines +61 to +62
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might not actually need this shared volume at all. Because lnd and circuitbreaker are in the same pod, they already share a filesystem. So you'd just need to configure circuitbreaker to read from the right location. There's also apparently an issue getting the network value into the config.

When I warnet deploy resources/networks/hello I see this error in the circuitbreaker container:

2025-04-29T13:54:23.979Z    INFO    Circuit Breaker starting    {"version": ""}
2025-04-29T13:54:23.979Z    INFO    Opening database    {"path": "/root/.circuitbreaker/circuitbreaker.db"}
2025-04-29T13:54:24.007Z    INFO    Applied migrations    {"count": 4}
2025-04-29T13:54:24.007Z    ERROR    Unexpected exit    {"err": "unable to read macaroon path (check the network setting!): open /root/.lnd/data/chain/bitcoin/regtest/admin.macaroon: no such file or directory"}
main.main
    /src/main.go:152
runtime.main
    /usr/local/go/src/runtime/proc.go:250
Stream closed EOF for default/tank-0003-ln (circuitbreaker)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Camillarhi did you try removing the shared volume?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking into other comments. I will look into this as soon as I can

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pinheadmz , After testing both approaches, I've confirmed that a shared volume is necessary for reliable file access between containers in the same pod. While containers in a pod share the same network namespace, they maintain separate filesystems unless explicitly shared through volumes.

{{- with .Values.extraContainers }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- if .Values.circuitbreaker.enabled }}
- name: circuitbreaker
image: {{ .Values.circuitbreaker.image | quote }}
imagePullPolicy: IfNotPresent
args:
- "--network={{ .Values.global.chain }}"
- "--rpcserver=localhost:{{ .Values.RPCPort }}"
- "--tlscertpath=/tls.cert"
- "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
- "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
volumeMounts:
- name: shared-volume
mountPath: /root/.lnd/
- name: config
mountPath: /tls.cert
subPath: tls.cert
{{- end }}
Comment on lines +66 to +82
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there an issue including circuitbreaker as extraContainers container?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was not able to access the value of the port and network variables as the extraContainers is being parsed by a YAML parser before Helm gets to process the templates.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm the whole point of those extraContainers was for the cb plugin ... :-/

Copy link
Author

@Camillarhi Camillarhi May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I understand, but I could not access a dynamic network or port while using it. The current approach also gives the option of enabling circuitbreaker if needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pinheadmz I assume it is acceptable to define network(chain) in network.yaml or node-defaults.yaml based on user desired scenario. if so then extraContainers should be possible...

volumes:
{{- with .Values.volumes }}
{{- toYaml . | nindent 4 }}
{{- end }}
- configMap:
name: {{ include "lnd.fullname" . }}
name: config
- name: shared-volume
emptyDir: {}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 4 }}
Expand Down
5 changes: 5 additions & 0 deletions resources/charts/bitcoincore/charts/lnd/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -132,3 +132,8 @@ config: ""
defaultConfig: ""

channels: []

circuitbreaker:
enabled: false # Default to disabled
image: carlakirkcohen/circuitbreaker:attackathon-test
httpPort: 9235
4 changes: 4 additions & 0 deletions resources/networks/hello/network.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ nodes:
target: tank-0004-ln
capacity: 100000
push_amt: 50000
circuitbreaker:
enabled: true # This enables circuitbreaker for this node
httpPort: 9235 # Can override defaults per-node

- name: tank-0004
addnode:
Expand Down Expand Up @@ -85,3 +88,4 @@ plugins: # Each plugin section has a number of hooks available (preDeploy, post
entrypoint: "../../plugins/hello"
helloTo: "postNetwork!"
podName: "hello-post-network"

4 changes: 4 additions & 0 deletions test/data/ln/network.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ nodes:
target: tank-0004-ln
capacity: 100000
push_amt: 50000
circuitbreaker:
enabled: true
httpPort: 9235

- name: tank-0004
addnode:
- tank-0000
Expand Down
119 changes: 119 additions & 0 deletions test/ln_basic_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@

import json
import os
import random
import subprocess
import time
from pathlib import Path
from time import sleep

import requests
from test_base import TestBase

from warnet.process import stream_command
Expand All @@ -24,11 +28,18 @@ def __init__(self):
"tank-0005-ln",
]

self.cb_port = 9235
self.cb_node = "tank-0003-ln"
self.port_forward = None

def run_test(self):
try:
# Wait for all nodes to wake up. ln_init will start automatically
self.setup_network()

# Test circuit breaker API
self.test_circuit_breaker_api()

# Send a payment across channels opened automatically by ln_init
self.pay_invoice(sender="tank-0005-ln", recipient="tank-0003-ln")

Expand All @@ -39,6 +50,7 @@ def run_test(self):
self.pay_invoice(sender="tank-0000-ln", recipient="tank-0002-ln")

finally:
self.cleanup_kubectl_created_services()
self.cleanup()

def setup_network(self):
Expand Down Expand Up @@ -120,6 +132,113 @@ def scenario_open_channels(self):
self.log.info(f"Running scenario from: {scenario_file}")
self.warnet(f"run {scenario_file} --source_dir={self.scen_dir} --debug")

def test_circuit_breaker_api(self):
self.log.info("Testing Circuit Breaker API")

# Set up port forwarding to the circuit breaker
cb_url = self.setup_api_access(self.cb_node)

self.log.info(f"Testing Circuit Breaker API at {cb_url}")

# Test /info endpoint
info = self.cb_api_request(cb_url, "get", "/info")
assert "version" in info, "Circuit breaker info missing version"

# Test /limits endpoint
limits = self.cb_api_request(cb_url, "get", "/limits")
assert isinstance(limits, dict), "Limits should be a dictionary"

self.log.info("✅ Circuit Breaker API tests passed")

def setup_api_access(self, pod_name):
"""Set up Kubernetes Service access to the Circuit Breaker API"""
# Create a properly labeled service using kubectl expose
service_name = f"{pod_name}-svc"

self.log.info(f"Creating service {service_name} for pod {pod_name}")
try:
subprocess.run(
[
"kubectl",
"expose",
"pod",
pod_name,
"--name",
service_name,
"--port",
str(self.cb_port),
"--target-port",
str(self.cb_port),
],
check=True,
)
except subprocess.CalledProcessError as e:
self.log.error(f"Failed to create service: {e.stderr}")
raise
Comment on lines +159 to +177
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could add service.yaml definition instead of creating service on the fly...
image

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I will do this instead

Copy link
Contributor

@pinheadmz pinheadmz May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is overkill. See the legacy test, you can just execute an API request form inside the cb container. (this exact code wont work any more but maybe gives a good hint)

warnet/test/ln_test.py

Lines 34 to 39 in 01195c2

def get_cb_forwards(self, index):
cmd = "wget -q -O - 127.0.0.1:9235/api/forwarding_history"
res = self.wait_for_rpc(
"exec_run", [index, ServiceType.CIRCUITBREAKER.value, cmd, self.network_name]
)
return json.loads(res)

Copy link
Author

@Camillarhi Camillarhi May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into the legacy test but was unable to replicate it or access circuitbreaker through localhost(127.0.0.1) without port forwarding. I’ll do more research to figure out how to properly access it, but I’m also open to suggestions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could add a util funciton to k8s.py to execute an arbitrary command inside a container (surprised we dont have that alreaady actually....) and then execute the command curl 127.0.0.1/whatever/cb/api...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh right or do something like this in the test:

cmd = f"kubectl -n {namespace} exec {tank} --container {BITCOINCORE_CONTAINER} -- bitcoin-cli {method} {' '.join(map(str, params))}"
return run_command(cmd)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I will try this


time.sleep(51) # Wait for the service to be created

service_url = f"http://{service_name}:{self.cb_port}/api"
self.service_to_cleanup = service_name
self.log.info(f"Service URL: {service_url}")

self.log.info(f"Successfully created service at {service_url}")
return service_url

def cb_api_request(self, base_url, method, endpoint, data=None):
"""Universal API request handler with proper path handling"""
try:
# Parse the base URL components
url_parts = base_url.split("://")[1].split("/")
service_name = url_parts[0].split(":")[0]
port = url_parts[0].split(":")[1] if ":" in url_parts[0] else "80"
base_path = "/" + "/".join(url_parts[1:]) if len(url_parts) > 1 else "/"

# Set up port forwarding
local_port = random.randint(10000, 20000)
pf = subprocess.Popen(
["kubectl", "port-forward", f"svc/{service_name}", f"{local_port}:{port}"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)

try:
# Wait for port-forward to establish
time.sleep(2)

# Construct the full local URL with proper path handling
full_path = base_path.rstrip("/") + "/" + endpoint.lstrip("/")
local_url = f"http://localhost:{local_port}{full_path}"

self.log.debug(f"Attempting API request to: {local_url}")

# Make the request
if method.lower() == "get":
response = requests.get(local_url, timeout=30)
else:
response = requests.post(local_url, json=data, timeout=30)

response.raise_for_status()
return response.json()

finally:
pf.terminate()
pf.wait()

except Exception as e:
self.log.error(f"API request to {local_url} failed: {str(e)}")
raise

def cleanup_kubectl_created_services(self):
"""Clean up any created resources"""
if hasattr(self, "service_to_cleanup") and self.service_to_cleanup:
self.log.info(f"Deleting service {self.service_to_cleanup}")
subprocess.run(
["kubectl", "delete", "svc", self.service_to_cleanup],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)


if __name__ == "__main__":
test = LNBasicTest()
Expand Down
Loading