Circuit breaker plugin #688

Camillarhi · 2025-03-06T15:36:31Z

No description provided.

pinheadmz · 2025-03-07T16:41:14Z

weird error trying to warnet deploy resources/networks/hello/ but might be related to #689 like, plugin.py needs chmod a+x or something.

Queuing postNetwork plugin command: hello with {'entrypoint': '../../plugins/hello', 'helloTo': 'postNetwork!', 'podName': 'hello-post-network'}
Starting postNetwork plugins
Completed postNetwork plugins
Queuing postDeploy plugin command: hello with {'entrypoint': '../../plugins/hello', 'podName': 'hello-post-deploy', 'helloTo': 'postDeploy!'}
Queuing postDeploy plugin command: simln with {'entrypoint': '../../plugins/simln', 'activity': '[{"source": "tank-0003-ln", "destination": "tank-0005-ln", "interval_secs": 1, "amount_msat": 2000}]'}
Queuing postDeploy plugin command: circuitbreaker with {'entrypoint': '../../plugins/circuitbreaker', 'podName': 'circuitbreaker-pod', 'rpcserver': '172.29.34.166:10009', 'httplisten': '0.0.0.0:9235'}
Starting postDeploy plugins
Process Process-11:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/matthewzipkin/Desktop/work/warnet/src/warnet/process.py", line 7, in run_command
    raise Exception(result.stderr)
Exception: bash: line 1: resources/networks/hello/../../plugins/circuitbreaker/plugin.py: Permission denied

Camillarhi · 2025-03-08T00:41:22Z

weird error trying to warnet deploy resources/networks/hello/ but might be related to #689 like, plugin.py needs chmod a+x or something.

Queuing postNetwork plugin command: hello with {'entrypoint': '../../plugins/hello', 'helloTo': 'postNetwork!', 'podName': 'hello-post-network'}
Starting postNetwork plugins
Completed postNetwork plugins
Queuing postDeploy plugin command: hello with {'entrypoint': '../../plugins/hello', 'podName': 'hello-post-deploy', 'helloTo': 'postDeploy!'}
Queuing postDeploy plugin command: simln with {'entrypoint': '../../plugins/simln', 'activity': '[{"source": "tank-0003-ln", "destination": "tank-0005-ln", "interval_secs": 1, "amount_msat": 2000}]'}
Queuing postDeploy plugin command: circuitbreaker with {'entrypoint': '../../plugins/circuitbreaker', 'podName': 'circuitbreaker-pod', 'rpcserver': '172.29.34.166:10009', 'httplisten': '0.0.0.0:9235'}
Starting postDeploy plugins
Process Process-11:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/matthewzipkin/Desktop/work/warnet/src/warnet/process.py", line 7, in run_command
    raise Exception(result.stderr)
Exception: bash: line 1: resources/networks/hello/../../plugins/circuitbreaker/plugin.py: Permission denied

Yeah, I believe this issue is related to #689. To resolve it, I ran chmod a+x on the relevant plugin files and converted line Endings with dos2unix. That got everything working for me.

willcl-ark · 2025-03-09T20:30:17Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/plugins/circuitbreaker/plugin.py

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

* plugin: remove circuit-breaker from the plugin section * template: add extracontainers for circuitbreaker * pod: update pod for extra containers

Circuitbreaker4

…i/warnet into circuit-breaker-plugin

pinheadmz

I noticed a few things to check. Coudn't get the branch to run locally probably just because of the image error?

Also checkout the legacy branch for how we did integration tests for circuitbreaker:

https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py#L34-L39

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/networks/hello/network.yaml

pinheadmz

Looking good, thanks for the update. Be sure to check out the legacy ln test that covered circuitbreaker: https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py

I don't think the hello network is very useful actually, and it would be better to add the circuit breaker plugin to one of the networks in test/data so it can actually be run and tested by ci.

pinheadmz · 2025-04-29T13:57:46Z

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

+        - name: shared-volume
+          mountPath: /root/.lnd/


I think you might not actually need this shared volume at all. Because lnd and circuitbreaker are in the same pod, they already share a filesystem. So you'd just need to configure circuitbreaker to read from the right location. There's also apparently an issue getting the network value into the config.

When I warnet deploy resources/networks/hello I see this error in the circuitbreaker container:

2025-04-29T13:54:23.979Z INFO Circuit Breaker starting {"version": ""} 2025-04-29T13:54:23.979Z INFO Opening database {"path": "/root/.circuitbreaker/circuitbreaker.db"} 2025-04-29T13:54:24.007Z INFO Applied migrations {"count": 4} 2025-04-29T13:54:24.007Z ERROR Unexpected exit {"err": "unable to read macaroon path (check the network setting!): open /root/.lnd/data/chain/bitcoin/regtest/admin.macaroon: no such file or directory"} main.main /src/main.go:152 runtime.main /usr/local/go/src/runtime/proc.go:250 Stream closed EOF for default/tank-0003-ln (circuitbreaker)

@Camillarhi did you try removing the shared volume?

I was looking into other comments. I will look into this as soon as I can

@pinheadmz , After testing both approaches, I've confirmed that a shared volume is necessary for reliable file access between containers in the same pod. While containers in a pod share the same network namespace, they maintain separate filesystems unless explicitly shared through volumes.

resources/charts/bitcoincore/charts/lnd/values.yaml

Camillarhi · 2025-04-29T14:26:20Z

Looking good, thanks for the update. Be sure to check out the legacy ln test that covered circuitbreaker: https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py

I don't think the hello network is very useful actually, and it would be better to add the circuit breaker plugin to one of the networks in test/data so it can actually be run and tested by ci.

Thanks. I have added circuitbreaker to a network and some unit tests for it has been included as well

resources/charts/bitcoincore/charts/lnd/templates/configmap.yaml

macgyver13 · 2025-04-29T19:31:59Z

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

+    {{- if .Values.circuitbreaker.enabled }}
+    - name: circuitbreaker
+      image: {{ .Values.circuitbreaker.image | quote }}
+      imagePullPolicy: IfNotPresent
+      args:
+        - "--network={{ .Values.global.chain }}"
+        - "--rpcserver=localhost:{{ .Values.RPCPort }}"
+        - "--tlscertpath=/tls.cert"
+        - "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
+        - "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
+      volumeMounts:
+        - name: shared-volume
+          mountPath: /root/.lnd/
+        - name: config
+          mountPath: /tls.cert
+          subPath: tls.cert
+    {{- end }}


Was there an issue including circuitbreaker as extraContainers container?

Yes, I was not able to access the value of the port and network variables as the extraContainers is being parsed by a YAML parser before Helm gets to process the templates.

Hm the whole point of those extraContainers was for the cb plugin ... :-/

Yes, I understand, but I could not access a dynamic network or port while using it. The current approach also gives the option of enabling circuitbreaker if needed.

@pinheadmz I assume it is acceptable to define network(chain) in network.yaml or node-defaults.yaml based on user desired scenario. if so then extraContainers should be possible...

macgyver13 · 2025-04-29T19:39:27Z

test/ln_basic_test.py

+        try:
+            subprocess.run(
+                [
+                    "kubectl",
+                    "expose",
+                    "pod",
+                    pod_name,
+                    "--name",
+                    service_name,
+                    "--port",
+                    str(self.cb_port),
+                    "--target-port",
+                    str(self.cb_port),
+                ],
+                check=True,
+            )
+        except subprocess.CalledProcessError as e:
+            self.log.error(f"Failed to create service: {e.stderr}")
+            raise


You could add service.yaml definition instead of creating service on the fly...

Thanks. I will do this instead

I think this is overkill. See the legacy test, you can just execute an API request form inside the cb container. (this exact code wont work any more but maybe gives a good hint)

warnet/test/ln_test.py

Lines 34 to 39 in 01195c2

def get_cb_forwards(self, index):

cmd = "wget -q -O - 127.0.0.1:9235/api/forwarding_history"

res = self.wait_for_rpc(

"exec_run", [index, ServiceType.CIRCUITBREAKER.value, cmd, self.network_name]

)

return json.loads(res)

I looked into the legacy test but was unable to replicate it or access circuitbreaker through localhost(127.0.0.1) without port forwarding. I’ll do more research to figure out how to properly access it, but I’m also open to suggestions.

you could add a util funciton to k8s.py to execute an arbitrary command inside a container (surprised we dont have that alreaady actually....) and then execute the command curl 127.0.0.1/whatever/cb/api...

oh right or do something like this in the test:

cmd = f"kubectl -n {namespace} exec {tank} --container {BITCOINCORE_CONTAINER} -- bitcoin-cli {method} {' '.join(map(str, params))}" return run_command(cmd)

Thanks. I will try this

Camillarhi mentioned this pull request Mar 6, 2025

Circuitbreaker4 #687

Closed

willcl-ark mentioned this pull request Mar 9, 2025

Enable lnd pod logging #691

Merged

macgyver13 reviewed Mar 9, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Show resolved Hide resolved

macgyver13 suggested changes Mar 9, 2025

View reviewed changes

resources/plugins/circuitbreaker/plugin.py Outdated Show resolved Hide resolved

macgyver13 suggested changes Mar 18, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

Camillarhi and others added 18 commits March 19, 2025 02:16

circuit breaker plugin readme

73edcab

circuit breaker plugin skeleton setup

63d41bb

Circuit breaker plugin integration

83f9f41

circuit breaker setup

caeb5ad

circuit breaker setup

9d42a45

connect to the local lnd node and access circuit breaker from the UI

bccc111

use nodeport to expose circuit breaker UI

9bcadb6

update lnd pod to run circuit breaker

370a830

plugin: remove circuit-breaker from the plugin section

01c04f1

plugin: remove circuit-breaker from the plugin section

a1c8645

template: add extracontainers for circuitbreaker

51e2f61

pod: update pod for extra containers

00ce51c

update volume mounts

8cecc14

Circuitbreaker Update (#4)

067d98e

* plugin: remove circuit-breaker from the plugin section * template: add extracontainers for circuitbreaker * pod: update pod for extra containers

update init containers

f2f23d7

update shared volume from values.yaml

a2f25aa

circuit breaker plugin readme

aef4f9c

circuit breaker plugin skeleton setup

0418553

Camillarhi force-pushed the circuit-breaker-plugin branch from ce43d60 to 0418553 Compare March 19, 2025 01:22

Camillarhi and others added 3 commits March 19, 2025 02:23

Merge pull request #7 from Camillarhi/circuitbreaker4

a1bbd23

Circuitbreaker4

remove plugin implementation

91da2a1

Merge branch 'circuit-breaker-plugin' of https://github.com/Camillarh…

6745906

…i/warnet into circuit-breaker-plugin

Camillarhi added 4 commits March 22, 2025 23:42

update line endings

8572824

update macaroon path to use hex value

58257a9

removed init containers and shared volumes

acf25a1

add circuit breaker to the extra containers

8e7b135

pinheadmz reviewed Apr 7, 2025

View reviewed changes

Camillarhi added 2 commits April 29, 2025 00:31

update network and port to be pulled as global helm variables

8c541b1

remove extracontainers from test file

b775b6b

pinheadmz reviewed Apr 29, 2025

View reviewed changes

Camillarhi added 2 commits April 29, 2025 15:14

Include unit tests for circuitbreaker

bb663f8

update file linting and formating

746b9f7

Camillarhi added 10 commits April 29, 2025 16:13

fix failing tests

914ba77

increase timeout on port-forwarding

f0b0b9b

increase timeout on port-forwarding

074192e

Set up Kubernetes Service access to the Circuit Breaker API

dc56158

update file linting and formating

03954ca

update file linting and formating

b634e1e

Set up Kubernetes Service access to the Circuit Breaker API

64c1f1f

remove interactive flags

dc82fa4

remove interactive flags

c265bb1

remove interactive flags

9035bd6

macgyver13 reviewed Apr 29, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/configmap.yaml Outdated Show resolved Hide resolved

macgyver13 reviewed Apr 29, 2025

View reviewed changes

Camillarhi added 6 commits April 29, 2025 20:52

update circuitbreaker request method

46bb5b6

update docker image tag

c4289d1

remove change

0e440f5

fix lint

b4d66e7

remove change

08903ec

Set up Kubernetes Service access to the Circuit Breaker API

c044f17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Circuit breaker plugin #688

Circuit breaker plugin #688

Camillarhi commented Mar 6, 2025

pinheadmz commented Mar 7, 2025

Camillarhi commented Mar 8, 2025

willcl-ark commented Mar 9, 2025 •

edited by bdp-DrahtBot

Loading

pinheadmz left a comment

pinheadmz left a comment

pinheadmz Apr 29, 2025

pinheadmz May 1, 2025

Camillarhi May 1, 2025

Camillarhi May 1, 2025

Camillarhi commented Apr 29, 2025

macgyver13 Apr 29, 2025

Camillarhi Apr 29, 2025

pinheadmz May 1, 2025

Camillarhi May 1, 2025 •

edited

Loading

macgyver13 May 1, 2025

macgyver13 Apr 29, 2025

Camillarhi Apr 29, 2025

pinheadmz May 1, 2025 •

edited

Loading

Camillarhi May 1, 2025 •

edited

Loading

pinheadmz May 1, 2025

pinheadmz May 1, 2025

Camillarhi May 1, 2025

	def get_cb_forwards(self, index):
	cmd = "wget -q -O - 127.0.0.1:9235/api/forwarding_history"
	res = self.wait_for_rpc(
	"exec_run", [index, ServiceType.CIRCUITBREAKER.value, cmd, self.network_name]
	)
	return json.loads(res)

Circuit breaker plugin #688

Are you sure you want to change the base?

Circuit breaker plugin #688

Conversation

Camillarhi commented Mar 6, 2025

pinheadmz commented Mar 7, 2025

Camillarhi commented Mar 8, 2025

willcl-ark commented Mar 9, 2025 • edited by bdp-DrahtBot Loading

Conflicts

pinheadmz left a comment

Choose a reason for hiding this comment

pinheadmz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Camillarhi commented Apr 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Camillarhi May 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinheadmz May 1, 2025 • edited Loading

Choose a reason for hiding this comment

Camillarhi May 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willcl-ark commented Mar 9, 2025 •

edited by bdp-DrahtBot

Loading

Camillarhi May 1, 2025 •

edited

Loading

pinheadmz May 1, 2025 •

edited

Loading

Camillarhi May 1, 2025 •

edited

Loading