Stream Queues Port Range; K8's issues

[nlm4145]

Hello, we're hosting rabbitmq on our k8's cluster (utilizing the bitnami chart) and are trying to use stream queues. We are having node to node communication issues (replication) revolving around port range 6000-6500 and istio. is it possible to configure this port range? We are deploying using bitnami's chart and are opening the ports using the service.extraPorts parameter and are running into limits.

[nlm4145]

There is also a limit on open ports in a kubernetes service being 250.

https://docs.digitalocean.com/products/kubernetes/details/limits/#:~:text=The%20number%20of%20open%20ports%20allowed%20in%20a%20Kubernetes%20service%20is%20limited%20to%20250.

[nlm4145]

here are some error logs:

2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> crasher:
2024-06-05T23:49:00.140062517Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> initial call: osiris_replica_reader:init/1
2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> pid: <0.64562.0>
2024-06-05T23:49:00.140080001Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> registered_name: []
2024-06-05T23:49:00.140083027Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> exception error: no match of right hand side value {error,einval}
2024-06-05T23:49:00.140090308Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> in function osiris_replica_reader:init/1 (src/osiris_replica_reader.erl, line 122)
2024-06-05T23:49:00.140093210Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> in call from gen_server:init_it/2 (gen_server.erl, line 980)
2024-06-05T23:49:00.140096533Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> in call from gen_server:init_it/6 (gen_server.erl, line 935)
2024-06-05T23:49:00.140100481Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> ancestors: [osiris_replica_reader_sup,osiris_sup,<0.226.0>]
2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> message_queue_len: 2
2024-06-05T23:49:00.140106827Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> messages: [{tcp_closed,#Port<0.20722>},
2024-06-05T23:49:00.140109449Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> {'DOWN',#Ref<0.2605680904.2340683777.88832>,port,
2024-06-05T23:49:00.140111500Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> #Port<0.20722>,noproc}]
2024-06-05T23:49:00.140114271Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> links: [<0.231.0>]
2024-06-05T23:49:00.140117520Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> dictionary: []
2024-06-05T23:49:00.140120335Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> trap_exit: true
2024-06-05T23:49:00.140123273Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> status: running
2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> heap_size: 610
2024-06-05T23:49:00.140128331Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> stack_size: 28
2024-06-05T23:49:00.140130322Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> reductions: 830
2024-06-05T23:49:00.140132279Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0> neighbours:
2024-06-05T23:49:00.140134284Z 2024-06-05 23:49:00.139541+00:00 [error] <0.64562.0>
2024-06-05 23:49:00.142631+00:00 [warning] <0.64560.0> rabbit_stream_coordinator: Error while starting replica for __assets_1717631154451014611 on node rabbit@rabbitmq-beta-2.rabbitmq-beta-headless.rabbitmq.svc.cluster.local in 1 : {{{{badmatch,{error,einval}},[{osiris_replica_reader,init,1,[...]},{gen_server,init_it,2,...},{gen_server,init_it,...},{proc_lib,...}]},{child,undefined,#Ref<52050.826815780.2877292546.81925>,{osiris_replica_reader,start_link,...},temporary,false,...}},{gen_server,call,[<52050.24651.0>,await,infinity]}}

[nlm4145]

2024-06-05T23:49:21.940956776Z 2024-06-05 23:49:21.940763+00:00 [info] <0.27105.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:21.942189489Z 2024-06-05 23:49:21.941293+00:00 [info] <0.27105.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:21.946598440Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ** Generic server <0.27105.0> terminating
2024-06-05T23:49:21.946607076Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ** Last message in was {continue,
2024-06-05T23:49:21.946609743Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> #{name => "__assets_1717631154451014611",
2024-06-05T23:49:21.946612037Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> nodes =>
2024-06-05T23:49:21.946614861Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ['rabbit@rabbitmq-beta-0.rabbitmq-beta-headless.rabbitmq.svc.cluster.local',
2024-06-05T23:49:21.946624741Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 'rabbit@rabbitmq-beta-2.rabbitmq-beta-headless.rabbitmq.svc.cluster.local',
2024-06-05T23:49:21.946626736Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 'rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local'],
2024-06-05T23:49:21.946628724Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> reference =>
2024-06-05T23:49:21.946630436Z 2024-06-05 23:49:
21.945578+00:00 [error] <0.27105.0> {resource,<<"/">>,queue,<<"assets">>},
2024-06-05T23:49:21.946632065Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> epoch => 1,leader_pid => <15100.5351.0>,
2024-06-05T23:49:21.946634256Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> leader_node =>
2024-06-05T23:49:21.946636527Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 'rabbit@rabbitmq-beta-0.rabbitmq-beta-headless.rabbitmq.svc.cluster.local',
2024-06-05T23:49:21.946639647Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> replica_nodes =>
2024-06-05T23:49:21.946642286Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ['rabbit@rabbitmq-beta-2.rabbitmq-beta-headless.rabbitmq.svc.cluster.local',
2024-06-05T23:49:21.946645254Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 'rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local'],
2024-06-05T23:49:21.946647495Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> retention => [{max_age,86400000}],
2024-06-05T23:49:21.946650244Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> event_formatter =>
2024-06-05T23:49:21.946652005Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {rabbit_stream_queue,format_osiris_event,
2024-06-05T23:49:21.946654059Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [{resource,<<"/">>,queue,<<"assets">>}]},
2024-06-05T23:49:21.946655716Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> max_segment_size_bytes => 50000000}}
2024-06-05T23:49:21.946657499Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ** When Server state == "osiris_replica:format_status/1 crashed"
2024-06-05T23:49:21.946659224Z 2024-06-05 23:
49:21.945578+00:00 [error] <0.27105.0> ** Reason for termination ==
2024-06-05T23:49:21.946660905Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> ** {{{badmatch,{error,einval}},
2024-06-05T23:49:21.946662561Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [{osiris_replica_reader,init,1,
2024-06-05T23:49:21.946664456Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [{file,"src/osiris_replica_reader.erl"},
2024-06-05T23:49:21.946666383Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {line,122}]},
2024-06-05T23:49:21.946668025Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {gen_server,init_it,2,[{file,"gen_server.erl"},{line,980}]},
2024-06-05T23:49:21.946669616Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {gen_server,init_it,6,[{file,"gen_server.erl"},{line,935}]},
2024-06-05T23:49:21.946671247Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]},
2024-06-05T23:49:21.946672927Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {child,undefined,#Ref<0.2902212600.462159873.152119>,
2024-06-05T23:49:21.946676638Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {osiris_replica_reader,start_link,
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [#{name =>
2024-06-05T23:49:21.946679986Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> <<"__assets_1717631154451014611">>,
2024-06-05T23:49:21.946681620Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> port => 6051,
2024-06-05T23:49:21.946701392Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> hosts =>
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [{10,214,93,144},
2024-06-05T23:49:21.946707618Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> "rabbitmq-beta-1",
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> "rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local"],
2024-06-05T23:49:21.946711646Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> reference =>
2024-06-05T23:49:21.946713889Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> {resource,<<"/">>,queue,<<"assets">>},
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> leader_pid => <15100.5351.0>,
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> transport => tcp,
2024-06-05T23:49:21.946719123Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> connection_token =>
2024-06-05T23:49:21.946720791Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> <<17,6,65,252,171,197,47,23,25,183,
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 244,27,194,48,186,70,127,195,82,183,
2024-06-05T23:49:21.946724262Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 27,174,208,233,64,243,29,252,244,
2024-06-05T23:49:21.946728797Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> 192,65,113>>,
2024-06-05T23:49:21.946730843Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> replica_pid => <0.27105.0>,
2024-06-05T23:49:21.946732518Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> start_offset => {0,empty}}]},
2024-06-05T23:49:21.946734586Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> temporary,false,5000,worker,
2024-06-05T23:49:21.946736311Z 2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0> [osiris_replica_reader]}}
2024-06-05 23:49:21.945578+00:00 [error] <0.27105.0>
2024-06-05T23:49:21.948129868Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> crasher:
2024-06-05T23:49:21.948141244Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> initial call: osiris_replica:init/1
2024-06-05T23:49:21.948144844Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> pid: <0.27105.0>
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> registered_name: []
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> exception exit: {{{badmatch,{error,einval}},
2024-06-05T23:49:21.948152929Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{osiris_replica_reader,init,1,
2024-06-05T23:49:21.948160760Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{file,"src/osiris_replica_reader.erl"},
2024-06-05T23:49:21.948163702Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {line,122}]},
2024-06-05T23:49:21.948166250Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {gen_server,init_it,2,
2024-06-05T23:49:21.948168674Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{file,"gen_server.erl"},{line,980}]},
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {gen_server,init_it,6,
2024-06-05T23:49:21.948172981Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{file,"gen_server.erl"},{line,935}]},
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {proc_lib,init_p_do_apply,3,
2024-06-05T23:49:21.948177395Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{file,"proc_lib.erl"},{line,241}]}]},
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {child,undefined,#Ref<0.2902212600.462159873.152119>,
2024-06-05T23:49:21.948181851Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {osiris_replica_reader,start_link,
2024-06-05T23:49:21.948184261Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [#{name => <<"__assets_1717631154451014611">>,
2024-06-05T23:49:21.948186765Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> port => 6051,
2024-06-05T23:49:21.948189197Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> hosts =>
2024-06-05T23:49:21.948191708Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{10,214,93,144},
2024-06-05T23:49:21.948194347Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> "rabbitmq-beta-1",
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> "rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local"],
2024-06-05T23:49:21.948199648Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> reference =>
2024-06-05T23:49:21.948202111Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {resource,<<"/">>,queue,<<"assets">>},
2024-06-05T23:49:21.948204662Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> leader_pid => <15100.5351.0>,transport => tcp,
2024-06-05T23:49:21.948206995Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> connection_token =>
2024-06-05T23:49:21.948209256Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> <<17,6,65,252,171,197,47,23,25,183,244,27,
2024-06-05T23:49:21.948211620Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> 194,48,186,70,127,195,82,183,27,174,208,
2024-06-05T23:49:21.948214151Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> 233,64,243,29,252,244,192,65,113>>,
2024-06-05T23:49:21.948216503Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> replica_pid => <0.27105.0>,
2024-06-05T23:49:21.948218900Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> start_offset => {0,empty}}]},
2024-06-05T23:49:21.948221378Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> temporary,false,5000,worker,
2024-06-05T23:49:21.948228628Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [osiris_replica_reader]}}
2024-06-05T23:49:21.948231014Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> in function gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-06-05T23:49:21.948233367Z 2024-06-05 23:49:21.946545+0
0:00 [error] <0.27105.0> ancestors: [osiris_server_sup,osiris_sup,<0.230.0>]
2024-06-05T23:49:21.948235818Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> message_queue_len: 1
2024-06-05T23:49:21.948238190Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> messages: [{'$gen_call',{<15100.71251.0>,
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [alias|
2024-06-05T23:49:21.948246112Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> #Ref<15100.0.9120131.2605680904.2340749313.112412>]},
2024-06-05T23:49:21.948248548Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> await}]
2024-06-05T23:49:21.948250923Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> links: [<0.27106.0>,#Port<0.22852>,<0.234.0>]
2024-06-05T23:49:21.948253351Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
2024-06-05T23:49:21.948255699Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> bits => 58,uniform => #Fun<rand.1.65977474>,
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> uniform_n => #Fun<rand.2.65977474>,
2024-06-05T23:49:21.948260560Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> jump => #Fun<rand.3.65977474>},
2024-06-05T23:49:21.948263024Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [161240828028852230|148094100223370277]}}]
2024-06-05T23:49:21.948265428Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> trap_exit: true
2024-06-05T23:49:21.948268031Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> status: running
2024-06-05T23:49:21.948270436Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> heap_size: 28690
2024-06-05T23:49:21.948272843Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> stack_size: 28
2024-06-05T23:49:21.948275154Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> reductions: 62275
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> neighbours:
2024-06-05T23:49:21.948279820Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> neighbour:
2024-06-05T23:49:21.948282177Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> pid: <0.27106.0>
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> registered_name: []
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> initial_call: {erlang,apply,2}
2024-06-05T23:49:21.948289580Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> current_function: {prim_inet,accept0,3}
2024-06-05T23:49:21.948292047Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> ancestors: []
2024-06-05T23:49:21.948294460Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> message_queue_len: 0
2024-06-05T23:49:21.948296815Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> links: [<0.27105.0>]
2024-06-05T23:49:21.948299207Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> trap_exit: false
2024-06-05T23:49:21.948301637Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> status: waiting
2024-06-05T23:49:21.948304292Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> heap_size: 233
2024-06-05T23:49:21.948309348Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> stack_size: 12
2024-06-05T23:49:21.948311728Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> reductions: 72
2024-06-05T23:49:21.948314037Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> current_stacktrace: [{prim_inet,accept0,3,[]},
2024-06-05T23:49:21.948318723Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {inet_tcp,accept,1,[{file,"inet_tcp.erl"},{line,219}]},
2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {osiris_replica,accept,4,
2024-06-05T23:49:21.948325192Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> [{file,"src/osiris_replica.erl"},
2024-06-05T23:49:21.948327574Z 2024-06-05 23:49:21.946545+00:00 [error] <0.27105.0> {line,303}]}]
2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> supervisor: {local,osiris_server_sup}
2024-06-05T23:49:21.948509341Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> errorContext: child_terminated
2024-06-05T23:49:21.948511893Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> reason: {{{badmatch,{error,einval}},
2024-06-05T23:49:21.948514289Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [{osiris_replica_reader,init,1,
2024-06-05T23:49:21.948517059Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [{file,"src/osiris_replica_reader.erl"},{line,122}]},
2024-06-05T23:49:21.948519568Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {gen_server,init_it,2,[{file,"gen_server.erl"},{line,980}]},
2024-06-05T23:49:21.948522073Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {gen_server,init_it,6,[{file,"gen_server.erl"},{line,935}]},
2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {proc_lib,init_p_do_apply,3,
2024-06-05T23:49:21.948527603Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [{file,"proc_lib.erl"},{line,241}]}]},
2024-06-05T23:49:21.948530088Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {child,undefined,#Ref<0.2902212600.462159873.152119>,
2024-06-05T23:49:21.948532639Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {osiris_replica_reader,start_link,
2024-06-05T23:49:21.948535155Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [#{name => <<"__assets_1717631154451014611">>,
2024-06-05T23:49:21.948537517Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> port => 6051,
2024-06-05T23:49:21.948540031Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> hosts =>
2024-06-05T23:49:21.948542395Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [{10,214,93,144},
2024-06-05T23:49:21.948557647Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> "rabbitmq-beta-1",
2024-06-05T23:49:21.948561050Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> "rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local"],
2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> reference => {resource,<<"/">>,queue,<<"assets">>},
2024-06-05T23:49:21.948566070Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> leader_pid => <15100.5351.0>,transport => tcp,
2024-06-05T23:49:21.948568561Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> connection_token =>
2024-06-05T23:49:21.948570976Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> <<17,6,65,252,171,197,47,23,25,183,244,27,194,48,
2024-06-05T23:49:21.948577848Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> 186,70,127,195,82,183,27,174,208,233,64,243,29,
2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> 252,244,192,65,113>>,
2024-06-05T23:49:21.948583217Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> replica_pid => <0.27105.0>,
2024-06-05T23:49:21.948585478Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> start_offset => {0,empty}}]},
2024-06-05T23:49:21.948587848Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> temporary,false,5000,worker,
2024-06-05T23:49:21.948590254Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> [osiris_replica_reader]}}
2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> offender: [{pid,<0.27105.0>},
2024-06-05T23:49:21.948596231Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {id,"__assets_1717631154451014611"},
2024-06-05T23:49:21.948599108Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {mfargs,{osiris_replica,start_link,undefined}},
2024-06-05T23:49:21.948601367Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {restart_type,temporary},
2024-06-05T23:49:21.948603672Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {significant,false},
2024-06-05T23:49:21.948605931Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {shutdown,5000},
2024-06-05T23:49:21.948608426Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0> {child_type,worker}]
2024-06-05T23:49:21.948611040Z 2024-06-05 23:49:21.947622+00:00 [error] <0.234.0>
2024-06-05 23:49:21.957461+00:00 [info] <0.27107.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:21.958160414Z 2024-06-05 23:49:21.957978+00:00 [info] <0.27107.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05 23:49:21.964185+00:00 [error] <0.27107.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71259.0> exited with noproc
2024-06-05T23:49:21.977415859Z 2024-06-05 23:49:21.977181+00:00 [info] <0.27109.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:21.978640567Z 2024-06-05 23:49:21.977727+00:00 [info] <0.27109.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:21.984430815Z 2024-06-05 23:49:21.984245+00:00 [error] <0.27109.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71265.0> exited with noproc
2024-06-05T23:49:21.997448784Z 2024-06-05 23:49:21.997253+00:00 [info] <0.27111.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:21.998080718Z 2024-06-05 23:49:21.997898+00:00 [info] <0.27111.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:22.004881667Z 2024-06-05 23:49:22.004652+00:00 [error] <0.27111.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71271.0> exited with noproc
2024-06-05T23:49:22.017852896Z 2024-06-05 23:49:22.017662+00:00 [info] <0.27113.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:22.019030820Z 2024-06-05 23:49:22.018842+00:00 [info] <0.27113.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osir
is replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:22.026009200Z 2024-06-05 23:49:22.025816+00:00 [error] <0.27113.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71277.0> exited with noproc
2024-06-05T23:49:22.039354891Z 2024-06-05 23:49:22.039190+00:00 [info] <0.27115.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:22.040705764Z 2024-06-05 23:49:22.039640+00:00 [info] <0.27115.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:22.046620881Z 2024-06-05 23:49:22.046297+00:00 [error] <0.27115.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71283.0> exited with noproc
2024-06-05T23:49:22.059856953Z 2024-06-05 23:49:22.059631+00:00 [info] <0.27117.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:22.061215418Z 2024-06-05 23:49:22.060206+00:00 [info] <0.27117.0> __assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:22.067023194Z 2024-06-05 23:49:22.066734+00:00 [error] <0.27117.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71289.0> exited with noproc
2024-06-05T23:49:22.079742591Z 2024-06-05 23:49:22.079534+00:00 [info] <0.27119.0> Stream: __assets_1717631154451014611 will use /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-beta-1.rabbitmq-beta-headless.rabbitmq.svc.cluster.local/stream/__assets_1717631154451014611 for osiris log data directory
2024-06-05T23:49:22.081106681Z 2024-06-05 23:49:22.080091+00:00 [info] <0.27119.0> _
_assets_1717631154451014611 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}
2024-06-05T23:49:22.087155179Z 2024-06-05 23:49:22.086893+00:00 [error] <0.27119.0> __assets_1717631154451014611 [osiris_replica:handle_info/2] replica reader <15100.71295.0> exited with noproc