K8SPS-283: Add HAProxy for group replication #399

egegunes · 2023-07-04T08:13:02Z

CHANGE DESCRIPTION

Problem:
Marco doesn't like MySQL Router.

Solution:
We're allowing users to deploy HAProxy with group replication.

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are the manifests (crd/bundle) regenerated if needed?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported PS version?
Does the change support oldest and newest supported Kubernetes version?

build/haproxy-entrypoint.sh

build/haproxy_check_primary.sh

build/haproxy_check_replicas.sh

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

build/haproxy-entrypoint.sh

hors · 2023-07-27T08:26:58Z

deploy/cr.yaml

+    readinessProbe:
+      initialDelaySeconds: 30
+      timeoutSeconds: 10
+      periodSeconds: 10
+      failureThreshold: 3
+      successThreshold: 1
+
+    livenessProbe:
+      initialDelaySeconds: 15
+      timeoutSeconds: 10
+      periodSeconds: 10
+      failureThreshold: 3
+      successThreshold: 1


please comment these lines by default

hors · 2023-07-27T08:33:28Z

build/haproxy_check_replicas.sh

+	local level=$1
+	local message=$2
+
+	echo "$(/usr/bin/date +%Y-%m-%dT%H:%M:%S%z) [${level}] ${message}"


The support team wants to have JSON format like it was required in PXC. It simplifies there life when you need to parse something.

but I don't really like it since we have logs in mixed format right now:

{"time":"28/Jul/2023:06:55:50.971", "level": "INFO", "message": "10.100.1.7:33062 @@super_read_only: 0 Applier: ON"} {"time":"28/Jul/2023:06:55:50.974", "level": "INFO", "message": "10.100.1.7:33062 for backend mysql-replicas is NOT OK"} [WARNING] (19) : Server mysql-replicas/cluster1-mysql-0 is DOWN, reason: External check error, code: 1, check duration: 149ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeu ed, 0 remaining in queue. {"time":"28/Jul/2023:06:55:52.629", "level": "INFO", "message": "10.100.2.7:33062 @@super_read_only: 1 Applier: ON"} {"time":"28/Jul/2023:06:55:52.631", "level": "INFO", "message": "10.100.2.7:33062 for backend mysql-replicas is OK"} {"time":"28/Jul/2023:06:55:54.309", "level": "INFO", "message": "10.100.0.16:33062 @@super_read_only: 1 Applier: ON"} {"time":"28/Jul/2023:06:55:54.312", "level": "INFO", "message": "10.100.0.16:33062 for backend mysql-replicas is OK"}

hors · 2023-07-27T08:33:53Z

deploy/cr.yaml

+      readinessProbe:
+        timeoutSeconds: 3
+        periodSeconds: 5
+        failureThreshold: 3
+        successThreshold: 1
+
+      livenessProbe:
+        timeoutSeconds: 3
+        periodSeconds: 5
+        failureThreshold: 3
+        successThreshold: 1


please comment these lines by default

hors · 2023-07-27T14:04:52Z

build/haproxy-entrypoint.sh

 if [ "$1" = 'haproxy' ]; then
 	if [ ! -f '/etc/haproxy/mysql/haproxy.cfg' ]; then
 		cp /opt/percona/haproxy.cfg /etc/haproxy/mysql
 	fi

 	haproxy_opt='-W -db -f /opt/percona/haproxy-global.cfg -f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'
+


@egegunes please use the same approach for custom conf as we have for PXC operator
https://github.com/percona/percona-docker/blob/main/haproxy/dockerdir/entrypoint.sh#L17C5-L28

In case of custom config we need to replace whole conf

deploy/cr.yaml

build/haproxy-entrypoint.sh

build/haproxy_check_primary.sh

build/haproxy_check_replicas.sh

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions · 2023-07-28T07:03:14Z

build/haproxy_check_primary.sh

+    local level=$1
+    local message=$2
+    local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")


[shfmt] _{reported by reviewdog 🐶}

Suggested change

local level=$1

local message=$2

local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")

local level=$1

local message=$2

local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")

github-actions · 2023-07-28T07:03:15Z

build/haproxy_check_primary.sh

+    local message=$2
+    local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")
+
+    echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"


[shfmt] _{reported by reviewdog 🐶}

Suggested change

echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"

echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions · 2023-07-28T07:03:26Z

build/haproxy_check_replicas.sh

+    local level=$1
+    local message=$2
+    local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")


[shfmt] _{reported by reviewdog 🐶}

Suggested change

local level=$1

local message=$2

local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")

local level=$1

local message=$2

local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")

github-actions · 2023-07-28T07:03:26Z

build/haproxy_check_replicas.sh

+    local message=$2
+    local date=$(/usr/bin/date +"%d/%b/%Y:%H:%M:%S.%3N")
+
+    echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"


[shfmt] _{reported by reviewdog 🐶}

Suggested change

echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"

echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"

github-actions · 2023-07-28T16:10:51Z

build/haproxy-entrypoint.sh

+  if [ ! -f '/etc/haproxy/mysql/haproxy.cfg' ]; then
+    cp /opt/percona/haproxy.cfg /etc/haproxy/mysql
+  fi
+
+  custom_conf='/etc/haproxy-custom/haproxy.cfg'
+  if [ -f "$custom_conf" ]; then
+    log "haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg"
+    haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg || EC=$?
+    if [ -n "$EC" ]; then
+      log "The custom config $custom_conf is not valid and will be ignored."
+    fi
+  fi
+
+  haproxy_opt='-W -db '
+  if [ -f "$custom_conf" -a -z "$EC" ]; then
+    haproxy_opt+="-f $custom_conf "
+  else
+    haproxy_opt+='-f /opt/percona/haproxy-global.cfg '
+  fi
+
+  haproxy_opt+='-f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'

-	haproxy_opt='-W -db -f /opt/percona/haproxy-global.cfg -f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'
+  if [ -f '/etc/haproxy/config/haproxy.cfg' ]; then
+    haproxy_opt="${haproxy_opt} -f /etc/haproxy/config/haproxy.cfg"
+  fi


[shfmt] _{reported by reviewdog 🐶}

Suggested change

if [ ! -f '/etc/haproxy/mysql/haproxy.cfg' ]; then

cp /opt/percona/haproxy.cfg /etc/haproxy/mysql

fi

custom_conf='/etc/haproxy-custom/haproxy.cfg'

if [ -f "$custom_conf" ]; then

log "haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg"

haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg || EC=$?

if [ -n "$EC" ]; then

log "The custom config $custom_conf is not valid and will be ignored."

fi

fi

haproxy_opt='-W -db '

if [ -f "$custom_conf" -a -z "$EC" ]; then

haproxy_opt+="-f $custom_conf "

else

haproxy_opt+='-f /opt/percona/haproxy-global.cfg '

fi

haproxy_opt+='-f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'

haproxy_opt='-W -db -f /opt/percona/haproxy-global.cfg -f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'

if [ -f '/etc/haproxy/config/haproxy.cfg' ]; then

haproxy_opt="${haproxy_opt} -f /etc/haproxy/config/haproxy.cfg"

fi

if [ ! -f '/etc/haproxy/mysql/haproxy.cfg' ]; then

cp /opt/percona/haproxy.cfg /etc/haproxy/mysql

fi

custom_conf='/etc/haproxy-custom/haproxy.cfg'

if [ -f "$custom_conf" ]; then

log "haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg"

haproxy -c -f $custom_conf -f /etc/haproxy/mysql/haproxy.cfg || EC=$?

if [ -n "$EC" ]; then

log "The custom config $custom_conf is not valid and will be ignored."

fi

fi

haproxy_opt='-W -db '

if [ -f "$custom_conf" -a -z "$EC" ]; then

haproxy_opt+="-f $custom_conf "

else

haproxy_opt+='-f /opt/percona/haproxy-global.cfg '

fi

haproxy_opt+='-f /etc/haproxy/mysql/haproxy.cfg -p /etc/haproxy/mysql/haproxy.pid -S /etc/haproxy/mysql/haproxy-main.sock'

if [ -f '/etc/haproxy/config/haproxy.cfg' ]; then

haproxy_opt="${haproxy_opt} -f /etc/haproxy/config/haproxy.cfg"

fi

JNKPercona · 2023-07-31T08:49:11Z

Test name	Status
async-ignore-annotations	passed
auto-config	passed
config	passed
config-router	passed
demand-backup	passed
gr-bootstrap	passed
gr-demand-backup	passed
gr-haproxy	passed
gr-ignore-annotations	passed
gr-init-deploy	passed
gr-one-pod	passed
gr-scaling	passed
gr-tls-cert-manager	passed
haproxy	passed
init-deploy	passed
limits	passed
monitoring	passed
one-pod	passed
scaling	passed
service-per-pod	passed
sidecars	passed
tls-cert-manager	passed
users	passed
version-service	passed
We run 24 out of 24

commit: 8b99311
image: perconalab/percona-server-mysql-operator:PR-399-8b99311

pull-request-size bot added the size/L 100-499 lines label Jul 4, 2023

github-actions bot reviewed Jul 4, 2023

View reviewed changes

egegunes force-pushed the K8SPS-283 branch from c43d319 to aeee50f Compare July 5, 2023 08:58

egegunes added 2 commits July 5, 2023 12:01

K8SPS-283: Add HAProxy for group replication

01a3032

fix tests

16cca14

egegunes force-pushed the K8SPS-283 branch from aeee50f to 16cca14 Compare July 5, 2023 09:01

egegunes and others added 3 commits July 5, 2023 12:03

Apply suggestions from code review

458111b

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

fix endpoint if router and haproxy is both disabled

c7f3887

fix tests

9626151

egegunes marked this pull request as ready for review July 7, 2023 06:52

egegunes requested review from tplavcic, nmarukovich, hors, inelpandzic and pooknull as code owners July 7, 2023 06:52

inelpandzic previously approved these changes Jul 10, 2023

View reviewed changes

egegunes added 4 commits July 11, 2023 10:08

don't use ERROR prefix for haproxy checks

6c6038f

enable mysql admin port

5c64874

don't allow enabling proxies at the same time

935c370

cleanup proxy resources after it's disabled

6f1295a

egegunes dismissed inelpandzic’s stale review via 6545f6c July 11, 2023 09:25

pull-request-size bot added size/XL 500-999 lines and removed size/L 100-499 lines labels Jul 11, 2023

add gr-haproxy test

58d1a31

egegunes force-pushed the K8SPS-283 branch from 6545f6c to 58d1a31 Compare July 11, 2023 11:06

egegunes added 5 commits July 12, 2023 09:14

fix tests

b093191

Merge branch 'main' into K8SPS-283

3793730

fix haproxy checks

275ec01

fix tests

12764c1

fix gr-haproxy

988ce81

inelpandzic previously approved these changes Jul 26, 2023

View reviewed changes

egegunes added 3 commits July 27, 2023 09:59

add custom config for haproxy

c4a916e

configure probes for haproxy

9ac46e5

allow configuring env variables for components

0c704be

egegunes dismissed inelpandzic’s stale review via 0c704be July 27, 2023 07:42

pull-request-size bot added size/XXL 1000+ lines and removed size/XL 500-999 lines labels Jul 27, 2023

github-actions bot reviewed Jul 27, 2023

View reviewed changes

build/haproxy-entrypoint.sh Show resolved Hide resolved

hors requested changes Jul 27, 2023

View reviewed changes

address review comments

140e60a

github-actions bot reviewed Jul 28, 2023

View reviewed changes

egegunes and others added 2 commits July 28, 2023 10:02

Apply suggestions from code review

5acfa6e

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Apply suggestions from code review

070a3bd

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Jul 28, 2023

View reviewed changes

Apply suggestions from code review

5f7b5be

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Jul 28, 2023

View reviewed changes

percona deleted a comment from github-actions bot Jul 28, 2023

hors added 2 commits July 28, 2023 19:10

update k8s to 1.24

d15103c

Merge branch 'main' into K8SPS-283

c376372

github-actions bot reviewed Jul 28, 2023

View reviewed changes

fix haproxy entrypoint

8b99311

hors self-requested a review July 31, 2023 08:50

hors approved these changes Jul 31, 2023

View reviewed changes

hors merged commit 2ea88df into main Jul 31, 2023
8 of 10 checks passed

hors deleted the K8SPS-283 branch July 31, 2023 08:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K8SPS-283: Add HAProxy for group replication #399

K8SPS-283: Add HAProxy for group replication #399

egegunes commented Jul 4, 2023 •

edited

Loading

hors Jul 27, 2023

egegunes Jul 28, 2023

hors Jul 27, 2023

egegunes Jul 28, 2023

egegunes Jul 28, 2023

hors Jul 27, 2023

egegunes Jul 28, 2023

hors Jul 27, 2023

egegunes Jul 28, 2023

github-actions bot Jul 28, 2023

github-actions bot Jul 28, 2023

github-actions bot Jul 28, 2023

github-actions bot Jul 28, 2023

github-actions bot Jul 28, 2023

JNKPercona commented Jul 31, 2023

	echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"
	echo "{\"time\":\"${date}\", \"level\": \"${level}\", \"message\": \"${message}\"}"

K8SPS-283: Add HAProxy for group replication #399

K8SPS-283: Add HAProxy for group replication #399

Conversation

egegunes commented Jul 4, 2023 • edited Loading

CHANGE DESCRIPTION

CHECKLIST

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Jul 28, 2023

Choose a reason for hiding this comment

github-actions bot Jul 28, 2023

Choose a reason for hiding this comment

github-actions bot Jul 28, 2023

Choose a reason for hiding this comment

github-actions bot Jul 28, 2023

Choose a reason for hiding this comment

github-actions bot Jul 28, 2023

Choose a reason for hiding this comment

JNKPercona commented Jul 31, 2023

egegunes commented Jul 4, 2023 •

edited

Loading