[BugFix] support to run fe-proxy under customize cluster domain instead of using default one: cluster.local

[yandongxiao]

Description

StarRocks/starrocks#49984

We expose a parameter for operator to allow users to specify the dns domain suffix in k8s.

verify the modification:

1. sample configuration for nginx.conf

pid   /tmp/nginx.pid;
worker_processes 4;
include /usr/share/nginx/modules/*.conf;
events {
  worker_connections 256;
}

http {
  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;
  client_max_body_size 0;
  ignore_invalid_headers off;
  underscores_in_headers on;
  proxy_read_timeout 600s;
  proxy_http_version 1.1;

  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  default_type        application/octet-stream;

  server {
    listen 8080;
    resolver kube-dns.kube-system.svc.cluster.ydx valid=10s;
    proxy_intercept_errors on;
    recursive_error_pages on;

    location /nginx/health {
      access_log off;
      return 200;
    }

    location / {
      # see https://serverfault.com/questions/240476/how-to-force-nginx-to-resolve-dns-of-a-dynamic-hostname-everytime-when-doing-p/593003#593003 for why we use set
      set $fe_service "http://kube-starrocks-fe-service.default.svc.cluster.ydx:8030";
      proxy_pass $fe_service;
      proxy_set_header Expect $http_expect;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      error_page 307 = @handle_redirect;
    }

    location /api/transaction/load {
      set $fe_service "http://kube-starrocks-fe-service.default.svc.cluster.ydx:8030";
      proxy_pass $fe_service;
      proxy_pass_request_body off;
      proxy_set_header Expect $http_expect;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      error_page 307 = @handle_redirect;
    }

    location ~ ^/api/.*/.*/_stream_load$ {
      set $fe_service "http://kube-starrocks-fe-service.default.svc.cluster.ydx:8030";
      proxy_pass $fe_service;
      proxy_pass_request_body off;
      proxy_set_header Expect $http_expect;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      error_page 307 = @handle_redirect;
    }

    location @handle_redirect {
      if ($upstream_http_location ~ "kube-starrocks-fe-search") {
        rewrite ^ /_redirect_to_fe last;
      }
      if ($upstream_http_location !~ "kube-starrocks-fe-search") {
        rewrite ^ /_redirect_to_others last;
      }
    }

    location /_redirect_to_fe {
      set $redirect_uri '$upstream_http_location';
      proxy_pass $redirect_uri;
      proxy_set_header Expect $http_expect;
      proxy_pass_request_body off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      error_page 307 = @handle_redirect;
    }

    location /_redirect_to_others {
      set $redirect_uri '$upstream_http_location';
      proxy_pass $redirect_uri;
      proxy_set_header Expect $http_expect;
      proxy_pass_request_body on;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      error_page 307 = @handle_redirect;
    }
  }
}

2. execute stream load request.

root@kube-starrocks-be-0:/opt/starrocks# curl --location-trusted -u root: -H "label:custom_label212" -H "Expect:100-continue" -H "column_separator:," -H "columns: col1, col2" -T aaa.data -XPUT http://kube-starrocks-fe-proxy-service:8080/api/control_db/test_table/_stream_load
{
    "TxnId": 6,
    "Label": "custom_label212",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 4,
    "NumberLoadedRows": 4,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 38,
    "LoadTimeMs": 206,
    "BeginTxnTimeMs": 9,
    "StreamLoadPlanTimeMs": 121,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 52,
    "CommitAndPublishTimeMs": 21
}

Checklist

For operator, please complete the following checklist:

• run make generate to generate the code.
• run golangci-lint run to check the code style.
• run make test to run UT.
• run make manifests to update the yaml files of CRD.