Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

Add HA configuration #23

Closed
wants to merge 3 commits into from
Closed

Add HA configuration #23

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
34fbf9b
add ha configuration
May 10, 2017
d0c094c
Merge remote-tracking branch 'upstream/master' into ha_configuration
Aug 1, 2017
ed7bb4f
fix typo
Aug 1, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions container/appmaster/runit/service/flink/run
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,37 @@ function add_kerberos_configurations() {
fi
}

function add_ha_configurations() {
if [[ "${FLINK_HA_ENABLED}" == true ]]; then
export FLINK_JAVA_OPTS="$FLINK_JAVA_OPTS -Dhigh-availability=ZOOKEEPER"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you need to set high-availability via FLINK_JAVA_OPTS as opposed to the config file?

Copy link
Contributor Author

@skonto skonto Aug 1, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I follow the pattern there. This pre-existed just used previous work, didnt refactor it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have a look at the add_if_non_empty function, it is basically a wrapper for appending to FLINK_JAVA_OPTS. For the sake of uniformity, please use it to set the high-availability flag.

I have a minor concern about the sheer number of settings being configured as dynamic properties, as opposed to being configured in flink-conf.yaml. Perhaps some of these HA settings can be set statically.

Copy link
Contributor Author

@skonto skonto Aug 2, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sure for the first option I missed that. By the way it wont work for every property, there is one that causes problems:
2017-08-02 00:00:20,277 INFO org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner - -Dhigh-availability.job.delay="10
2017-08-02 00:00:20,277 INFO org.apache.flink.mesos.runtime.clusterframework.MesosApplicationMasterRunner - s"

The property Dhigh-availability.job.delay="10 s" is not preserved when passed due to spaces.
In on order to fix this you can either pass each arg explicitly with quotes or if you want to keep a global variable you need to use eval. The first option is not handy as we have many variables and the second one is not a good choice since it opens security problems if we don't sanitize every input. It gets ugly so I will try put HA options in the file.

add_if_non_empty high-availability.cluster-id "${FLINK_HIGH_AVAILABILITY_CLUSTER_ID}"
add_if_non_empty high-availability.storageDir "${FLINK_HIGH_AVAILABILITY_STORAGE_DIR}"
add_if_non_empty high-availability.zookeeper.quorum "${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_QUORUM}"
add_if_non_empty high-availability.jobmanager.port "${FLINK_HIGH_AVAILABILITY_JOBMANAGER_PORT}"
add_if_non_empty high-availability.job.delay "${FLINK_HIGH_AVAILABILITY_JOB_DELAY}"
add_if_non_empty high-availability.zookeeper.path.root "${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_PATH_ROOT}"
# used for the client at ha services initiated by the mesos app master runner
add_if_non_empty ghigh-availability.zookeeper.client.session-timeout \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(typo) ghigh-availability

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix thnx.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

"${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_CLIENT_SESSION_TIMEOUT}"
add_if_non_empty high-availability.zookeeper.client.connection-timeout \
"${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_CLIENT_CONNECTION_TIMEOUT}"
add_if_non_empty high-availability.zookeeper.client.retry-wait \
"${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_CLIENT_RETRY_WAIT}"
add_if_non_empty high-availability.zookeeper.client.max-retry-attempts \
"${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_CLIENT_MAX_RETRY_ATTEMPTS}"
add_if_non_empty high-availability.zookeeper.path.running-registry \
"${FLINK_HIGH_AVAILABILITY_ZOOKEEPER_PATH_RUNNING_REGISTRY}"
fi
}

function update_log_level() {
if [[ "${FLINK_LOG_LEVEL}" != "" ]]; then
sed -ie 's/log4j.rootLogger=INFO, file/log4j.rootLogger='"${FLINK_LOG_LEVEL}"', file/g' ${FLINK_HOME}/conf/log4j.properties
fi
}

add_flink_configurations
add_ha_configurations
add_mesos_configurations
add_ssl_configurations
add_kerberos_configurations
Expand Down