Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

[improve][functions]18532: JavaInstanceStarter support reading configs from file #5181

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

imaffe
Copy link

@imaffe imaffe commented Nov 27, 2022

Fixes apache#18532

Master Issue: apache#18532

Motivation

Reading config options from command line is not very flexible in a cloud-native environment. For example, in distroless containers where there is no shell access, it's hard to determine a config value from env vars.

Using file to read the configs allow a more flexible configuration pattern.

Modifications

  1. Added JavaInstanceConfiguration extends PulsarConfiguration to support reading configs from file.
  2. Declare all JCommander fields as non-required
  3. Add custom required field validation and default value logic.
  4. Added test case to make sure the command line args is not broken.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

  • Added JavaInstanceStarterTest to make sure using command line args does not break.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@imaffe imaffe changed the title 18532: support reading configs from file [improve][functions]18532: JavaInstanceStarter support reading configs from file Nov 27, 2022
@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch from 54f6865 to 8b018f6 Compare November 27, 2022 04:25
Comment on lines 168 to 169
@Parameter(names = "--config_file", description = "The config file for instance to use, default "
+ "use coomand line args")
public String configFile = null;

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. this option should be exclusive to other previous configs. User either set it or set others, but not both at the same time.
  2. there should be a manual check that all previously required fields are present (either in the configfile or provided by user.)

Copy link
Author

@imaffe imaffe Nov 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The two config scheme is not exclusive to each other. The users can specify part of the configs using command line and the rest using a config file, but the command line args has a higher priority than the file configs. .
  2. The previously required fields are checked in validateRequiredConfigs();, it will be called after we read config values from command line and config files (if any)

listConverter = StringConverter.class)
public String jarFile;
public String jarFile = null;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String default value is null already, no need to explicit set it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I kept them explicitly null is because : I was not sure how will the JCommander set the default value for String (JVM sets default value to null, but not sure if JCommander follows the rule). I did an experiement that JCommader initialize the value to null as well , but I figured it might be more clear that we explicitly set the value to null to avoid ambiguity (JCommander documentation didn't mention what the default value for string is if not provided one). The null value here serves a purpose : to distinguish between if a value is set or not provided. But sure since we know the value will be set to null even we do not do so explicitly, we can simplify the statments~

Comment on lines 111 to 112
public Boolean useTls = null;

@Parameter(names = "--tls_allow_insecure", description = "Allow insecure tls connection\n")
public String tlsAllowInsecureConnection = Boolean.TRUE.toString();
@Parameter(names = "--tls_allow_insecure", description = "Allow insecure tls connection\n", arity = 1)
public Boolean tlsAllowInsecureConnection = null;

@Parameter(names = "--hostname_verification_enabled", description = "Enable hostname verification")
public String tlsHostNameVerificationEnabled = Boolean.FALSE.toString();
@Parameter(names = "--hostname_verification_enabled", description = "Enable hostname verification", arity = 1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should set a meaningful value for Boolean type variables (either true or false instead of null)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default value is set in another method useDefaultValueIfBothFileAndJCommanderNotProvided. Using null because I want to distinguish between "not provided" and "provided a value but value is false"

@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch 2 times, most recently from dd74939 to 93ab5ee Compare December 26, 2022 16:27
@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch 2 times, most recently from 3a05b0e to 4cda499 Compare January 3, 2023 05:33
@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch 7 times, most recently from a7b1ac8 to 6cece3e Compare February 2, 2023 14:24
fix checkstyle

not using * import

1. trigger CI

try do not change the vairable type

skip file

ignore unit test

do not use = null and skip set default value

default value should be null

uncomment file reader

make file config and commandline config exclusive

change the configuration logic

remove TODO
@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch from 6cece3e to b351e0e Compare February 2, 2023 14:29
@imaffe imaffe force-pushed the affe/use-file-to-read-properties branch from 250c41d to 39d63e7 Compare February 3, 2023 05:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Support reading config options from file in Function Java Runner (JavaInstanceStarter)
2 participants