Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for boolean type conversion #78

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

akurdyukov
Copy link

Fix for #77

Copy link
Owner

@bryzgaloff bryzgaloff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @akurdyukov and thank you for your contribution! Please excuse me for a late reply.

I have added several comments. However, I believe the solution to this should be more fundamental: we should hard-code clickhouse_driver.Client's signature into the plugin's code, so that the code fully controls all of the arguments (in contrast to a pretty faulty **connection_kwargs approach) and converts them to proper types. This will make the behaviour more predictable.

In other words, I believe there should be a preliminary PR which replaces Client(**…) with Client(host=…, …) following its signature from clickhouse-driver here:

return clickhouse_driver.Client(**conn_to_kwargs(conn, self._database))

And then the PR which you currently work on will easily implement handling this specific str-to-bool case.

Comment on lines 100 to 103
if val in ('y', 'yes', 't', 'true', 'on', '1'):
return 1
elif val in ('n', 'no', 'f', 'false', 'off', '0'):
return 0
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do these lists of values come from?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a generic list of possible bool values people use

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite biased. Instead let's support only true/false options. It is impossible to guess what a random user may consider to be a true/false str representation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, only 'true', 'True', 'fase', 'False' supported

src/airflow_clickhouse_plugin/hooks/clickhouse.py Outdated Show resolved Hide resolved
tests/unit/hooks/test_clickhouse.py Outdated Show resolved Hide resolved
src/airflow_clickhouse_plugin/hooks/clickhouse.py Outdated Show resolved Hide resolved
src/airflow_clickhouse_plugin/hooks/clickhouse.py Outdated Show resolved Hide resolved
src/airflow_clickhouse_plugin/hooks/clickhouse.py Outdated Show resolved Hide resolved
src/airflow_clickhouse_plugin/hooks/clickhouse.py Outdated Show resolved Hide resolved
@akurdyukov
Copy link
Author

Thanks for the review! I fixed most of review comments.

Regarding the first one about the method of passing arguments to clickhouse_driver.Client - currently there's 23 arguments, most of them are optionals. So, minimal boilerplate version should use something like Pydantic. And it looks like a little overkill to me. What do you think?

else:
raise ValueError("invalid truth value %r" % (val,))
raise ValueError(f'invalid truth value {str_value!r}')
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else is redundant here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, fixed

Comment on lines 7 to 17
def test_correct_true(self):
self.assertTrue(strtobool('true'))

def test_correct_one(self):
self.assertTrue(strtobool('1'))

def test_correct_false(self):
self.assertFalse(strtobool('false'))

def test_correct_zero(self):
self.assertFalse(strtobool('0'))
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A quick best practice comment: the tests must cover all the supported input values, not only a few. A full coverage is required. However, this is a boilerplate code: you may use self.subTest functionality to check all truthy and all falsy values in a loop instead of creating a test per value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, fixed

else:
raise ValueError("invalid truth value %r" % (val,))
raise ValueError(f'invalid truth value {str_value!r}')
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why "truth" value btw? What if it was intended to be a falsy one? :)

Suggested change
raise ValueError(f'invalid truth value {str_value!r}')
raise ValueError(f'unsupported value: {str_value!r}')

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@bryzgaloff
Copy link
Owner

Regarding the first one about the method of passing arguments to clickhouse_driver.Client - currently there's 23 arguments, most of them are optionals. So, minimal boilerplate version should use something like Pydantic. And it looks like a little overkill to me. What do you think?

Hi @akurdyukov, yes, definitely using an external library is an overkill here. I suggest to hardcode all the arguments: clickhouse-driver is not expected to change them often, so we may accept new PRs once the arguments change in the underlying library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants