-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dask-SQL not wrapping partition value in quotes #179
Comments
Thanks for the bug report @gallamine - that definitely is a bug. |
@nils-braun I'm interested. Trying to think of an elegant way of knowing whether the partition value should be enclosed in quotations or not. Is there a good way you know of for determining what the type of |
@gallamine is it a problem to enclose all partition values in quotes? In our setup, it looks like if a partition has an Only putting quotes around strings that contain at least one non-digit, such as what is in #180 would be problematic for my case. There exists a table that has a partition value |
Hi @Robert-Christensen-visa , @gallamine - that would be very nice and would simplify our lives a lot! Sorry @gallamine for not continuing the discussion on the PRs and here. Can we pick this up again? |
@Robert-Christensen-visa or @gallamine would it be possible to build a small reproducer ? I'm having some trouble generating the data partitioned in the way you've described |
Hi @quasiben. Here is a short script that will do something that causes an error for me: HDFS_BASE_PATH="..." # place the path to put the table
hdfs dfs -mkdir ${HDFS_BASE_PATH}/month01
hdfs dfs -mkdir ${HDFS_BASE_PATH}/month02
echo "hello world,1234" | hdfs dfs -put - ${HDFS_BASE_PATH}/data0001
echo "foo bar,4321" | hdfs dfs -put - ${HDFS_BASE_PATH}/data0001
hive -e "CREATE EXTERNAL TABLE partitiontest (name STRING, number INT) \
PARTITIONED BY (month STRING) \
ROW FORMAT DELIMITED \
FIELDS TERMINATED BY ',' \
LINES TERMINATED BY '\n' \
LOCATION ${HDFS_BASE_PATH}
hive -e "ALTER TABLE partitiontest ADD PARTITION (month='01') LOCATION ${HDFS_BASE_PATH}/month01"
hive -e "ALTER TABLE partitiontest ADD PARTITION (month='02') LOCATION ${HDFS_BASE_PATH}/month02" When I create a table like the one above and try to query it using from dask_sql import Context
c = Context()
c.create_table("partitiontest", get_hive_connection().cursor(), hive_table_name="partitiontest") The error returned is long, but here is a small portion that would be most relevant here:
When it collects information about the partitions it will find two partitions, |
Hello, I am querying a Hive table that is partitioned using string values that look like a date -
YYYY-MM-DD
. Mycreate_table
command fails with aHiveSQLException
because the-
isn't being escaped.For example:
this should instead read
partition_col="2019-01-21"
.I am using
dask-sql==0.3.6
The text was updated successfully, but these errors were encountered: