Ensure CURLOPT_CAINFO correctly configured for each use of libcurl

[alarocca-apixio]

The certFileLocations are currently hardcoded:

uc_catalog/src/uc_api.cpp

Line 20 in 8547541

static string certFileLocations[] = {

DuckDB has a config for ca_cert_file. Can we use this or some new config to set a custom location? My environment does not have a certFile in any of the existing locations and I don't have write access to the directories referenced:

• /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
• /etc/pki/tls/certs/ca-bundle.crt
• /etc/ssl/ca-bundle.pem
• /etc/ssl/cert.pem
• /etc/ssl/certs/ca-certificates.crt

As there are no matching files in my environment, I get the following error:

IOException: IO Error: Curl Request to '{UC_ENDPOINT}/api/2.1/unity-catalog/schemas?catalog_name={CATALOG_NAME}' failed with error: 'Problem with the SSL CA cert (path? access rights?)'

[alarocca-apixio]

I was able to try again in an environment that does in fact have "/etc/ssl/certs/ca-certificates.crt", but it still isn't working, perhaps this is related to #3?

I was unable to try the solution mentioned in that Issue as I don't have write access to /etc/pki/tls/certs

[samansmink]

@alarocca-apixio Thanks for reporting, would you mind sharing what distribution of Linux you are on? Also could you confirm you are on the latest version of uc_catalog by running UPDATE EXTENSIONS in duckdb and sending me the output of that?

[alarocca-apixio]

I am on Ubuntu 22.04.4 LTS

UPDATE EXTENSIONS output: [('delta', 'core', 'NO_UPDATE_AVAILABLE', 'f71402e', 'f71402e'), ('uc_catalog', 'core_nightly', 'UPDATED', '8547541', '1c359eb'), ('httpfs', 'core', 'NO_UPDATE_AVAILABLE', 'v1.1.3', 'v1.1.3')]

I am using pip install duckdb==1.1.3

I wondered if perhaps I didn't get the certificate related change that was pushed yesterday. It looks like there may be an update, so I added the UPDATE EXTENSIONS to my code:

con = duckdb.connect(":memory:")
con.execute(f"""
  INSTALL uc_catalog from core_nightly;
  INSTALL delta from core;
  UPDATE EXTENSIONS;
  LOAD uc_catalog;
  LOAD delta;
  CREATE SECRET (
    TYPE UC,
    TOKEN '{UC_TOKEN}',
    ENDPOINT '{UC_ENDPOINT}',
    AWS_REGION '{UC_AWS_REGION}'
  );
  ATTACH '{CATALOG_NAME}' AS {CATALOG_NAME} (TYPE UC_CATALOG);
""")
con.execute("SHOW ALL TABLES;")

Now I'm getting a new error: NotImplementedException: Not implemented Error: Tried to fallback to unknown type for 'varchar(128)'

So, I think the original error on this Issue has actually been resolved. I haven't yet looked into the new error.

[alarocca-apixio]

I saw elsewhere that the first request always failed, so I added this:

try:
  con.execute("SHOW ALL TABLES;")
except Exception as e:
  print(e)
tables = con.execute("SHOW ALL TABLES").fetchall()
print(f"Found {len(tables)} tables")

And that worked for listing the tables!

Not implemented Error: Tried to fallback to unknown type for 'void' # From the first attempt)
Found 4663 tables # Success!

However, the curl error seems to re-appear when trying to query a table, this time with a different endpoint:
IOException: IO Error: Curl Request to '{UC_ENDPOINT}/api/2.1/unity-catalog/temporary-table-credentials' failed with error: 'Problem with the SSL CA cert (path? access rights?)'

So, maybe the original issue is still relevant? Perhaps the curl library isn't being consistently configured with CURLOPT_CAINFO?

Edit: I looked at the code and it looks like the Cert location isn't being updated in GetCredentialsRequest()