Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQL Commiter with SCRAM-based authentication fails #64

Open
schudoku opened this issue May 23, 2022 · 1 comment
Open

SQL Commiter with SCRAM-based authentication fails #64

schudoku opened this issue May 23, 2022 · 1 comment

Comments

@schudoku
Copy link

I am using norconex filesystem crawler 2.9.1 and want to commit to a postgre database 13.4. The database requires user authentication which is SCRAM-based. In the collector config I specified the following commiter:

  <committer class="com.norconex.committer.sql.SQLCommitter">
    <driverPath>C:\dev\databases\drivers\postgresql-42.3.5.jar</driverPath>
    <driverClass>org.postgresql.Driver</driverClass>
    <connectionUrl>jdbc:postgresql://localhost:64000/databasename</connectionUrl>
    <tableName>crawler.crawl_test</tableName>
    
    <createTableSQL>
  	CREATE TABLE {tableName} (
  		{primaryKey} VARCHAR(32672) NOT NULL,
  		content CLOB,
  		PRIMARY KEY ( {primaryKey} ),
  		title   VARCHAR(256)
  		author  VARCHAR(256)
  	)
    </createTableSQL>
    
    <createFieldSQL>
  	ALTER TABLE {tableName} ADD {fieldName} VARCHAR(5000)
    </createFieldSQL>
    
    <fixFieldValues>true</fixFieldValues>
    
    <credentials>
  	<username> ---PLAIN-USERNAME-HERE--- </username>
  	<password> ---PLAIN-PW-HERE--- </password-->
    </credentials>
    
  </committer>

And this is the stacktrace I get:

[..]
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler finishing: committing documents.
INFO [AbstractFileQueueCommitter] Committing 266 files
INFO [SQLCommitter] Sending 100 commit operations to SQL database.
INFO [SQLCommitter] Checking if table "crawler.crawl_1" exists...
INFO [SQLCommitter] Table "crawler.crawl_1" does not exist. Attempting to create it...
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler executed in 4 seconds.
ERROR [JobSuite] Execution failed for job: Crawler For ProSi Reports
com.norconex.committer.core.CommitterException: Could not commit batch to database.
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:480)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.sql.SQLCommitter.commit(SQLCommitter.java:425)
at com.norconex.collector.core.crawler.AbstractCrawler.execute(AbstractCrawler.java:274)
at com.norconex.collector.core.crawler.AbstractCrawler.doExecute(AbstractCrawler.java:228)
at com.norconex.collector.core.crawler.AbstractCrawler.startExecution(AbstractCrawler.java:184)
at com.norconex.jef4.job.AbstractResumableJob.execute(AbstractResumableJob.java:49)
at com.norconex.jef4.suite.JobSuite.runJob(JobSuite.java:357)
at com.norconex.jef4.suite.JobSuite.doExecute(JobSuite.java:294)
at com.norconex.jef4.suite.JobSuite.execute(JobSuite.java:167)
at com.norconex.collector.core.AbstractCollector.start(AbstractCollector.java:150)
at com.norconex.collector.core.AbstractCollectorLauncher.launch(AbstractCollectorLauncher.java:95)
at com.norconex.collector.fs.FilesystemCollector.main(FilesystemCollector.java:76)
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2294)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2039)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:204)
at org.apache.commons.dbutils.QueryRunner.update(QueryRunner.java:419)
at com.norconex.committer.sql.SQLCommitter.ensureTable(SQLCommitter.java:607)
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:463)
... 15 more
Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided.
at org.postgresql.core.v3.ConnectionFactoryImpl.lambda$doAuthentication$4(ConnectionFactoryImpl.java:791)
at org.postgresql.core.v3.AuthenticationPluginManager.withPassword(AuthenticationPluginManager.java:81)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:788)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:223)
at org.postgresql.Driver.makeConnection(Driver.java:402)
at org.postgresql.Driver.connect(Driver.java:261)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2304)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2290)
... 21 more
INFO [JobSuite] Running Crawler For ProSi Reports: END (Mon May 23 09:41:40 CEST 2022)

It says no password provided, but I did. Could you help me?

Regards!

@essiembre
Copy link
Contributor

Not sure if you still have the issue, but it should work with PostgreSQL JDBC drivers v42.2+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants