Feature/airtable include fields

[dkindlund]

Hey @HenryHengZJ , this PR covers a bunch of quality of life improvements to the Airtable DocumentLoader node, including:

• Support for Including Only Specific Fields from the Airtable table:

  • This is huge if your source Airtable has like 100+ fields but you only want like 5 fields to actually be stored in the Vector DB.
  • Without this feature, you're forcing any Airtable users who have 100+ fields to essentially sync their 5 fields they want to store in a Vector DB to first replicate those 5 fields over to a separate, intermediate Airtable base (which gets expensive quickly).
  • Include Only Fields can be a comma-separated list of field names or field IDs. If the field name itself contains commas, then it's recommended to only use the field ID instead of the name.
  • Switched the GET to a POST for fetching from Airtable -- this is because if you want to include more than about 50 fields, you would hit the max limit of the number of characters that can be put into the URL of a GET request before Airtable throws errors. Their recommended workaround is to switch from GET to POST accordingly, so that you can specify an unlimited number of include fields -- reference: https://support.airtable.com/docs/enforcement-of-url-length-limit-for-web-api-requests

• Quality of LIfe Improvements are:

  • Checking to make sure the Base ID and Table ID are specified (at a minimum).
  • Made the Return All and Limit fields optional instead of both required. Before, this was confusing to the user because it wasn't clear WHY a user needs to specify any sort of Limit at all, if Return All is true.
  • Now, we added explicit logic such that when Return All is set to true, then the Limit field is ignored completely (previously, there was a bug where even if Return All was true it would still only process up to the number of records specified by Limit -- which was really confusing to the user).
  • Lastly, if you specify a Limit of greater than 100 records (like 264), this code will actually load all 264 records successfully (previously, there was a bug in the loadLimit() function because there was no pagination logic implemented -- so it would only load up to 100 records from Airtable -- which is the maximum number of records that can be returned from a single Airtable API call).

Hopefully, this code makes sense -- I've been using it for a couple of days now and it seems to be working for me well. Let me know if you have any questions. Thanks!

[dkindlund]

Screenshot:

[HenryHengZJ]

wow this is awesome!! thanks a lot for the improvement!

[doumlegare]

Hello,
When you use airtable into flowise you upload it for the first time into vectordb like pineconeand create an automatisation to update the file into db automaticly when a uopdate is made in airtable?
Thanks for sharing your experience

[dkindlund]

Hi @doumlegare , the DocumentLoader node doesn't support syncing Airtable record updates into any vector DB at this time. This is a known limitation within Flowise. In short, any documents you load through this automation will be added into the vector DB, always. If you had already loaded an earlier version of the document (from Airtable) into the vector DB and you re-run it again, you will end up with 2 documents in the vector DB (duplicates).

For now, you have to implement a mechanism outside of Flowise to keep track of which documents were already added to the vector DB and which documents need to be updated in the vector DB.

Long term fix is discussed here:
#1638

[doumlegare]

Thanks for your fast response @dkindlund
So how do you use Airtable as a database right now?
Do you have a flow exemple?

[dkindlund]

@doumlegare , I'm basically using it similar to how this video describes using Airtable:
https://www.youtube.com/watch?v=5U22PUI7jP0

And yeah, I'm having to implement my own tracking mechanism outside of Flowise to make sure I'm not upserting duplicate documents into the vector store DB.