A great Warehouse Foreman is key to good warehouse operations. In this session, we show how to build one for a Data Warehouse by leveraging dbt metadata to tailor LLMs to specific context, helping practitioners, users and analysts with business knowledge and even SQL generation. By intersecting Data Engineering best practices and Gen AI, this approach demonstrates a practical application for integrating the automated documentation of dbt with a foundation model by using creative RAG techniques and Prompt Engineering to improve an LLM’s ability to answer domain-specific questions and generate better SQL code from natural language inputs.
Codebase for the Data Makers Fest 2024 talk "Foreman: Building a tailored data assistant using dbt metadata".