Klaus.AI is a prototype AI that helps you learn and play new board games. Using Google Gemini, Klaus parses the PDF rules of a new game and uses a live feed from a camera and microphone to advise the players. Klaus was built as part of the Hacking Agents Hackathon NYC hosted by DataStax Langflow, DigitalOcean, Torc.dev, TwelveLabs, and Twilio.
We love learning new, complex ("crunchy") board games. However, often we spend more time learning the rules and debating the optimal strategies than actually playing. Klaus.AI aims to be your intelligent board game companion. It's designed to:
- Walk players through game setup.
- Interpret and answer questions about game rules in real-time.
- Analyze the current game state using live video and audio input.
- Provide contextual advice and strategic recommendations.
- PDF Rule Parsing: Upload the PDF rulebook for a new game, and processes it to understand the game's mechanics.
- Live Game Assistance Page: A dedicated page displays a live feed from your webcam and a log of game interactions.
- Voice-Activated Queries: Simply hold down the backtick (`) key and ask your questions.
- Visual Game State Analysis: Klaus.AI captures a screenshot of your game board when you ask a question.
- Intelligent Advice: Google Gemini analyzes the game rules, your query, and the visual game state to provide:
- Clarifications on game rules.
- Strategic recommendations on what moves to consider.
- Action Logging: Keeps a running log of user actions and AI recommendations.
- Rule Ingestion: The user uploads a PDF of the board game rules.
- Processing with Langflow & Gemini:
- Langflow is used to create a workflow that parses and processes the PDF content.
- Google Gemini assists in understanding the rules and extracting key information.
- This process generates:
- A system prompt that guides Gemini on how to act as a knowledgeable game assistant.
- A schema that identifies the game pieces and components Gemini should expect to see.
- Live Game Interface: The user is directed to the main game page, which includes:
- A live video feed from the player's webcam.
- A log to display interactions and advice.
- Player Interaction:
- The player holds the backtick (`) key to activate their microphone.
- The player speaks their query (e.g., "What can I do with this piece?", "Is this a legal move?").
- The system captures the audio and a current screenshot from the webcam.
- AI Analysis & Response:
- The captured audio (transcribed via the Web Speech API) and the game board screenshot are sent to Google Gemini.
- Crucially, the system prompt and the game piece schema (generated from the PDF) are also passed to Gemini with every query. This gives Gemini the necessary context about the specific game's rules and components.
- Gemini processes all this information to understand the query in the context of the current game state and rules.
- Klaus.AI then presents Gemini's advice or rule clarification to the player.
- Google Gemini: Used for natural language understanding, rule interpretation, visual analysis, generating game advice, and importantly, for generating the primary codebase of this project.
- Google AI Studio: The development environment where written prompts were used to instruct Gemini to write the application code and guide the integration of various services.
- Frontend: React/Tailwind/TypeScript
- Backend: Node.js, Langflow, DigitalOcean
A unique aspect of Klaus.AI's development was the extensive use of AI for code generation. The frontend (React/TypeScript) and backend (Node.js) code was primarily generated by Google Gemini through iterative written prompting in Google AI Studio. The team guided the AI in structuring the application and integrating the necessary APIs and services.
- Ryan Swift
- Jon Gottfried
- Mike Swift
- Google AI Studio (really!)
This project was built as part of the Hacking Agents Hackathon NYC (May 2025) and we thank the hosts: DataStax Langflow, DigitalOcean, Torc.dev, TwelveLabs, and Twilio.