Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor to uset Structured Output Framework method #90

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

jimthompson5802
Copy link
Owner

Closes #89

Successful run with automated tester

time src/agent/app_embedvec_tester.py --puzzle_setup_fp data/automated_test_set_0.jsonl 
Running Connection Solver Agent Tester 0.15.0-dev

>>>>SOLVING PUZZLE 1
Setting up Puzzle Words: ['nets', 'return', 'heat', 'jazz', 'mom', 'shift', 'kayak', 'option', 'rain', 'sleet', 'level', 'racecar', 'bucks', 'tab', 'hail', 'snow']

ENTERED SETUP_PUZZLE

Generating vocabulary and embeddings for the words...this may take several seconds 

Generating embeddings for the definitions

Storing vocabulary and embeddings in external database

ENTERED EMBEDVEC_RECOMMENDER
found count: 0, mistake_count: 0
words_remaining: ['nets', 'return', 'heat', 'jazz', 'mom', 'shift', 'kayak', 'option', 'rain', 'sleet', 'level', 'racecar', 'bucks', 'tab', 'hail', 'snow']
(101, 101)
(101, 101)
candidate_lists size: 73

EMBEDVEC_RECOMMENDER: RECOMMENDED WORDS ['hail', 'rain', 'sleet', 'snow'] with connection Weather-related precipitation forms.
Weather-related precipitation forms. ~ wet weather: ['hail', 'rain', 'sleet', 'snow'] == ['hail', 'rain', 'sleet', 'snow']
Recommendation ['hail', 'rain', 'sleet', 'snow'] is correct

ENTERED EMBEDVEC_RECOMMENDER
found count: 1, mistake_count: 0
words_remaining: ['nets', 'return', 'heat', 'jazz', 'mom', 'shift', 'kayak', 'option', 'level', 'racecar', 'bucks', 'tab']
(75, 75)
(75, 75)
candidate_lists size: 49

EMBEDVEC_RECOMMENDER: RECOMMENDED WORDS ['bucks', 'heat', 'jazz', 'shift'] with connection The words are related to actions that involve sudden movements or excitement.
Recommendation ['bucks', 'heat', 'jazz', 'shift'] is incorrect, one away from correct

ENTERED ONE-AWAY ANALYZER
found count: 1, mistake_count: 1

>>>Number of single topic groups: 2
More than one single-topic group recommendations, selecting one at random.

>>>Selected single-topic group:
Recommended Group: ('bucks', 'jazz', 'shift')
Connection Description: The three words 'bucks', 'jazz', and 'shift' can all be related to the single topic of basketball. 'Bucks' refers to the Milwaukee Bucks, an NBA team. 'Jazz' refers to the Utah Jazz, another NBA team. 'Shift' can refer to a change in strategy, lineup, or play during a basketball game. Therefore, the context of basketball encompasses all three words.

>>>One-away group recommendations:
one_away_group_recommendation is a new recommendation
using one_away_group_recommendation

EMBEDVEC_RECOMMENDER: RECOMMENDED WORDS ['bucks', 'jazz', 'return', 'shift'] with connection The common connection among the anchor words 'bucks', 'jazz', and 'shift' is that they all have meanings related to change or movement. 'Bucks' can refer to a sudden jolt or movement (as in a horse bucking), 'jazz' can imply improvisation or a shift in musical rhythm, and 'shift' directly relates to change or movement in position or direction. Among the candidate words, 'return' is most connected to this theme, as it implies a shift or change back to a previous state or position.
Recommendation ['bucks', 'jazz', 'return', 'shift'] is incorrect
Changing the recommender from 'embedvec_recommender' to 'llm_recommender'

ENTERED LLM_RECOMMENDER
found count: 1, mistake_count: 2
attempt_count: 1
words_remaining: ['tab', 'bucks', 'racecar', 'level', 'option', 'kayak', 'shift', 'mom', 'jazz', 'heat', 'return', 'nets']

LLM_RECOMMENDER: RECOMMENDED WORDS ['kayak', 'level', 'mom', 'racecar'] with connection palindromes
palindromes ~ palindromes: ['kayak', 'level', 'mom', 'racecar'] == ['kayak', 'level', 'mom', 'racecar']
Recommendation ['kayak', 'level', 'mom', 'racecar'] is correct

ENTERED LLM_RECOMMENDER
found count: 2, mistake_count: 2
attempt_count: 1
words_remaining: ['shift', 'option', 'return', 'nets', 'heat', 'jazz', 'tab', 'bucks']

LLM_RECOMMENDER: RECOMMENDED WORDS ['option', 'return', 'shift', 'tab'] with connection keyboard keys
keyboard keys ~ keyboard keys: ['option', 'return', 'shift', 'tab'] == ['option', 'return', 'shift', 'tab']
Recommendation ['option', 'return', 'shift', 'tab'] is correct

ENTERED LLM_RECOMMENDER
found count: 3, mistake_count: 2
attempt_count: 1
words_remaining: ['jazz', 'bucks', 'nets', 'heat']

LLM_RECOMMENDER: RECOMMENDED WORDS ['bucks', 'heat', 'jazz', 'nets'] with connection NBA basketball teams
NBA basketball teams ~ nba teams: ['bucks', 'heat', 'jazz', 'nets'] == ['bucks', 'heat', 'jazz', 'nets']
Recommendation ['bucks', 'heat', 'jazz', 'nets'] is correct
SOLVED THE CONNECTION PUZZLE!!!


FINAL PUZZLE STATE:
{'current_tool': 'llm_recommender',
 'found_count': 4,
 'invalid_connections': [['9f6b9eccfa3e67bcf843952fc50cd9e5',
                          ['bucks', 'heat', 'jazz', 'shift']],
                         ['9ef03eb4eb521c3a6d6864a8544581b8',
                          ['bucks', 'jazz', 'shift', 'return']]],
 'llm_retry_count': 0,
 'llm_temperature': 0.7,
 'mistake_count': 2,
 'puzzle_status': 'initialized',
 'recommendation_answer_status': 'correct',
 'recommendation_correct_groups': [['hail', 'rain', 'sleet', 'snow'],
                                   ['kayak', 'level', 'mom', 'racecar'],
                                   ['option', 'return', 'shift', 'tab'],
                                   ['bucks', 'heat', 'jazz', 'nets']],
 'recommendation_count': 6,
 'recommended_connection': 'NBA basketball teams',
 'recommended_correct': True,
 'recommended_words': ['bucks', 'heat', 'jazz', 'nets'],
 'tool_status': 'puzzle_completed',
 'tool_to_use': 'END',
 'vocabulary_db_fp': '/tmp/tmp7l50yz3u.db',
 'words_remaining': []}

FOUND SOLUTIONS
[   ['hail', 'rain', 'sleet', 'snow'],
    ['kayak', 'level', 'mom', 'racecar'],
    ['option', 'return', 'shift', 'tab'],
    ['bucks', 'heat', 'jazz', 'nets']]
ALL GROUPS FOUND
[   [   ['hail', 'rain', 'sleet', 'snow'],
        ['kayak', 'level', 'mom', 'racecar'],
        ['option', 'return', 'shift', 'tab'],
        ['bucks', 'heat', 'jazz', 'nets']]]
   solved_puzzle  number_found                                       groups_found
0           True             4  [[hail, rain, sleet, snow], [kayak, level, mom...

real    0m29.272s
user    0m6.479s
sys     0m1.336s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement LLM JSON data outputs to Structure Output framework
1 participant