Add streaming support for openai v0 (#951)

* Mark instrumentation points for SDK (#1009) * Mark instrumentation points for SDK * Remove duplicated assertion * Fixup: assert attribute not function --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Prefix conversation id with llm (#1012) * Change conversation_id->llm.conversation_id * Fixup formatting * Add support for Meta Llama2. (#1010) * Add support for Llama2. * Fixup: lint errors * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: hmstepanek <[email protected]> * Add bedrock feedback into preview (#1030) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <[email protected]> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <[email protected]> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <[email protected]> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Apply suggestions from code review * Remove unused import * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <[email protected]> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <[email protected]> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <[email protected]> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> * Initial feedback commit for botocore * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <[email protected]> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <[email protected]> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <[email protected]> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Bedrock feedback w/ testing for titan and jurassic models * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <[email protected]> * Fix merge conflicts * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek <[email protected]> * Add to and move feedback tests * Handle 0.32.0.post1 version in tests (#963) * Remove response_id dependency in bedrock * Change API name * Update moto * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Change ids to match other tests * move message_ids declaration outside for loop * Add comment to tox.ini * Drop py27 from memcache testing. * Drop pypy27 from memcache testing. * Update flaskrestx testing #1004 * Remove tastypie 0.14.3 testing * Remove tastypie 0.14.3 testing * Remove python 3.12 support (for now) * Remove untouched files from diff list --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Timothy Pansino <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Fix instrumentation for openai 1.8.0 * Add LLM attribute to transactions. (#1050) * Add LLM attr to transactions. * Remove newlines. * Add llm attribute to transaction event default attrs list. * Linting. * Remove imports for moto on py37. (#1053) * Update botocore tests. (#1051) * Update moto decorators in tests. * Remove py27 botocore dependencies. * Drop testing for Python 3.7 * Add support for streaming in openai * Add support for streaming errors * Support async generators * Add support for error during streaming * Ignore v1 tests in v0 * Refactor generator wrappers --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: hmstepanek <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Timothy Pansino <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]>
newrelic · Feb 23, 2024 · ce31bc7 · ce31bc7
1 parent 25da462
commit ce31bc7
Show file tree

Hide file tree

Showing 8 changed files with 2,024 additions and 180 deletions.
diff --git a/newrelic/config.py b/newrelic/config.py
@@ -2062,6 +2062,12 @@ def _process_module_builtin_defaults():
         "newrelic.hooks.mlmodel_openai",
         "instrument_openai_util",
     )
+    _process_module_definition(
+        "openai.api_resources.abstract.engine_api_resource",
+        "newrelic.hooks.mlmodel_openai",
+        "instrument_openai_api_resources_abstract_engine_api_resource",
+    )
+
     _process_module_definition(
         "openai.resources.chat.completions",
         "newrelic.hooks.mlmodel_openai",

diff --git a/newrelic/hooks/mlmodel_openai.py b/newrelic/hooks/mlmodel_openai.py
diff --git a/tests/external_botocore/test_botocore_sqs.py b/tests/external_botocore/test_botocore_sqs.py
@@ -30,7 +30,6 @@
 BOTOCORE_VERSION = get_package_version_tuple("botocore")
 
 url = "sqs.us-east-1.amazonaws.com"
-
 if BOTOCORE_VERSION < (1, 29, 0):
     url = "queue.amazonaws.com"
 

diff --git a/tests/mlmodel_openai/_mock_external_openai_server.py b/tests/mlmodel_openai/_mock_external_openai_server.py
@@ -30,6 +30,325 @@
 #    created by an external call.
 # 3) This app runs on a separate thread meaning it won't block the test app.
 
+STREAMED_RESPONSES = {
+    "Stream parsing error.": [
+        {
+            "Content-Type": "text/event-stream",
+            "openai-model": "gpt-3.5-turbo-0613",
+            "openai-organization": "new-relic-nkmd8b",
+            "openai-processing-ms": "516",
+            "openai-version": "2020-10-01",
+            "x-ratelimit-limit-requests": "200",
+            "x-ratelimit-limit-tokens": "40000",
+            "x-ratelimit-remaining-requests": "199",
+            "x-ratelimit-remaining-tokens": "39940",
+            "x-ratelimit-reset-requests": "7m12s",
+            "x-ratelimit-reset-tokens": "90ms",
+            "x-request-id": "49dbbffbd3c3f4612aa48def69059ccd",
+        },
+        200,
+        [
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [
+                    {"index": 0, "delta": {"role": "assistant", "content": ""}, "logprobs": None, "finish_reason": None}
+                ],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "212"}, "logprobs": None, "finish_reason": None}],
+            },
+        ],
+    ],
+    "Invalid API key.": [
+        {"Content-Type": "application/json; charset=utf-8", "x-request-id": "4f8f61a7d0401e42a6760ea2ca2049f6"},
+        401,
+        {
+            "error": {
+                "message": "Incorrect API key provided: DEADBEEF. You can find your API key at https://platform.openai.com/account/api-keys.",
+                "type": "invalid_request_error",
+                "param": None,
+                "code": "invalid_api_key",
+            }
+        },
+    ],
+    "Model does not exist.": [
+        {
+            "Content-Type": "application/json; charset=utf-8",
+            "x-request-id": "cfdf51fb795362ae578c12a21796262c",
+        },
+        404,
+        {
+            "error": {
+                "message": "The model `does-not-exist` does not exist",
+                "type": "invalid_request_error",
+                "param": None,
+                "code": "model_not_found",
+            }
+        },
+    ],
+    "You are a scientist.": [
+        {
+            "Content-Type": "text/event-stream",
+            "openai-model": "gpt-3.5-turbo-0613",
+            "openai-organization": "new-relic-nkmd8b",
+            "openai-processing-ms": "516",
+            "openai-version": "2020-10-01",
+            "x-ratelimit-limit-requests": "200",
+            "x-ratelimit-limit-tokens": "40000",
+            "x-ratelimit-remaining-requests": "199",
+            "x-ratelimit-remaining-tokens": "39940",
+            "x-ratelimit-reset-requests": "7m12s",
+            "x-ratelimit-reset-tokens": "90ms",
+            "x-request-id": "49dbbffbd3c3f4612aa48def69059ccd",
+        },
+        200,
+        [
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [
+                    {"index": 0, "delta": {"role": "assistant", "content": ""}, "logprobs": None, "finish_reason": None}
+                ],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "212"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " degrees"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " Fahrenheit"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " is"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " equal"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " to"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " "}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "100"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " degrees"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " Celsius"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "."}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {}, "logprobs": None, "finish_reason": "stop"}],
+            },
+        ],
+    ],
+}
+
+RESPONSES_V1 = {
+    "You are a scientist.": [
+        {
+            "Content-Type": "text/event-stream",
+            "openai-model": "gpt-3.5-turbo-0613",
+            "openai-organization": "foobar-jtbczk",
+            "openai-processing-ms": "516",
+            "openai-version": "2020-10-01",
+            "x-ratelimit-limit-requests": "200",
+            "x-ratelimit-limit-tokens": "40000",
+            "x-ratelimit-remaining-requests": "196",
+            "x-ratelimit-remaining-tokens": "39880",
+            "x-ratelimit-reset-requests": "23m5.129s",
+            "x-ratelimit-reset-tokens": "180ms",
+            "x-request-id": "5c53c9b80af57a1c9b38568f01dcde7f",
+        },
+        200,
+        [
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [
+                    {"index": 0, "delta": {"role": "assistant", "content": ""}, "logprobs": None, "finish_reason": None}
+                ],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "212"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " degrees"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " Fahrenheit"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " is"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " equal"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " to"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " "}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "100"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " degrees"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": " Celsius"}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {"content": "."}, "logprobs": None, "finish_reason": None}],
+            },
+            {
+                "id": "chatcmpl-87sb95K4EF2nuJRcTs43Tm9ntTemv",
+                "object": "chat.completion.chunk",
+                "created": 1706565311,
+                "model": "gpt-3.5-turbo-0613",
+                "system_fingerprint": None,
+                "choices": [{"index": 0, "delta": {}, "logprobs": None, "finish_reason": "stop"}],
+            },
+        ],
+    ]
+}
 RESPONSES_V1 = {
     "You are a scientist.": [
         {
@@ -380,7 +699,7 @@ def simple_get(openai_version, extract_shortened_prompt):
     def _simple_get(self):
         content_len = int(self.headers.get("content-length"))
         content = json.loads(self.rfile.read(content_len).decode("utf-8"))
-
+        stream = content.get("stream", False)
         prompt = extract_shortened_prompt(content)
         if not prompt:
             self.send_response(500)
@@ -392,6 +711,8 @@ def _simple_get(self):
 
         if openai_version < (1, 0):
             mocked_responses = RESPONSES
+            if stream:
+                mocked_responses = STREAMED_RESPONSES
         else:
             mocked_responses = RESPONSES_V1
 
@@ -414,7 +735,16 @@ def _simple_get(self):
         self.end_headers()
 
         # Send response body
-        self.wfile.write(json.dumps(response).encode("utf-8"))
+        if stream and status_code < 400:
+            for resp in response:
+                data = json.dumps(resp).encode("utf-8")
+                if prompt == "Stream parsing error.":
+                    # Force a parsing error by writing an invalid streamed response.
+                    self.wfile.write(b"data: %s" % data)
+                else:
+                    self.wfile.write(b"data: %s\n\n" % data)
+        else:
+            self.wfile.write(json.dumps(response).encode("utf-8"))
         return
 
     return _simple_get