Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebSocket connection terminates unexpectedly during conversation in Python SDK demo #434

Open
jcassady opened this issue Dec 13, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@jcassady
Copy link

jcassady commented Dec 13, 2024

Description

When running the ElevenLabs Python SDK Conversational AI demo on a Raspberry Pi 5, the WebSocket connection terminates unexpectedly after a few conversation exchanges. The agent successfully responds to the first two conversation exchanges, but during the third exchange, the voice agent's reply is cut off mid-response, followed by a WebSocket connection closure error.

Steps to reproduce

  1. Set up environment variables (AGENT_ID and ELEVENLABS_API_KEY)
  2. Run the demo script from elevenlabs-examples repository
  3. Begin conversation with the agent
  4. After approximately 2-3 exchanges, the connection fails

What happened instead

The WebSocket connection closes unexpectedly with a timeout error (0.5 seconds) in the websocket receive operation, terminating the conversation prematurely:

Exception in thread Thread-1 (_run):
...
  File ".../elevenlabs/conversational_ai/conversation.py", line 152, in _run
    message = json.loads(ws.recv(timeout=0.5))
                         ^^^^^^^^^^^^^^^^^^^^
...

See additional context for the complete console log.

Relevant parameters or configurations

  • Running on Raspberry Pi 5
  • OS: Debian GNU/Linux 12 (bookworm)
  • Kernel: 6.6.62+rpt-rpi-2712
  • Architecture: aarch64
  • Audio devices:
    • USB2.0 Device (card 2) # speaker
    • CMTECK (card 3) # mic

Code example

poetry run python convai/demo.py

joelcassady@jcpi:~/git/elevenlabs-examples/examples/conversational-ai/python $ poetry run python convai/demo.py 
ALSA lib pcm_asym.c:105:(_snd_pcm_asym_open) capture slave is not defined
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.front.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM front
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround51.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround21
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround51.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround21
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround40.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround40
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround51.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround41
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround51.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround50
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround51.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround51
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.surround71.0:CARD=0'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.iec958.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM iec958
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.iec958.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM spdif
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.iec958.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM spdif
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
ALSA lib pcm_oss.c:397:(_snd_pcm_oss_open) Cannot open device /dev/dsp
ALSA lib pcm_oss.c:397:(_snd_pcm_oss_open) Cannot open device /dev/dsp
ALSA lib pcm_a52.c:1001:(_snd_pcm_a52_open) a52 is only for playback
ALSA lib confmisc.c:1369:(snd_func_refer) Unable to find definition 'cards.0.pcm.iec958.0:CARD=0,AES0=6,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5180:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5703:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2666:(snd_pcm_open_noupdate) Unknown PCM iec958:{AES0 0x6 AES1 0x82 AES2 0x0 AES3 0x2 CARD 0}
ALSA lib confmisc.c:160:(snd_config_get_card) Invalid field card
ALSA lib pcm_usb_stream.c:482:(_snd_pcm_usb_stream_open) Invalid card 'card'
ALSA lib confmisc.c:160:(snd_config_get_card) Invalid field card
ALSA lib pcm_usb_stream.c:482:(_snd_pcm_usb_stream_open) Invalid card 'card'
ALSA lib pcm_dmix.c:999:(snd_pcm_dmix_open) unable to open slave
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock

Exception in thread Thread-1 (_run):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/elevenlabs/conversational_ai/conversation.py", line 152, in _run
    message = json.loads(ws.recv(timeout=0.5))
                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/websockets/sync/connection.py", line 211, in recv
    raise self.protocol.close_exc from self.recv_exc
websockets.exceptions.ConnectionClosedError: no close frame received or sent
Conversation ID: xb6l1EECD2wF3GsFthfF
Traceback (most recent call last):
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/elevenlabs/conversational_ai/default_audio_interface.py", line 82, in _in_callback
    self.input_callback(in_data)
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/elevenlabs/conversational_ai/conversation.py", line 141, in input_callback
    ws.send(
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/websockets/sync/connection.py", line 297, in send
    with self.send_context():
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/joelcassady/.cache/pypoetry/virtualenvs/convai-mHPLvgrG-py3.11/lib/python3.11/site-packages/websockets/sync/connection.py", line 745, in send_context
    raise self.protocol.close_exc from original_exc
websockets.exceptions.ConnectionClosedError: no close frame received or sent
^C^C

Additional context

Error Pattern

  • Behavior: Connection consistently fails after 2-3 successful exchanges
  • Impact: Makes the demo unusable for extended conversations as it requires restart after each connection drop

Error Details

The error occurs in two stages:

  1. Initial failure in the receive operation with 0.5s timeout
  2. Subsequent failure when trying to send data over the closed connection
@jcassady jcassady added the bug Something isn't working label Dec 13, 2024
@jcassady
Copy link
Author

jcassady commented Dec 18, 2024

The issue reported here may be related to conversation limit on free-tier accounts. In some cases, conversations are also being unexpectedly cut off even before reaching this limit. To address this, you can update the demo.py script with additional error handling that gracefully manages these situations.

The core issue appears to stem from unhandled ConnectionClosedError exceptions, particularly when the session exceeds its duration or encounters a WebSocket issue.

Here’s a simplified version of the script to illustrate a potential workaround:

import os
from websockets.exceptions import ConnectionClosedError
from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface

def main():
    AGENT_ID = os.environ.get('AGENT_ID')
    API_KEY = os.environ.get('ELEVENLABS_API_KEY')

    if not AGENT_ID:
        print("AGENT_ID environment variable must be set")
        return

    client = ElevenLabs(api_key=API_KEY)
    conversation = None

    try:
        # Initialize conversation
        conversation = Conversation(
            client,
            AGENT_ID,
            requires_auth=bool(API_KEY),
            audio_interface=DefaultAudioInterface()
        )

        # Start the session
        print("Starting conversation session...")
        conversation.start_session()
        conversation_id = conversation.wait_for_session_end()
        print(f"Conversation ended. ID: {conversation_id}")

    except ConnectionClosedError as e:
        error_msg = str(e).lower()
        if "exceeded maximum duration" in error_msg:
            print("Session duration exceeded. Please restart the session.")
        else:
            print(f"WebSocket error: {e}")

    except Exception as e:
        print(f"Unexpected error occurred: {e}")

    finally:
        # Ensure session cleanup
        if conversation:
            try:
                conversation.end_session()
                print("Session ended successfully.")
            except Exception as cleanup_error:
                print(f"Error during cleanup: {cleanup_error}")

if __name__ == "__main__":
    main()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant