added the aws and gcp connectors #187

Ris-code · 2024-08-22T17:42:28Z

Fix for - #173

KaranrajM · 2024-08-26T05:12:40Z

Hi @Ris-code can you add these env variables in docker-compose.yaml under language service? Also can you please add these necessary GCP and AWS packages to pyproject.toml file by using poetry add command.

I hope the packages are tested with keys and its working. Rest of the code looks good.

Ris-code · 2024-08-26T06:22:54Z

Hi @Ris-code can you add these env variables in docker-compose.yaml under language service? Also can you please add these necessary GCP and AWS packages to pyproject.toml file by using poetry add command.

I hope the packages are tested with keys and its working. Rest of the code looks good.

Ok, will do it by today.

…mpose.yml and added the required packages in pyproject.toml

Ris-code · 2024-08-26T13:59:32Z

Hi @KaranrajM , I have added the env variables as well as updated the pyproject.toml . Please review it once.

KaranrajM · 2024-08-28T05:16:27Z

Hi @Ris-code I have left some new reviews for the latest commit. Please take a look and make the required changes when you can.

Ris-code · 2024-08-28T05:19:35Z

Hi @Ris-code I have left some new reviews for the latest commit. Please take a look and make the required changes when you can.

Sure.

Ris-code · 2024-08-28T06:17:48Z

Hi @Ris-code I have left some new reviews for the latest commit. Please take a look and make the required changes when you can.

I could not find any review. Could you please assist me where it is?

KaranrajM · 2024-08-28T06:21:00Z

Hi @Ris-code I have left some new reviews for the latest commit. Please take a look and make the required changes when you can.

I could not find any review. Could you please assist me where it is?

You can see them at the top when you open this PR conversation. Let me tag you in those comments.

sameersegal · 2024-08-28T14:52:53Z

docker-compose.yml

@@ -34,7 +34,7 @@ services:
      volumes:
        - "kafka_data:/bitnami"
        - ./scripts/create-all-topics.sh:/usr/bin/create-topics.sh
-      command: "sh -c './run.sh & /usr/bin/create-topics.sh && tail -f /dev/null'"
+      command: "sh -c './run.sh && tail -f /dev/null'"


Why did we remove the call to create-topic.sh? It creates KAFKA topics if they haven't been created before. It's an important step in "single-command docker run", which makes it easy for first time users.

sameersegal · 2024-08-28T14:55:32Z

@shreypandey please review the changes here

language/src/speech_processor.py

KaranrajM · 2024-08-27T05:07:12Z

docker-compose.yml

@@ -34,7 +34,7 @@ services:
      volumes:
        - "kafka_data:/bitnami"
        - ./scripts/create-all-topics.sh:/usr/bin/create-topics.sh
-      command: "sh -c './run.sh & /usr/bin/create-topics.sh && tail -f /dev/null'"


Can u revert this change as it seems to ur local change and not needed for this PR?

KaranrajM · 2024-08-27T05:08:35Z

.env-dev.template

+# AWS_DEFAULT_REGION=
+
+# Create the S3 bucket
+# BUCKET_NAME=


can we change BUCKET_NAME to S3_BUCKET_NAME to be more specific everywhere?

language/pyproject.toml

KaranrajM · 2024-08-27T05:11:33Z

docker-compose.yml

+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
+      - BUCKET_NAME=${BUCKET_NAME}


Same BUCKET_NAME to S3_BUCKET_NAME

… reverted the changes in docker-compose

Ris-code · 2024-08-28T17:55:58Z

@KaranrajM I have done the changes. Please review them once.

VedantKhairnar · 2024-09-01T19:29:11Z

Hello @Ris-code
Will you pls check the PR-Issue linking guidelines and make the changes accordingly in the PR description(need to mention the Issue number).
This helps in efficient progress tracking. Thanks.

KaranrajM · 2024-09-03T06:28:29Z

.env.template

+# Set AWS keys if using AWS instances
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_DEFAULT_REGION=


Can you please put these three keys below the azure speech keys (i.e move these keys above)? Also comment before them saying these are Azure speech/translation keys.

Should I also shift the google application credential below the AWS ?

GCP credentials are used for both storage and language purposes right? It can be there. Also have you confirmed whether the packages are working?

Yes, I have tested each and every package and they are working fine.

@KaranrajM I have shifted the AWS credentials above. Please review it.

Hello @KaranrajM
Can we pls get the PR reviewed? We need it to be merged by 15th Sept for it to be considered of our ongoing challenge. Thanks.

Yes sure @VedantKhairnar Will review and try to merge it by today if everything looks good.

KaranrajM

Hi @Ris-code I have reviewed the code and some cosmetic changes need to happen. Once these are done, I can merge it for sure.

KaranrajM · 2024-09-13T05:06:07Z

.env-dev.template

@@ -38,6 +43,12 @@ PUBLIC_URL_PREFIX= # Set Tunnel URL if using local storage
 # AZURE_STORAGE_ACCOUNT_KEY=
 # AZURE_STORAGE_CONTAINER=

+# Create the S3 bucket


Can u please change this sentence to "Set S3 Bucket name when using AWS speech services" in both .env.template and .env-dev.template files? Also you can move this S3_BUCKET_NAME to below AWS Speech/Translation Keys as they are only used there and nowhere else.

KaranrajM · 2024-09-13T05:07:37Z

docker-compose.yml

@@ -78,6 +78,11 @@ services:
      - AZURE_STORAGE_ACCOUNT_KEY=${AZURE_STORAGE_ACCOUNT_KEY}
      - AZURE_STORAGE_CONTAINER=${AZURE_STORAGE_CONTAINER}
      - PUBLIC_URL_PREFIX=${PUBLIC_URL_PREFIX}
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}


Can you move the three AWS speech/translation credentials along S3_BUCKET_NAME up to above STORAGE_TYPE=${STORAGE_TYPE} line?

KaranrajM · 2024-09-13T05:08:09Z

language/src/speech_processor.py

+import requests
+from botocore.exceptions import BotoCoreError, ClientError
+import asyncio
+import requests


Remove one requests import as there are two present.

KaranrajM · 2024-09-13T05:09:34Z

language/src/speech_processor.py

@@ -391,27 +398,198 @@ async def text_to_speech(
        # )
        return new_audio_content



Add a new line here as whenever a new class is present, two new lines should be there.

KaranrajM · 2024-09-13T05:28:21Z

language/src/speech_processor.py


-        # raise ExceptionGroup("CompositeSpeechProcessor text to speech failed", excs)
+        raise ExceptionGroup("CompositeSpeechProcessor text to speech failed", excs)


Can you please add a new line at the end of the file?

KaranrajM · 2024-09-13T05:37:03Z

language/src/translator.py

+            source_language=source_language_code,
+            target_language=destination_language_code,
+        )
+        return response['translatedText']



Add a new line before every class to make it 2 lines in total.

KaranrajM · 2024-09-13T05:40:39Z

docker-compose.yml

+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
+      - S3_BUCKET_NAME=${S3_BUCKET_NAME}
+      - GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}


We may need to mount GOOGLE_APPLICATION_CREDENTIALS json file during runtime as local files cannot be accessed in a docker env without embedding them in docker image or mounting them during runtime. As the GCP creds file is highly sensitive, we can safely mount them during runtime. We can try something like this

environment:
- GOOGLE_APPLICATION_CREDENTIALS=/app/credentials/google-credentials.json
volumes:
- ${GOOGLE_APPLICATION_CREDENTIALS}:/app/credentials/google-credentials.json:ro

Let me know what you think and any other approach is also welcome. Once this change is made, please test them after building the language image and running the services through it.

Ris-code · 2024-09-14T21:58:32Z

@KaranrajM I have done all the mentioned changes and tested after mounting the GCP credentials during runtime. Please review all the changes.

KaranrajM · 2024-10-10T09:06:12Z

Hi @Ris-code I have tried to build a docker-image of language using this code, it throws an error when GOOGLE_APPLICATION_CREDENTIALS is empty in .env-dev. Can you handle this case as well as people mostly use either of the three services (Azure or AWS or GCP) not all of them? So we need to handle the case of gcp not being used then the volume should not be mounted in docker-compose.yml if the GOOGLE_APPLICATION_CREDENTIALS is not set. Rest of the code looks good. Please fix the docker-compose.yml change so this code can be merged. Let me know if you need any help.

added the aws and gcp connectors

4f3ec60

added the GCP and AWS env variables in language services of docker-co…

9b6211f

…mpose.yml and added the required packages in pyproject.toml

sameersegal reviewed Aug 28, 2024

View reviewed changes

KaranrajM reviewed Aug 28, 2024

View reviewed changes

added more indian languages for AWS and GCP, updated the poetry.lock,…

1209f33

… reverted the changes in docker-compose

KaranrajM reviewed Sep 3, 2024

View reviewed changes

shifted the aws credentials to key section in .env.template

8bc8984

KaranrajM reviewed Sep 13, 2024

View reviewed changes

made the cosmetic changes and mounted the gcp credentials during runtime

9e15e64

Lekhanrao linked an issue Sep 17, 2024 that may be closed by this pull request

Issue: Support for cloud speech & translation connectors #173

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added the aws and gcp connectors #187

added the aws and gcp connectors #187

Ris-code commented Aug 22, 2024 •

edited

Loading

KaranrajM commented Aug 26, 2024

Ris-code commented Aug 26, 2024

Ris-code commented Aug 26, 2024

KaranrajM commented Aug 28, 2024

Ris-code commented Aug 28, 2024

Ris-code commented Aug 28, 2024

KaranrajM commented Aug 28, 2024

sameersegal Aug 28, 2024 •

edited

Loading

sameersegal commented Aug 28, 2024

KaranrajM Aug 27, 2024

KaranrajM Aug 28, 2024

KaranrajM Aug 27, 2024

KaranrajM Aug 28, 2024

KaranrajM Aug 27, 2024

KaranrajM Aug 28, 2024

Ris-code commented Aug 28, 2024

VedantKhairnar commented Sep 1, 2024

KaranrajM Sep 3, 2024

Ris-code Sep 3, 2024

KaranrajM Sep 4, 2024

Ris-code Sep 5, 2024

Ris-code Sep 5, 2024

VedantKhairnar Sep 12, 2024

KaranrajM Sep 12, 2024

KaranrajM left a comment

KaranrajM Sep 13, 2024

KaranrajM Sep 13, 2024 •

edited

Loading

KaranrajM Sep 13, 2024

KaranrajM Sep 13, 2024

KaranrajM Sep 13, 2024

KaranrajM Sep 13, 2024

KaranrajM Sep 13, 2024 •

edited

Loading

Ris-code commented Sep 14, 2024

KaranrajM commented Oct 10, 2024 •

edited

Loading

		@@ -391,27 +398,198 @@ async def text_to_speech(
		# )
		return new_audio_content


		# raise ExceptionGroup("CompositeSpeechProcessor text to speech failed", excs)
		raise ExceptionGroup("CompositeSpeechProcessor text to speech failed", excs)

added the aws and gcp connectors #187

Are you sure you want to change the base?

added the aws and gcp connectors #187

Conversation

Ris-code commented Aug 22, 2024 • edited Loading

KaranrajM commented Aug 26, 2024

Ris-code commented Aug 26, 2024

Ris-code commented Aug 26, 2024

KaranrajM commented Aug 28, 2024

Ris-code commented Aug 28, 2024

Ris-code commented Aug 28, 2024

KaranrajM commented Aug 28, 2024

sameersegal Aug 28, 2024 • edited Loading

Choose a reason for hiding this comment

sameersegal commented Aug 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ris-code commented Aug 28, 2024

VedantKhairnar commented Sep 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KaranrajM left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KaranrajM Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KaranrajM Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Ris-code commented Sep 14, 2024

KaranrajM commented Oct 10, 2024 • edited Loading

Ris-code commented Aug 22, 2024 •

edited

Loading

sameersegal Aug 28, 2024 •

edited

Loading

KaranrajM Sep 13, 2024 •

edited

Loading

KaranrajM Sep 13, 2024 •

edited

Loading

KaranrajM commented Oct 10, 2024 •

edited

Loading