Module for sending datasets to Google cloud #18
Conversation
josephchimebuka
changed the title
|
@akiraonstarknet and @ManvithaMolakala Can you pls review |
| @@ -0,0 +1,86 @@ | |||
| import logging | |||
There was a problem hiding this comment.
Please add comments throughtout.
src/data/datasets.py
Outdated
| from google.cloud import bigquery | ||
| from google.cloud import storage | ||
|
|
||
| import yaml |
There was a problem hiding this comment.
Please update pyproject.toml file
tests/test_datasets.py
Outdated
| @@ -0,0 +1,30 @@ | |||
| import unittest | |||
There was a problem hiding this comment.
In this project, we use pytest. Please write the code using pytest for uniformity
tests/test_datasets.py
Outdated
| @@ -0,0 +1,30 @@ | |||
| import unittest | |||
| from unittest.mock import Mock, patch | |||
There was a problem hiding this comment.
Explicitly mention in the README or comments if the script depends on gcloud auth for credentials or requires a GOOGLE_APPLICATION_CREDENTIALS environment variable.
There was a problem hiding this comment.
If additional configurations are needed (e.g., a secrets file or service account JSON), document their usage and location.
tests/test_datasets.py
Outdated
| # Assert no exceptions raised (would require more detailed mocks for deeper validation) | ||
|
|
||
| if __name__ == '__main__': | ||
| unittest.main() |
There was a problem hiding this comment.
Please address the above comments before I take a final look.
…ate, delete, and list functionalities
…ate, delete, and list functionalities
Closes #17
Created a bucket to monitors a Cloud Storage for file uploads or changes.
Wrote a function for uploading the data
Uploaded a file and tested it on big query to see the datasets
