Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for different Storage Backend during local development #2650

Open
AravindPrabhs opened this issue Sep 2, 2024 · 1 comment
Open
Labels
enhancement New feature or request storage

Comments

@AravindPrabhs
Copy link

Is your feature request related to a problem? Please describe.

I would like to use a S3 Compatible Storage Backend during local development and configure it using config.toml.

Describe the solution you'd like

I would like to be able to specify the following env variables to the storage api server:

STORAGE_BACKEND=s3

#######################################
# S3 Backend
#######################################
STORAGE_S3_BUCKET=supa-storage-bucket
STORAGE_S3_MAX_SOCKETS=200
STORAGE_S3_ENDPOINT=http://127.0.0.1:9230
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_REGION=local

# unsure if these are right but don't think connection is being made to minio anyway
AWS_ACCESS_KEY_ID=minioadmin 
AWS_SECRET_ACCESS_KEY=minioadmin

Though, separate I wonder if it possible to have a more general API to pass/pipe in whatever environment variables are specified in the config.toml for the particular service so it is easier to try out individual feature changes without forking the CLI.

Describe alternatives you've considered

My original problem is the file storage backend is not performant for range requests on large files please see my original discussion in the storage repo (supabase/storage#540).

@AravindPrabhs
Copy link
Author

I managed to connect the minIO docker instance by adding environment variables field to the storage struct.

// pkg/config/config.io
storage struct {
...
EnvironmentVariables map[string]string    `toml:"env_environment_variables,omitempty"`
}

and then modified start.go so the environment variables are passed to the container

for key, value := range utils.Config.Storage.EnvironmentVariables {
   envVars = append(envVars, fmt.Sprintf("%s=%s", key, value))
}

happy to put in a pull request along these lines, and would this approach be worth extending for other services ?

@sweatybridge sweatybridge added storage enhancement New feature or request labels Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request storage
Projects
None yet
Development

No branches or pull requests

2 participants