Merge pull request #73 from philippgille/add-s3-export-import-example

Add example code for S3 export/import

.github/workflows/go.yml

@@ -60,3 +60,8 @@ jobs:
       run: |
         cd examples/semantic-search-arxiv-openai
         go build -v ./...
+
+    - name: S3 export/import
+      run: |
+        cd examples/s3-export-import
+        go build -v ./...

README.md

@@ -159,6 +159,7 @@ For the full interface see the Godoc: <https://pkg.go.dev/github.com/philippgill
   - [X] In-memory
   - [X] Optional immediate persistence (writes one file for each added collection and document, encoded as [gob](https://go.dev/blog/gob), optionally gzip-compressed)
   - [X] Backups: Export and import of the entire DB to/from a single file (encoded as [gob](https://go.dev/blog/gob), optionally gzip-compressed and AES-GCM encrypted)
+    - Includes methods for generic `io.Writer`/`io.Reader` so you can plug S3 buckets and other blob storage, see [examples/s3-export-import](examples/s3-export-import) for example code
 - Data types:
   - [X] Documents (text)
 

examples/README.md

@@ -13,3 +13,5 @@
    - Uses OpenAI for creating the embeddings
 4. [WebAssembly](webassembly)
    - This example shows how `chromem-go` can be compiled to WebAssembly and then used from JavaScript in a browser
+5. [S3 Export/Import](s3-export-import)
+   - This example shows how to export the DB to and import it from any S3-compatible blob storage service

examples/s3-export-import/README.md

@@ -0,0 +1,28 @@
+# S3 Export/Import
+
+This example shows how to export the DB to and import it from any S3-compatible blob storage service.
+
+- The example uses [MinIO](https://github.com/minio/minio), but any S3-compatible storage works.
+- The example uses [gocloud.dev](https://github.com/google/go-cloud) Go "Cloud Development Kit" from Google for interfacing with any S3-compatible storage, and because it provides methods for creating writers and readers that make it easy to use with `chromem-go`.
+
+## How to run
+
+1. Prepare the S3-compatible storage
+   1. `docker run -d --rm --name minio -p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 quay.io/minio/minio:RELEASE.2024-05-01T01-11-10Z server /data --console-address ":9001"`
+   2. Open the MinIO Console in your browser: <http://localhost:9001>
+   3. Log in with user `minioadmin` and password `minioadmin`
+   4. Use the web UI to create a bucket named `mybucket`
+2. Set the OpenAI API key in your env as `OPENAI_API_KEY`
+3. `go run .`
+
+You can also check <http://localhost:9001/browser/mybucket> and see the exported DB as `chromem.gob.gz`.
+
+To stop the MinIO server run `docker stop minio`.
+
+## Output
+
+```text
+2024/05/04 19:24:07 Successfully exported DB to S3 storage.
+2024/05/04 19:24:07 Imported collection with 1 documents
+2024/05/04 19:24:07 Successfully imported DB from S3 storage.
+```

examples/s3-export-import/go.mod

@@ -0,0 +1,47 @@
+module github.com/philippgille/chromem-go/examples/s3-export-import
+
+go 1.21
+
+require (
+	github.com/philippgille/chromem-go v0.0.0
+	gocloud.dev v0.37.0
+)
+
+replace github.com/philippgille/chromem-go => ./../..
+
+require (
+	github.com/aws/aws-sdk-go v1.50.36 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.25.3 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1 // indirect
+	github.com/aws/aws-sdk-go-v2/config v1.27.7 // indirect
+	github.com/aws/aws-sdk-go-v2/credentials v1.17.7 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.3 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.16.9 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.3 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.3 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.5 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.5 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.51.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.20.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.28.4 // indirect
+	github.com/aws/smithy-go v1.20.1 // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
+	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/google/wire v0.6.0 // indirect
+	github.com/googleapis/gax-go/v2 v2.12.2 // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	golang.org/x/net v0.22.0 // indirect
+	golang.org/x/sys v0.18.0 // indirect
+	golang.org/x/text v0.14.0 // indirect
+	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
+	google.golang.org/api v0.169.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240311173647-c811ad7063a7 // indirect
+	google.golang.org/grpc v1.62.1 // indirect
+	google.golang.org/protobuf v1.33.0 // indirect
+)