Feat/s3 compatible api (#387)

* feat: accesss key * feat: access-key module & access-key cmds * fix: remove redunt error condition * opt: handle wrapper error and remove unused bucket type define in access-key module * feat: add daemon check before execute accesskey commands * optmize: access-key store prefix * optmize: not found error * feat: add s3 signature * chore: * chore: * chore: * chore: * chore: * feat: add store * chore: * feat: s3 access-key, server, handlers, statestore, filestore * chore: * optmize: code structure * style: s3 code structure * style: code structure * optmize: code structure * feat: add multiple context lock * feat: check auth * chore: * chore: * feat: add bucket service * chore: * mod: update bucket lock * chore: * chore: * chore: * del s3d * chore: s3 req & rsp structure * chore: * feat: add pubBucket api * feat: add more bucket api * chore: * chore: * feat: add request and response * chore: * feat: server build * chore: check acl * chore * chore: adjust bucket url * � * feat: add auth middleware * feat: adjust code structure * optmize: code structure & auth bug * feat: put object * chore: * mod: mod bucket parse req * optmize: adjust place of response error * chore: mig sig 01 * chore: clear sig * chore: mig sig 02 * chore: mig sig 03 * chore: mig sig 04 * optmize: refractor codes * optmize: rename auth to sign * optmize: code structure & h.name * fix: h.name * feat: put-object * feat: multipart * fix: nslock key * chore: rename s3 constructor file name * fix: tidy example go-ipfs-as-a-library go mod * chore: change default s3 server address to local * mod: add object api * feat: s3-compatible-api - 1. add start option and configure; 2. optmize providers interfaces and implements; 3. rewrite the server construct function * merge: object * chore: add object lock * chore: of delete objs * chore: * fix: list objects bug * chore: rename ListObjetV1Handler to ListObjectHandler, rename BTFS-Hash to CID * refractor: bucket service * refactor: object service * refractor: refract object service * refractor: handlers * refractor: bucket handler * refractor: bucket handler * refractor: response * refractor: response func * refractor: response * refractor: object * refractor: object * refractor: objects * refractor: btf api add timeout & add cid refs to enable referred cid can not be deleted * ref: fix delete object remove body * ref: format code * fix: routers * fix: add cors header * fix: router option * feat: add delete objects handler * refractor: multipart * ref: multipart * fix: multipart etag calculation * chore: add min part size todo * chore: upgrade 'github.com/anacrolix/torrent' from v1.47.0 to v1.52.5 * opt: comment and amz header * opt: code * opt: preflight cache max age * feat: bucket response add acl header * opt: change cid-list header to cid * fix: required check exlude unknow location * fix: get object acl * ref: requests * ref: complete refractor * fix: args parse * fix: get object unlock * fix: object acl writer * fix: delete objects error * fix: Sign handler name * fix: object name escape * fix: copy source validate * opt: s3 log * opt: s3 api log * fix: allow Cache-Control header in PutObject and CopyObject Action * opt: add access-key command taglines * opt: add accesskey command description * chore: add accesskey commands test path * chore: add accesskey test path * feat: add multibase commands (#354) * fix: return an explicit error instead of panic and optimize the error log (#339) * feat: accesss key * feat: access-key module & access-key cmds * fix: remove redunt error condition * opt: handle wrapper error and remove unused bucket type define in access-key module * fix: return an explicit error instead of panic and optimize the error log (#339) (#341) * feat: add multibase commands * feat: add daemon check before execute accesskey commands * optmize: access-key store prefix * optmize: not found error * feat: add s3 signature * chore: * chore: * chore: * chore: * chore: * feat: add store * chore: * feat: s3 access-key, server, handlers, statestore, filestore * chore: * optmize: code structure * style: s3 code structure * style: code structure * optmize: code structure * feat: add multiple context lock * feat: check auth * chore: * chore: * feat: add bucket service * chore: * mod: update bucket lock * chore: * chore: * chore: * del s3d * chore: s3 req & rsp structure * chore: * feat: add pubBucket api * feat: add more bucket api * chore: * chore: * feat: add request and response * chore: * feat: server build * chore: check acl * chore * chore: adjust bucket url * � * feat: add auth middleware * feat: adjust code structure * optmize: code structure & auth bug * feat: put object * chore: * mod: mod bucket parse req * optmize: adjust place of response error * chore: mig sig 01 * chore: clear sig * chore: mig sig 02 * chore: mig sig 03 * chore: mig sig 04 * optmize: refractor codes * optmize: rename auth to sign * optmize: code structure & h.name * fix: h.name * feat: put-object * feat: multipart * fix: nslock key * chore: rename s3 constructor file name * fix: tidy example go-ipfs-as-a-library go mod * chore: change default s3 server address to local * mod: add object api * feat: s3-compatible-api - 1. add start option and configure; 2. optmize providers interfaces and implements; 3. rewrite the server construct function * merge: object * chore: add object lock * chore: of delete objs * chore: * fix: list objects bug * chore: rename ListObjetV1Handler to ListObjectHandler, rename BTFS-Hash to CID * refractor: bucket service * refactor: object service * refractor: refract object service * refractor: handlers * refractor: bucket handler * refractor: bucket handler * refractor: response * refractor: response func * refractor: response * refractor: object * refractor: object * refractor: objects * refractor: btf api add timeout & add cid refs to enable referred cid can not be deleted * ref: fix delete object remove body * ref: format code * fix: routers * fix: add cors header * fix: router option * feat: add delete objects handler * refractor: multipart * ref: multipart * fix: multipart etag calculation * chore: add min part size todo * chore: upgrade 'github.com/anacrolix/torrent' from v1.47.0 to v1.52.5 * opt: comment and amz header * opt: code * opt: preflight cache max age * feat: bucket response add acl header * opt: change cid-list header to cid * fix: required check exlude unknow location * fix: get object acl * ref: requests * ref: complete refractor * fix: args parse * fix: get object unlock * fix: object acl writer * fix: delete objects error * fix: Sign handler name * fix: object name escape * fix: copy source validate * opt: s3 log * opt: s3 api log * fix: allow Cache-Control header in PutObject and CopyObject Action * opt: add access-key command taglines * opt: add accesskey command description --------- Co-authored-by: Steve <[email protected]> Co-authored-by: fish <[email protected]> * feat: add backup and recovery command (#355) * fix: return an explicit error instead of panic and optimize the error log (#339) (#341) * feat: add multibase commands (#342) * feat: add backup and recovery command * feat: beautify the log * feat: init add recovery option * fix: format error * feat: add backup and recovery command (#348) * feat: add backup and recovery command * feat: beautify the log * feat: init add recovery option * fix: format error * feat: log details --------- Co-authored-by: fish <[email protected]> Co-authored-by: Shawn-Huang-Tron <[email protected]>
bittorrent · Oct 10, 2023 · aa64156 · aa64156
1 parent 58978cb
commit aa64156
Show file tree

Hide file tree

Showing 100 changed files with 13,938 additions and 647 deletions.
diff --git a/blocks/blockstoreutil/remove.go b/blocks/blockstoreutil/remove.go
@@ -71,8 +71,8 @@ func RmBlocks(ctx context.Context, blocks bs.GCBlockstore, pins pin.Pinner, cids
 }
 
 // FilterPinned takes a slice of Cids and returns it with the pinned Cids
-// removed. If a Cid is pinned, it will place RemovedBlock objects in the given
-// out channel, with an error which indicates that the Cid is pinned.
+// removed. If a CID is pinned, it will place RemovedBlock objects in the given
+// out channel, with an error which indicates that the CID is pinned.
 // This function is used in RmBlocks to filter out any blocks which are not
 // to be removed (because they are pinned).
 func FilterPinned(ctx context.Context, pins pin.Pinner, out chan<- interface{}, cids []cid.Cid) []cid.Cid {

diff --git a/chain/chain.go b/chain/chain.go
@@ -5,12 +5,13 @@ import (
 	"encoding/hex"
 	"errors"
 	"fmt"
-	"github.com/bittorrent/go-btfs/chain/tokencfg"
 	"io"
 	"math/big"
 	"strings"
 	"time"
 
+	"github.com/bittorrent/go-btfs/chain/tokencfg"
+
 	"github.com/bittorrent/go-btfs/accounting"
 	"github.com/bittorrent/go-btfs/chain/config"
 	"github.com/bittorrent/go-btfs/settlement"
@@ -348,7 +349,7 @@ func initSwap(
 	priceOracle := priceoracle.New(currentPriceOracleAddress, transactionService)
 	_, err := priceOracle.CheckNewPrice(tokencfg.GetWbttToken()) // CheckNewPrice when node starts
 	if err != nil {
-		return nil, nil, errors.New("CheckNewPrice " + err.Error())
+		return nil, nil, errors.New("CheckNewPrice error, it may happens when contract call failed if bttc chain rpc is down, please try again")
 	}
 
 	swapProtocol := swapprotocol.New(overlayEthAddress, priceOracle)

diff --git a/cmd/btfs/daemon.go b/cmd/btfs/daemon.go
@@ -7,6 +7,8 @@ import (
 	"errors"
 	_ "expvar"
 	"fmt"
+	"github.com/bittorrent/go-btfs/s3"
+	"github.com/bittorrent/go-btfs/s3/api/services/accesskey"
 	"io/ioutil"
 	"math/rand"
 	"net"
@@ -100,6 +102,7 @@ const (
 	chainID                   = "chain-id"
 	// apiAddrKwd    = "address-api"
 	// swarmAddrKwd  = "address-swarm"
+	enableS3CompatibleAPIKwd = "s3-compatible-api"
 )
 
 // BTFS daemon test exit error code
@@ -227,6 +230,7 @@ Headers.
 		// TODO: add way to override addresses. tricky part: updating the config if also --init.
 		// cmds.StringOption(apiAddrKwd, "Address for the daemon rpc API (overrides config)"),
 		// cmds.StringOption(swarmAddrKwd, "Address for the swarm socket (overrides config)"),
+		cmds.BoolOption(enableS3CompatibleAPIKwd, "Enable s3-compatible-api server"),
 	},
 	Subcommands: map[string]*cmds.Command{},
 	NoRemote:    true,
@@ -713,6 +717,33 @@ If the user need to start multiple nodes on the same machine, the configuration
 		functest(cfg.Services.OnlineServerDomain, cfg.Identity.PeerID, hValue)
 	}
 
+	// Init s3 providers
+	err = s3.InitProviders(statestore)
+	if err != nil {
+		return err
+	}
+
+	// Init access-key
+	accesskey.InitService(s3.GetProviders())
+
+	// Start s3-compatible-api server
+	s3OptEnable, s3Opt := req.Options[enableS3CompatibleAPIKwd].(bool)
+	if s3OptEnable || (!s3Opt && cfg.S3CompatibleAPI.Enable) {
+		s3Server := s3.NewServer(cfg.S3CompatibleAPI)
+		err = s3Server.Start()
+		if err != nil {
+			fmt.Printf("S3-Compatible-API server: %v\n", err)
+			return
+		}
+		fmt.Printf("S3-Compatible-API server started, endpoint-url: http://%s\n", cfg.S3CompatibleAPI.Address)
+		defer func() {
+			err = s3Server.Stop()
+			if err != nil {
+				fmt.Printf("S3-Compatible-API server: %v\n", err)
+			}
+		}()
+	}
+
 	if SimpleMode == false {
 		// set Analytics flag if specified
 		if dc, ok := req.Options[enableDataCollection]; ok == true {

diff --git a/cmd/btfs/init.go b/cmd/btfs/init.go
@@ -10,6 +10,7 @@ import (
 	"path/filepath"
 	"strconv"
 	"strings"
+	"time"
 
 	"github.com/bittorrent/go-btfs/assets"
 	"github.com/bittorrent/go-btfs/chain"
@@ -36,6 +37,7 @@ const (
 	rmOnUnpinOptionName = "rm-on-unpin"
 	seedOptionName      = "seed"
 	simpleMode          = "simple-mode"
+	recoveryOptionName  = "recovery"
 	/*
 		passWordOptionName     = "password"
 		passwordFileoptionName = "password-file"
@@ -72,6 +74,7 @@ environment variable:
 		cmds.BoolOption(rmOnUnpinOptionName, "r", "Remove unpinned files.").WithDefault(false),
 		cmds.StringOption(seedOptionName, "s", "Import seed phrase"),
 		cmds.BoolOption(simpleMode, "sm", "init with simple mode or not."),
+		cmds.StringOption(recoveryOptionName, "Recovery data from a backup"),
 		/*
 			cmds.StringOption(passWordOptionName, "", "password for decrypting keys."),
 			cmds.StringOption(passwordFileoptionName, "", "path to a file that contains password for decrypting keys"),
@@ -140,7 +143,30 @@ environment variable:
 			password, _ := req.Options[passWordOptionName].(string)
 			passwordFile, _ := req.Options[passwordFileoptionName].(string)
 		*/
+		backupPath, ok := req.Options[recoveryOptionName].(string)
+		if ok {
+			btfsPath := env.(*oldcmds.Context).ConfigRoot
+			dstPath := filepath.Dir(btfsPath)
+			if fsrepo.IsInitialized(btfsPath) {
+				newPath := filepath.Join(dstPath, fmt.Sprintf(".btfs_backup_%d", time.Now().Unix()))
+				// newPath := filepath.Join(filepath.Dir(btfsPath), backup)
+				err := os.Rename(btfsPath, newPath)
+				if err != nil {
+					return err
+				}
+				fmt.Println("btfs configuration file already exists!")
+				fmt.Println("We have renamed it to ", newPath)
+			}
 
+			if err := commands.UnTar(backupPath, dstPath); err != nil {
+				err = commands.UnZip(backupPath, dstPath)
+				if err != nil {
+					return errors.New("your file format is not tar.gz or zip, please check again")
+				}
+			}
+			fmt.Println("Recovery successful!")
+			return nil
+		}
 		return doInit(os.Stdout, cctx.ConfigRoot, empty, nBitsForKeypair, profile, conf, keyType, importKey, seedPhrase, rmOnUnpin, simpleModeIn)
 	},
 }

diff --git a/core/commands/accesskey.go b/core/commands/accesskey.go
@@ -0,0 +1,175 @@
+package commands
+
+import (
+	"errors"
+	cmds "github.com/bittorrent/go-btfs-cmds"
+	"github.com/bittorrent/go-btfs/core/commands/cmdenv"
+	"github.com/bittorrent/go-btfs/s3/api/services/accesskey"
+)
+
+var AccessKeyCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Manage S3-Compatible-API access-keys.",
+		ShortDescription: "Commands for generate, update, get and list access-keys stored in this node.",
+	},
+	Subcommands: map[string]*cmds.Command{
+		"generate": accessKeyGenerateCmd,
+		"enable":   accessKeyEnableCmd,
+		"disable":  accessKeyDisableCmd,
+		"reset":    accessKeyResetCmd,
+		"delete":   accessKeyDeleteCmd,
+		"get":      accessKeyGetCmd,
+		"list":     accessKeyListCmd,
+	},
+	NoLocal: true,
+}
+
+func checkDaemon(env cmds.Environment) (err error) {
+	node, err := cmdenv.GetNode(env)
+	if err != nil {
+		return
+	}
+	if !node.IsDaemon {
+		err = errors.New("please start the node first")
+	}
+	return
+}
+
+var accessKeyGenerateCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Generate a new access-key record.",
+		ShortDescription: "Outputs the new created access-key record.",
+	},
+	Arguments: []cmds.Argument{},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		ack, err := accesskey.Generate()
+		if err != nil {
+			return
+		}
+		err = cmds.EmitOnce(res, ack)
+		return
+	},
+}
+
+var accessKeyEnableCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Set status of the specified access-key to enable.",
+		ShortDescription: "Outputs empty if the access-key has been set to enable or it was already enabled.",
+	},
+	Arguments: []cmds.Argument{
+		cmds.StringArg("key", true, true, "The key").EnableStdin(),
+	},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		key := req.Arguments[0]
+		err = accesskey.Enable(key)
+		return
+	},
+}
+
+var accessKeyDisableCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Set status of the specified access-key to enable.",
+		ShortDescription: "Outputs empty if the access-key has been set to disable or it was already disabled.",
+	},
+	Arguments: []cmds.Argument{
+		cmds.StringArg("key", true, true, "The key").EnableStdin(),
+	},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		key := req.Arguments[0]
+		err = accesskey.Disable(key)
+		return
+	},
+}
+
+var accessKeyResetCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Rest secret of the specified access-key.",
+		ShortDescription: "Outputs the updated access-key record if it's secret has been reset.",
+	},
+	Arguments: []cmds.Argument{
+		cmds.StringArg("key", true, true, "The key").EnableStdin(),
+	},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		key := req.Arguments[0]
+		err = accesskey.Reset(key)
+		return
+	},
+}
+
+var accessKeyDeleteCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Delete the specified access-key",
+		ShortDescription: "Outputs empty if access-key record has been deleted.",
+	},
+	Arguments: []cmds.Argument{
+		cmds.StringArg("key", true, true, "The key").EnableStdin(),
+	},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		key := req.Arguments[0]
+		err = accesskey.Delete(key)
+		return
+	},
+}
+
+var accessKeyGetCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "Get an access-key detail info.",
+		ShortDescription: "Outputs access-key record for the specified key.",
+	},
+	Arguments: []cmds.Argument{
+		cmds.StringArg("key", true, true, "The key").EnableStdin(),
+	},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		key := req.Arguments[0]
+		ack, err := accesskey.Get(key)
+		if err != nil {
+			return
+		}
+		err = cmds.EmitOnce(res, ack)
+		return
+	},
+}
+
+var accessKeyListCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline:          "List all access-keys.",
+		ShortDescription: "Outputs all non-deleted access-keys stored in current node.",
+	},
+	Arguments: []cmds.Argument{},
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) (err error) {
+		err = checkDaemon(env)
+		if err != nil {
+			return
+		}
+		list, err := accesskey.List()
+		if err != nil {
+			return
+		}
+		err = cmds.EmitOnce(res, list)
+		return
+	},
+}