doc: provider with example (#845)

README.md

@@ -29,7 +29,7 @@ $ go install github.com/skip-mev/connect/v2
 The connect repository is composed of the following core packages:
 
 * **abci** - This package contains the [vote extension](./abci/ve/README.md), [proposal](./abci/proposals/README.md), and [preblock handlers](./abci/preblock/oracle/README.md) that are used to broadcast oracle data to the network and to store it in the blockchain.
-* **oracle** - This [package](./oracle/) contains the main oracle that aggregates external data sources before broadcasting it to the network. You can reference the provider documentation [here](./providers/base/README.md) to get a high level overview of how the oracle works.
+* **oracle** - This [package](./oracle/) contains the main oracle that aggregates external data sources before broadcasting it to the network. You can reference the provider documentation [here](providers/README.md) to get a high level overview of how the oracle works.
 * **providers** - This package contains a collection of [websocket](./providers/websockets/README.md) and [API](./providers/apis/README.md) based data providers that are used by the oracle to collect external data.
 * **x/oracle** - This package contains a Cosmos SDK module that allows you to store oracle data on a blockchain.
 * **x/marketmap** - This [package](./x/marketmap/README.md) contains  a Cosmos SDK module that allows for market configuration to be stored and updated on a blockchain.

providers/EXAMPLE.md

@@ -0,0 +1,149 @@
+# Provider Walkthrough
+
+Let's walk through an example provider to see how it is implemented.
+
+## Binance API Provider
+
+The [Binance API provider](./apis/binance/README.md) can be used as a reference for how to implement an API provider.
+
+### Implementation
+
+The package is laid out as follows:
+
+* **binance/**: Contains the source code of the package.
+    * `api_handler.go`: Main implementation of the `PriceAPIDataHandler` interface.
+    * `utils.go`: Helper functions and configuration of the expected types to receive from the Binance API.
+
+
+The logic below defines our implementation of the Binance API provider and its constructor.
+
+```go
+// APIHandler implements the PriceAPIDataHandler interface for Binance.
+// for more information about the Binance API, refer to the following link:
+// https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md#public-api-endpoints
+type APIHandler struct {
+	// api is the config for the Binance API.
+	api config.APIConfig
+	// cache maintains the latest set of tickers seen by the handler.
+	cache types.ProviderTickers
+}
+
+// NewAPIHandler returns a new Binance PriceAPIDataHandler.
+func NewAPIHandler(
+	api config.APIConfig,
+) (types.PriceAPIDataHandler, error) {
+	if api.Name != Name {
+		return nil, fmt.Errorf("expected api config name %s, got %s", Name, api.Name)
+	}
+
+	if !api.Enabled {
+		return nil, fmt.Errorf("api config for %s is not enabled", Name)
+	}
+
+	if err := api.ValidateBasic(); err != nil {
+		return nil, fmt.Errorf("invalid api config for %s: %w", Name, err)
+	}
+
+	return &APIHandler{
+		api:   api,
+		cache: types.NewProviderTickers(),
+	}, nil
+}
+```
+
+The `CreateURL()` function implementation for the Binance APIHandler creates the URL needed for querying the Binance API.  The URL is created by appending a list of desired tickers to the base URL of `https://api.binance.com/api/v3/ticker/price?symbols`.
+The URL and logic for appending ticker IDs were chosen based on the [Binance API documentation](https://binance-docs.github.io/apidocs/spot/en/#symbol-price-ticker).
+
+```go
+// CreateURL returns the URL that is used to fetch data from the Binance API for the
+// given tickers.
+func (h *APIHandler) CreateURL(
+	tickers []types.ProviderTicker,
+) (string, error) {
+	var tickerStrings string
+	for _, ticker := range tickers {
+		tickerStrings += fmt.Sprintf("%s%s%s%s", Quotation, ticker.GetOffChainTicker(), Quotation, Separator)
+		h.cache.Add(ticker)
+	}
+
+	if len(tickerStrings) == 0 {
+		return "", fmt.Errorf("empty url created. invalid or no ticker were provided")
+	}
+
+	return fmt.Sprintf(
+		h.api.Endpoints[0].URL,
+		LeftBracket,
+		strings.TrimSuffix(tickerStrings, Separator),
+		RightBracket,
+	), nil
+}
+```
+
+The `ParseResponse()` function implementation for the Binance APIHandler handles the response returned from the Binance API.
+The function:
+
+* Decodes the response to a known type (using JSON parsing)
+* Resolves the response tickers to the requested tickers
+* Converts the returned price for each ticker to a `*big.Float` for internal oracle use
+
+```go
+// ParseResponse parses the response from the Binance API and returns a GetResponse. Each
+// of the tickers supplied will get a response or an error.
+func (h *APIHandler) ParseResponse(
+    tickers []types.ProviderTicker,
+    resp *http.Response,
+) types.PriceResponse {
+    // Parse the response into a BinanceResponse.
+    result, err := Decode(resp)
+    if err != nil {
+        return types.NewPriceResponseWithErr(
+            tickers,
+            providertypes.NewErrorWithCode(err, providertypes.ErrorFailedToDecode),
+        )
+    }
+
+    var (
+        resolved   = make(types.ResolvedPrices)
+        unresolved = make(types.UnResolvedPrices)
+    )
+
+    for _, data := range result {
+        // Filter out the responses that are not expected.
+        ticker, ok := h.cache.FromOffChainTicker(data.Symbol)
+        if !ok {
+            continue
+        }
+
+        price, err := math.Float64StringToBigFloat(data.Price)
+        if err != nil {
+			wErr := fmt.Errorf("failed to convert price %s to big.Float: %w", data.Price, err)
+			unresolved[ticker] = providertypes.UnresolvedResult{
+                ErrorWithCode: providertypes.NewErrorWithCode(wErr, providertypes.ErrorFailedToParsePrice),
+            }
+            continue
+        }
+
+        resolved[ticker] = types.NewPriceResult(price, time.Now().UTC())
+    }
+
+    // Add currency pairs that received no response to the unresolved map.
+    for _, ticker := range tickers {
+        _, resolvedOk := resolved[ticker]
+        _, unresolvedOk := unresolved[ticker]
+
+        if !resolvedOk && !unresolvedOk {
+            unresolved[ticker] = providertypes.UnresolvedResult{
+                ErrorWithCode: providertypes.NewErrorWithCode(fmt.Errorf("no response"), providertypes.ErrorNoResponse),
+            }
+        }
+    }
+
+    return types.NewPriceResponse(resolved, unresolved)
+}
+```
+
+### Wiring to the Oracle
+
+To wire the provider to the oracle, it must be added to the corresponding [Oracle Factory](./factories/README.md).
+Here, the provider will be added to `APIQueryHandlerFactory` since we are making an API provider.  The factories
+are pre-wired to the oracle, so when it starts up, it will now have registered the given new provider.

providers/base/README.md → providers/README.md

@@ -13,16 +13,19 @@ Each base provider implementation will be run in a separate goroutine by the mai
 
 The base provider constructs a response channel that it is always listening to and making updates as needed. Every interval, the base provider will fetch the data from the underlying data source and send the response to the response channel, respecting the number of concurrent requests to the rate limit parameters of the underlying source (if it has any).
 
-![Architecture Overview](./architecture.png)
+![Architecture Overview](architecture.png)
 
+## Example Provider Walkthrough
+
+A walkthrough of the implementation of an example provider can be found [here](./EXAMPLE.md) for reference.
 
 ## API (HTTP) Based Providers
 
-In order to implement API based providers, you must implement the [`APIDataHandler`](./api/handlers/api_data_handler.go) interface and the [`RequestHandler`](./api/handlers/request_handler.go) interfaces. The `APIDataHandler` is responsible for creating the URL to be sent to the HTTP client and parsing the response from the HTTP response. The `RequestHandler` is responsible for making the HTTP request and returning the response.
+In order to implement API based providers, you must implement the [`APIDataHandler`](base/api/handlers/api_data_handler.go) interface and the [`RequestHandler`](base/api/handlers/request_handler.go) interfaces. The `APIDataHandler` is responsible for creating the URL to be sent to the HTTP client and parsing the response from the HTTP response. The `RequestHandler` is responsible for making the HTTP request and returning the response.
 
-Once these two interfaces are implemented, you can then instantiate an [`APIQueryHandler`](./api/handlers/api_query_handler.go) and pass it to the base provider. The `APIQueryHandler` is abstracts away the logic for making the HTTP request and parsing the response. The base provider will then take care of the rest. The responses from the `APIQueryHandler` are sent to the base provider via a buffered channel. The base provider will then store the data in a thread safe map. To read more about the various API provider configurations available, please visit the [API provider configuration](../../oracle/config/api.go) documentation.
+Once these two interfaces are implemented, you can then instantiate an [`APIQueryHandler`](base/api/handlers/api_query_handler.go) and pass it to the base provider. The `APIQueryHandler` is abstracts away the logic for making the HTTP request and parsing the response. The base provider will then take care of the rest. The responses from the `APIQueryHandler` are sent to the base provider via a buffered channel. The base provider will then store the data in a thread safe map. To read more about the various API provider configurations available, please visit the [API provider configuration](../oracle/config/api.go) documentation.
 
-Alternatively, you can directly implement the [`APIFetcher`](./api/handlers/api_query_handler.go) interface. This is appropriate if you want to abstract over the various processes of interacting with GRPC, JSON-RPC, REST, etc. APIs.
+Alternatively, you can directly implement the [`APIFetcher`](base/api/handlers/api_query_handler.go) interface. This is appropriate if you want to abstract over the various processes of interacting with GRPC, JSON-RPC, REST, etc. APIs.
 
 ### APIDataHandler
 
@@ -93,9 +96,9 @@ type APIFetcher[K providertypes.ResponseKey, V providertypes.ResponseValue] inte
 
 ## Websocket-Based Providers
 
-In order to implement websocket-based providers, you must implement the [`WebSocketDataHandler`](./websocket/handlers/ws_data_handler.go) interface and the [`WebSocketConnHandler`](./websocket/handlers/ws_conn_handler.go) interfaces. The `WebSocketDataHandler` is responsible for parsing messages from the websocket connection, constructing heartbeats, and constructing the initial subscription message(s). This handler must manage all state associated with the websocket connection i.e. connection identifiers. The `WebSocketConnHandler` is responsible for making the websocket connection and maintaining it - including reads, writes, dialing, and closing.
+In order to implement websocket-based providers, you must implement the [`WebSocketDataHandler`](base/websocket/handlers/ws_data_handler.go) interface and the [`WebSocketConnHandler`](base/websocket/handlers/ws_conn_handler.go) interfaces. The `WebSocketDataHandler` is responsible for parsing messages from the websocket connection, constructing heartbeats, and constructing the initial subscription message(s). This handler must manage all state associated with the websocket connection i.e. connection identifiers. The `WebSocketConnHandler` is responsible for making the websocket connection and maintaining it - including reads, writes, dialing, and closing.
 
-Once these two interfaces are implemented, you can then instantiate an [`WebSocketQueryHandler`](./websocket/handlers/ws_query_handler.go) and pass it to the base provider. The `WebSocketQueryHandler` abstracts away the logic for connecting, reading, sending updates, and parsing responses all using the two interfaces above. The base provider will then take care of the rest - including storing the data in a thread safe manner. To read more about the various configurations available for websocket providers, please visit the [websocket provider configuration](../../oracle/config/websocket.go) documentation.
+Once these two interfaces are implemented, you can then instantiate an [`WebSocketQueryHandler`](base/websocket/handlers/ws_query_handler.go) and pass it to the base provider. The `WebSocketQueryHandler` abstracts away the logic for connecting, reading, sending updates, and parsing responses all using the two interfaces above. The base provider will then take care of the rest - including storing the data in a thread safe manner. To read more about the various configurations available for websocket providers, please visit the [websocket provider configuration](../oracle/config/websocket.go) documentation.
 
 ### WebSocketDataHandler
 

providers/apis/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-API providers utilize rest APIs to retrieve data from external sources. The data is then transformed into a common format and aggregated across multiple providers. To implement a new provider, please read over the base provider documentation in [`providers/base/README.md`](../base/README.md).
+API providers utilize rest APIs to retrieve data from external sources. The data is then transformed into a common format and aggregated across multiple providers. To implement a new provider, please read over the base provider documentation in [`providers/base/README.md`](../README.md).
 
 ## Supported Providers
 

providers/websockets/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-Websocket providers utilize websocket APIs / clients to retrieve data from external sources. The data is then transformed into a common format and aggregated across multiple providers. To implement a new provider, please read over the base provider documentation in [`providers/base/README.md`](../base/README.md).
+Websocket providers utilize websocket APIs / clients to retrieve data from external sources. The data is then transformed into a common format and aggregated across multiple providers. To implement a new provider, please read over the base provider documentation in [`providers/base/README.md`](../README.md).
 
 Websockets are preferred over REST APIs for real-time data as they only require a single connection to the server, whereas HTTP APIs require a new connection for each request. This makes websockets more efficient for real-time data. Additionally, web sockets typically have lower latency than HTTP APIs, which is important for real-time data.