diff --git a/.gitbook.yaml b/.gitbook.yaml index cd617da8..986a93ce 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -60,13 +60,14 @@ redirects: documentation/consume/other-ways-of-consuming/files: ./new/consume/other-sinks/files.md documentation/consume/other-ways-of-consuming/prometheus: ./new/consume/other-sinks/prometheus.md documentation/consume/other-ways-of-consuming/pubsub: ./new/consume/other-sinks/pubsub.md - intro-solana: ./new/getting-started/solana/intro-solana.md - intro-injective: ./new/getting-started/injective/intro-injective.md - - - - - - - - + intro-solana: ./new/how-to-guides/your-first-application/solana.md + intro-injective: ./new/how-to-guides/your-first-application/injective.md + documentation/intro-getting-started: ./new/how-to-guides/your-first-application/intro-your-first-application.md + documentation/intro-getting-started/intro-evm: ./new/how-to-guides/your-first-application/evm.md + documentation/intro-getting-started/intro-evm/evm-first-sps: ./new/how-to-guides/your-first-application/evm.md + documentation/intro-getting-started/intro-evm/evm-first-sps: ./new/how-to-guides/your-first-application/evm.md + documentation/intro-getting-started/intro-evm/emv-first-sql: ./new/how-to-guides/your-first-application/evm.md + documentation/intro-getting-started/intro-solana: ./new/how-to-guides/your-first-application/solana.md + documentation/intro-getting-started/intro-injective: ./new/how-to-guides/your-first-application/injective.md + documentation/intro-getting-started/intro-injective/injective-first-sql: ./new/how-to-guides/your-first-application/injective.md + documentation/intro-getting-started/intro-injective/injective-first-sps: ./new/how-to-guides/your-first-application/injective.md \ No newline at end of file diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 6532a241..d0a57012 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -4,14 +4,11 @@ ## Documentation -* [Getting Started](new/getting-started/intro-getting-started.md) - * [EVM](new/getting-started/evm/intro-evm.md) - * [Your First Ethereum Subgraph](new/getting-started/evm/evm-first-sps.md) - * [Your First EVM SQL Substreams](new/getting-started/evm/emv-first-sql.md) - * [Solana](new/getting-started/solana/intro-solana.md) - * [Injective](new/getting-started/injective/intro-injective.md) - * [Your First Injective Subgraph](new/getting-started/injective/injective-first-sps.md) - * [Your First Injective SQL Substreams](new/getting-started/injective/injective-first-sql.md) +* [How-To-Guides](new/getting-started/intro-how-to-guides.md) + * [Your First Substreams Application](new/how-to-guides/your-first-application/intro-your-first-application.md) + * [EVM](new/how-to-guides/your-first-application/evm.md) + * [Solana](new/how-to-guides/your-first-application/solana.md) + * [Injective](new/how-to-guides/your-first-application/injective.md) * [Consume Substreams](new/consume/consume.md) * [Install the CLI](new/common/installing-the-cli.md) * [Authentication](new/common/authentication.md) diff --git a/docs/new/getting-started/evm/emv-first-sql.md b/docs/new/getting-started/evm/emv-first-sql.md deleted file mode 100644 index 6a59224c..00000000 --- a/docs/new/getting-started/evm/emv-first-sql.md +++ /dev/null @@ -1,95 +0,0 @@ -The `substreams init` command allows you to easily auto-generate several types of Substreams projects. **In this specific tutorial, you will learn how to bootstrap an EVM SQL Substreams to filter all the EVM information (events and calls) with almost no-code needed** - -## Before You Begin - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Get an authentication token](../../common/authentication.md) - -## Create the Project - -The `substreams init` command allows you choose among several code-generation tools, depending on your needs. In this example, you will create an EVM Substreams module that streams data to a PostgreSQL database. - -1. In a command-line terminal, run `substreams init`. - -2. Choose `ethereum-sql`. -Now, you will input the parameters needed to create your EVM SQL Substreams. - -3. `Project name`: give a name to the project. This will be used to create the folder in your computer. - -4. `Select the chain`: select the EVM chain (Arbitrum, Ethereum Mainnet, Polygon...). - -5. `Contract address`: input the smart contract address that you want to index. -The ABI will be fetched from the corresponding blockchain scanner (e.g. Etherscan or Arbiscan). This might take a few seconds to complete. - -6. `Verify the ABI`: the CLI will try to download the ABI the given smart contract address. -**If it is able to fetch it**, the ABI will be shown in the screen, so you just have to confirm it is correct. -**If it is NOT able to fetch it**, probably because the smart contract is not verified, you can input the ABI yourself. - -7. `Initial block number`: input the block where you want to start indexing. - -8. `Contract short name`: give an alias to the smart contract. This is useful in case you will track several smart contracts. The generated code will be prefix with the alias name. - -9. `What do you want to track from this contract?`: you can choose to track events, calls or both. - -10. `Is it a factory contract?`: choose whether the smart contract is a factory contract. If it is, you will be required to provide the event that signals the creation of new contracts and the address of one of the created contracts (so that the ABI can be retrieved). - -11. `Add another contract`: choose if you want to index another contract. You can track as many contracts as you need. - -1. `SQL type`: you can choose the driver: PosgreSQL or Clickhouse. - -13. `Directory`: choose the directory where the project will be created. - -14. `Package build`: whether you want the Substreams package to the built. -The Substreams package is the binary that extracts all the data from the blockchain. -**Always select `Yes, build it`. Otherwise, you'd have to _unzip_ the project and build it yourself.** - -## Inspect the Project - -Open the generated project in an IDE of your choice (e.g. VSCode). - -
- -1. The `dev-environment` folder contains the Docker Compose file to spin up a local PostgreSQL database. - -2. The `schema.sql` file defines the SQL schema used in the database. The corresponding tables will be created based on the ABI of the smart contract. - -3. The `.spkg` file is a binary file that contains the Substreams itself. This file contains all the code necessary to extract the events and/or calls. - -## Run the Substreams and Store the Data in a Database - -In order to feed the data into PostgreSQL, you will need the [substreams-sql-sink](https://substreams.streamingfast.io/documentation/consume/sql/sql-sink) binary. The `substreams-sql-sink` binary will execute the `.spkg` file (i.e. run the Substreams) and send the data to Postgres. Essentially, it acts as a bridge between the Substreams execution and the PostgreSQL database. - -1. Get the `sink-sql-binary`: - -```bash -make get-sink-sql-binary -``` - -The binary will be stored in the `bin` folder of the project. - -1. Launch a local PostgreSQL databe using Docker. - -```bash -make launch-postgresql-db -``` - -{% hint style="info" %} -**Note**: Make sure you have Docker installed and running. -The Docker set-up will use ports `5432` and `8081`. If you run into issues, make sure this ports are not used in your computer. -{% endhint %} - -1. Set-up the the `substreams-sql-sink` to point to the right Substreams package (i.e. the spkg file of the project). - -```bash -make setup-sink -``` - -1. Start sinking the data by running the `substreams-sql-sink`. - -```bash -make run-sink -``` - -The SQL database will start getting data from the Substreams execution. - -You can now navigate to `https://localhost:8081` in your web browser to access the SQL data through PgWeb. \ No newline at end of file diff --git a/docs/new/getting-started/evm/evm-first-sps.md b/docs/new/getting-started/evm/evm-first-sps.md deleted file mode 100644 index 03f0c912..00000000 --- a/docs/new/getting-started/evm/evm-first-sps.md +++ /dev/null @@ -1,115 +0,0 @@ -The `substreams init` command allows you to easily auto-generate several types of Substreams projects. **In this specific tutorial, you will learn how to bootstrap an EVM Substreams-powered Subgraph to filter all the EVM information (events and calls) with almost no-code needed** - -## Before You Begin - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Get an authentication token](../../common/authentication.md) -- [Install the Graph CLI](https://thegraph.com/docs/en/quick-start/#2-install-the-graph-cli) - -## Create the Project - -The `substreams init` command allows you choose among several code-generation tools, depending on your needs. In this example, you will create an EVM Substreams module that streams data to a subgraph. - -1. In a command-line terminal, run `substreams init`. - -2. Choose `ethereum-subgraph`. -Now, you will input the parameters needed to create your EVM Substreams-powered Subgraph. - -3. `Project name`: give a name to the project. This will be used to create the folder in your computer. - -4. `Select the chain`: select the EVM chain (Arbitrum, Ethereum Mainnet, Polygon...) - -5. `Contract address`: input the smart contract address that you want to index. -The ABI will be fetched from the corresponding blockchain scanner (e.g. Etherscan or Arbiscan). This might take a few seconds to complete. - -6. `Verify the ABI`: the CLI will try to download the ABI the given smart contract address. -**If it is able to fetch it**, the ABI will be shown in the screen, so you just have to confirm it is correct. -**If it is NOT able to fetch it**, probably because the smart contract is not verified, you can input the ABI yourself. - -7. `Initial block number`: input the block where you want to start indexing. - -8. `Contract short name`: give an alias to the smart contract. This is useful in case you will track several smart contracts. The generated code will be prefix with the alias name. - -9. `What do you want to track from this contract?`: you can choose to track events, calls or both. - -10. `Is it a factory contract?`: choose whether the smart contract is a factory contract. If it is, you will be required to provide the event that signals the creation of new contracts and the address of one of the created contracts (so that the ABI can be retrieved). - -11. `Add another contract`: choose if you want to index another contract. You track as many contracts as you need. - -12. `Subgraph feed mode`: there are two in which subgraphs can receive data from Substreams: _entities_ and _triggers_ - - **Entities**: all the logic lives in the Substreams, so you can't modify the subgraph code in AssemblyScript. - - **Triggers**: the logic lives in the subgraph, so you **CAN** modify the subgraph code in AssemblyScript and create your own aggregations. In this case, the Substreams only acts as a bridge between the chain and the subgraph. - -{% hint style="info" %} -**Tip**: Choose _triggers_ if you want to place the code logic in the subgraph. You will be able to access the events and calls directly within the subgraph and create your own entities. -{% endhint %} - -13. `Directory`: choose the directory where the project will be created. - -14. `Package build`: whether you want the Substreams package to the built. -The Substreams package is the binary that extracts all the data from the blockchain. -**Always select `Yes, build it`. Otherwise, you'd have to _unzip_ the project and build it yourself.** - -## Inspect the Project - -Now, let's inspect the project and understand all the files created. Note that in this example, the following options have been chosen: -- Track events and calls. -- No factory contract. -- Use triggers to feed data into the subgraph. - -Depending on the options you choose in the codegen tool, the files created might differ, but the overall structure should remain the same. - -In an IDE of your choice (e.g. VSCode), open the project. - -
- -1. The `src/mappings.ts` file contains the source of the subgraph. Here, you define the `handleEvents` function, which receives all the filtered events from the Substreams. You can manipulate and aggregate the events to create the subgraph entities that you need. - -2. The `.spkg` file is the binary file generated by the Substreams. Essentially, this file contains the all the source code to filter the events that you selected when running the `substreams init` command. -The Graph Node (i.e. the software that runs your subgraph) uses the `spkg` file to extract the blockchain data and provide it as AssemblyScript code to the `handleEvents` function. - -3. The `schema.graphql` file contains the subgraph entities. You can use this file to define the output model of your subgraph. -The [entities defined in the GraphQL schema](https://thegraph.com/docs/en/developing/creating-a-subgraph/#the-graphql-schema) are generated into AssemblyScript code by running the `graph codegen` command. - -4. The `subgraph.yaml` file is the configuration file of the subgraph. It defines the source code of the subgraph (in this case, Substreams) and the handler that will receive the data extracted from the blockchain (in this case, the `handleEvents` function). - -## Build the Project - -Before deploying your subgraph, you must install and generate all the necessary code: -1. Run `npm install` to install the dependencies of the project. -2. Run `npm run generate` to generate the output of the Substreams (i.e. the events and calls) in AssemblyScript. This allows you to access the output of the Substreams inside the AssemblyScript code. -In this case, the `EventsCalls` object, representing the events and calls extracted from the blockchain. -3. Run `npm run codegen` to generate the subgraph entities that you have defined in the `schema.graphql` file. - -## Test the Subgraph - -After installing all the necessary depdendencies of the project, you are now able to deploy the subgraph a local Graph Node. - -**Before publishing to decentralized network, you can test your subgraph locally**, thus not incurring in any costs associated with the deployment. You can run a Graph Node instance (the software that indexers use) in a Docker environment, and replicate the deployment locally. - -In the project that you created previously, the `dev-environment` folders contains all the necessary files to spin up a Docker-based Graph Node environment (note that you will need Docker installed and available in your computer). - -1. Spin up the environment by running the `dev-environment/start.sh` script. -This script will create a Graph Node instance, a Postgres node and an IPFS node (all these dependencies are necessary for the local Graph Node to succesfully work). - -2. Once the Graph Node has started, in another command-line session, create a new subgraph using the NPM scripts provided in the project: - -{% hint style="info" %} -**Note**: The NPM scripts, defined in the `scripts` section of the `package.json` file, are just a wrapped of the Graph CLI commands. -{% endhint %} - -```bash -npm run create-local -``` - -The previous command connects to the local Graph Node and creates a subgraph. You can always remove the subgraph using the `npm run remove-local` command. - -3. Deploy the subgraph. - -```bash -npm run deploy-local -``` - -The previous command deploys the subgraph to the local Graph Node. You can now go back to the logs of the local Graph Node and check out the subgraph indexing. - -In a production environment, you will publish your subgraph to the [The Graph decentralized network](https://thegraph.com/docs/en/publishing/publishing-a-subgraph/) (note for EVM chains you can also use The Graph Studio). \ No newline at end of file diff --git a/docs/new/getting-started/evm/intro-evm.md b/docs/new/getting-started/evm/intro-evm.md deleted file mode 100644 index e31e1a29..00000000 --- a/docs/new/getting-started/evm/intro-evm.md +++ /dev/null @@ -1,26 +0,0 @@ -Substreams allows you to easily extract data from EVM blockchains. With Substreams, you can retrieve transactions, instructions or accounts, taking advantage of its powerful streaming technology. It's super fast! - -
- -## Getting Started - -There are two main concepts in Substreams: packages and modules. Essentially, **a _module_ is a Rust function that contains definitions to extract data from the blockchain**. Modules are grouped in **_packages_, which are binary files (`.spkg`) that contain one or several modules**. For example, you might have a package called `ethereum-common`, which has two modules: `get_transactions` and `get_events`. - -Developing a Substreams module requires some knowledge of Rust, but the **code-generation tools provided by the Substreams CLI make it so easy to bootstrap a Substreams project**. - -To consume Ethereum data with The Graph you have two options: - -**- Substreams-powered Subgraphs:** you import a Substreams module into the subgraph. Essentially, Substreams acts just as the extraction layer, but the business logic lives in the subgraph. The Substreams only provides _raw_ data (you use AssemblyScript to code your subgraph). - -**- Substreams directly:** you develop your own Substreams module (you use Rust to code your Substreams). Then, you can choose how to consume the data (SQL, Prometheus, streaming...). - -Choosing one over the other is up to you. Consider the needs of your application and the skills needed to develop. - -{% hint style="success" %} -**Tip**: If you do not know what subgraphs are, please take a look at [The Graph documentation](https://thegraph.com/docs/en/quick-start/). -{% endhint %} - -To get started, start auto-generating your Substreams using the `substreams init` command: - -- [Create Your First Substreams-powered Subgraph](../evm/evm-first-sps.md) -- [Create Your First SQL Substreams](../evm/emv-first-sql.md) diff --git a/docs/new/getting-started/injective/injective-first-sps.md b/docs/new/getting-started/injective/injective-first-sps.md deleted file mode 100644 index 7caa5c5c..00000000 --- a/docs/new/getting-started/injective/injective-first-sps.md +++ /dev/null @@ -1,97 +0,0 @@ -The `substreams init` command allows you to easily auto-generate several types of Substreams projects. **In this specific tutorial, you will learn how to bootstrap an Injective Substreams-powered Subgraph to filter all the Injective events that you want with almost no-code needed!** - -## Before You Begin - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Get an authentication token](../../common/authentication.md) -- [Install the Graph CLI](https://thegraph.com/docs/en/quick-start/#2-install-the-graph-cli) - -## Create the Project - -The `substreams init` command allows you choose among several code-generation tools, depending on your needs. In this example, you will create an Injective Substreams module that streams data to a subgraph. - -1. In a command-line terminal, run `substreams init`. -You can choose among several project types. - -1. Choose `injective-subgraph`. -Now, you will input the parameters needed to create your Injective Substreams-powered Subgraph. - -1. `Project name`: give a name to your project. -This will be the name of the folder in your filesystem. - -1. `Chain`: you can choose to index data on `Injective Mainnet` or `Injective Testnet`. - -1. `Start block`: first block where your Subgraph starts indexing data. - -1. `Filtering options`: the Injective data that you will received will be filtered based on the events (e.g. using the event type of the event attributes). There are several options: - - `Specific events`: you get a list of events filtered by the event type and/or the event attributes. Note that you will **only** get the events specified. - - `All events in transactions where at least one event matches your query`: in this case, you get **all** the events of the transaction as long as one event matches your criteria within the transaction. - - `Full transactions where at least one event matches your query`: you get a list transactions containing at least one of the events specified. - -The [Injective Explorer](https://explorer.injective.network/) gives you information about the different event types and attributes contained in a transaction. - -
- -Then, you will be asked to input the _type_ and/or _attributes_ that you want to use for the filtering. - -1. `Directory`: lastly, choose the directory where you want to create the project. - -Done! A new Substreams-powered Subgraph project will be created in the specified directory. - -## Inspect the Project - -Now, let's inspect the project and understand all the files created. In an IDE of your choice (e.g. VSCode), open the project. - -
- -1. The `src/mappings.ts` file contains the source of the subgraph. Here, you define the `handleEvents` function, which receives all the filtered events from the Substreams. You can manipulate and aggregate the events to create the subgraph entities that you need. - -2. The `.spkg` file is the binary file generated by the Substreams. Essentially, this file contains the all the source code to filter the events that you selected when running the `substreams init` command. - -The Graph Node (i.e. the software that runs your subgraph) uses the `spkg` file to extract the Injective data and provide it as AssemblyScript code to the `handleEvents` function. - -3. The `schema.graphql` file contains the subgraph entities. You can use this file to define the output model of your subgraph. -The [entities defined in the GraphQL schema](https://thegraph.com/docs/en/developing/creating-a-subgraph/#the-graphql-schema) are generated into AssemblyScript code by running the `graph codegen`. - -4. The `subgraph.yaml` file is the configuration file of the subgraph. It defines the source code of the subgraph (in this case, Substreams) and the handler that will receive the data extracted from Injective (in this case, the `handleEvents` function). - -## Build the Project - -Before deploying your subgraph, you must install and generate all the necessary code: -1. Run `npm install` to install the dependencies of the project. -2. Run `npm run generate` to generate the output of the Substreams (i.e. the Injective event list) in AssemblyScript. This allows you to access the output of the Substreams inside the AssemblyScript code. -In this case, the `EventList` object, representing a list of Injective events, is generated. -3. Run `npm run codegen` to generate the subgraph entities that you have defined in the `schema.graphql` file. - -## Test the Subgraph - -After installing all the necessary depdendencies of the project, you are now able to deploy the subgraph a local Graph Node. - -**Before publishing to decentralized network, you can test your subgraph locally**, thus not incurring in any costs associated with the deployment. You can run a Graph Node instance (the software that indexers use) in a Docker environment, and replicate the deployment locally. - -In the project that you created previously, the `dev-environment` folders contains all the necessary files to spin up a Docker-based Graph Node environment (note that you will need Docker installed and available in your computer). - -1. Spin up the environment by running the `dev-environment/start.sh` script. -This script will create a Graph Node instance, a Postgres node and an IPFS node (all these dependencies are necessary for the local Graph Node to succesfully work). - -2. Once the Graph Node has started, in another command-line session, create a new subgraph using the NPM scripts provided in the project: - -{% hint style="info" %} -**Note**: The NPM scripts, defined in the `scripts` section of the `package.json` file, are just a wrapped of the Graph CLI commands. -{% endhint %} - -```bash -npm run create-local -``` - -The previous command connects to the local Graph Node and creates a subgraph. You can always remove the subgraph using the `npm run remove-local` command. - -3. Deploy the subgraph. - -```bash -npm run deploy-local -``` - -The previous command deploys the subgraph to the local Graph Node. You can now go back to the logs of the local Graph Node and check out the subgraph indexing. - -In a production environment, you will publish your subgraph to the [The Graph decentralized network](https://thegraph.com/docs/en/publishing/publishing-a-subgraph/) (note that The Graph Studio, the hosted testing environment, is not supported for Injective). \ No newline at end of file diff --git a/docs/new/getting-started/injective/injective-first-sql.md b/docs/new/getting-started/injective/injective-first-sql.md deleted file mode 100644 index e3abee70..00000000 --- a/docs/new/getting-started/injective/injective-first-sql.md +++ /dev/null @@ -1,90 +0,0 @@ -The `substreams init` command allows you to easily auto-generate several types of Substreams projects. **In this specific tutorial, you will learn how to bootstrap an Injective SQL Substreams to filter all the Injective information (events and calls) with almost no-code needed** - -## Before You Begin - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Get an authentication token](../../common/authentication.md) - -## Create the Project - -The `substreams init` command allows you choose among several code-generation tools, depending on your needs. In this example, you will create an Injective Substreams module that streams data to a PostgreSQL database. - -1. In a command-line terminal, run `substreams init`. - -1. Choose `injective-sql`. -Now, you will input the parameters needed to create your Injective SQL Substreams. - -1. `Project name`: give a name to the project. This will be used to create the folder in your computer. - -1. `Select the chain`: you can choose to index data on `Injective Mainnet` or `Injective Testnet`. - -1. `Start block`: first block where your Subgraph starts indexing data. - -1. `Filtering options`: the Injective data that you will received will be filtered based on the events (e.g. using the event type of the event attributes). There are several options: - - `Specific events`: you get a list of events filtered by the event type and/or the event attributes. Note that you will **only** get the events specified. - - `All events in transactions where at least one event matches your query`: in this case, you get **all** the events of the transaction as long as one event matches your criteria within the transaction. - -The [Injective Explorer](https://explorer.injective.network/) gives you information about the different event types and attributes contained in a transaction. - -
- -Then, you will be asked to input the _type_ and/or _attributes_ that you want to use for the filtering. - -1. `SQL type`: you can choose the driver: PosgreSQL or Clickhouse. - -1. `Directory`: choose the directory where you want to create the project. - -1. `Package build`: whether you want the Substreams package to the built. -The Substreams package is the binary that extracts all the data from the blockchain. -**Always select `Yes, build it`. Otherwise, you'd have to _unzip_ the project and build it yourself.** - -## Inspect the Project - -Open the generated project in an IDE of your choice (e.g. VSCode). - -
- -1. The `dev-environment` folder contains the Docker Compose file to spin up a local PostgreSQL database. - -2. The `schema.sql` file defines the SQL schema used in the database. The corresponding tables will be created based on the ABI of the smart contract. - -3. The `.spkg` file is a binary file that contains the Substreams itself. This file contains all the code necessary to extract the events and/or calls. - -## Run the Substreams and Store the Data in a Database - -In order to feed the data into PostgreSQL, you will need the [substreams-sink-sql](https://substreams.streamingfast.io/documentation/consume/sql/sql-sink) binary. The `substreams-sink-sql` binary will execute the `.spkg` file (i.e. run the Substreams) and send the data to Postgres. Essentially, it acts as a bridge between the Substreams execution and the PostgreSQL database. - -1. Get the `substreams-sink-sql` binary: - -```bash -make get-sink-sql-binary -``` - -The binary will be stored in the `bin` folder of the project. - -1. Launch a local PostgreSQL databe using Docker. - -```bash -make launch-postgresql-db -``` - -{% hint style="info" %} -**Note**: Make sure you have Docker installed and running. -The Docker set-up will use ports `5432` and `8081`. If you run into issues, make sure this ports are not used in your computer. -{% endhint %} - -1. Set-up the the `substreams-sql-sink` to point to the right Substreams package (i.e. the spkg file of the project). - -```bash -make setup-sink -``` - -1. Start sinking the data by running the `substreams-sql-sink`. - -```bash -make run-sink -``` - -The SQL database will start getting data from the Substreams execution. - -You can now navigate to `https://localhost:8081` in your web browser to access the SQL data through PgWeb. \ No newline at end of file diff --git a/docs/new/getting-started/injective/intro-injective.md b/docs/new/getting-started/injective/intro-injective.md deleted file mode 100644 index 701c8b4a..00000000 --- a/docs/new/getting-started/injective/intro-injective.md +++ /dev/null @@ -1,27 +0,0 @@ -**With Substreams, you can extract data from the Injective blockchain**. Then, you can consume the data in several ways, such as a subgraph, through gRPC streaming, using a SQL database and many more. - -
- -# Getting Started - -There are two main concepts in Substreams: packages and modules. Essentially, **a _module_ is a Rust function that contains definitions to extract data from the blockchain**. Modules are grouped in **_packages_, which are binary files (`.spkg`) that contain one or several modules**. For example, you might have a package called `injective-common`, which has two modules: `get_transactions` and `get_events`. - -Developing a Substreams module requires some knowledge of Rust, but the **code-generation tools provided by the Substreams CLI make it so easy to bootstrap a Substreams project**. - -To consume Injective data with The Graph you have two options: - -**- Substreams-powered Subgraphs:** you import a Substreams module into the subgraph. Essentially, Substreams acts just as the extraction layer, but the business logic lives in the subgraph. The Substreams only provides _raw_ data (you use AssemblyScript to code your subgraph). - -**- Substreams directly:** you develop your own Substreams module (you use Rust to code your Substreams). Then, you can choose how to consume the data (SQL, Prometheus, streaming...). - -Choosing one over the other is up to you. Consider the needs of your application and the skills needed to develop. - -{% hint style="success" %} -**Tip**: If you do not know what subgraphs are, please take a look at [The Graph documentation](https://thegraph.com/docs/en/quick-start/). -{% endhint %} - -To get started, start auto-generating your Substreams using the `substreams init` command: - -- [Create Your First Substreams-powered Subgraph](../injective/injective-first-sps.md) -- [Create Your First SQL Substreams](../injective/injective-first-sql.md) - diff --git a/docs/new/getting-started/intro-getting-started.md b/docs/new/getting-started/intro-getting-started.md deleted file mode 100644 index 926c9c5c..00000000 --- a/docs/new/getting-started/intro-getting-started.md +++ /dev/null @@ -1,9 +0,0 @@ -Substreams supports several blockchain data models, so getting started differs based on the blockchain that you want to index data on. - -

Protobuf for the different supported chains

- -Please, refer to the corresponding _getting started_ tutorial depending on your needs: - -- [EVM-based blockchains](./evm/intro-evm.md) -- [Solana](./solana/intro-solana.md) -- [Injective](./injective/intro-injective.md) diff --git a/docs/new/getting-started/solana/intro-solana.md b/docs/new/getting-started/solana/intro-solana.md deleted file mode 100644 index 837fbac8..00000000 --- a/docs/new/getting-started/solana/intro-solana.md +++ /dev/null @@ -1,62 +0,0 @@ -Substreams allows you to easily extract data from the the Solana blockchain. With Substreams, you can retrieve transactions, instructions or accounts, taking advantage of its powerful streaming technology. It's super fast! - -
- -## Getting Started - -First, you must consider whether you want to develop your own Substreams or consume a ready-to-use Substreams. It is possible that someone has already built a Substreams package to extract the data you want; you can explore Substreams packages in the [Substreams Registry](https://substreams.dev). - -**If you have found a Substreams package that fits your needs**, then explore the [Consume Substreams](../consume/consume.md) section. At the most basic level you should cover: - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Authentication](../../common/authentication.md) -- [Packages](../../common/packages.md) -- Choose how you want to consume the data: - - [Send the data to a SQL database.](../../consume/sql/sql.md) - - [Stream the data from your application.](../../consume/stream/stream.md) - - [Send the data to a subgraph.](../../consume/subgraph/subgraph.md) - -**If you can't find a Substreams package that fits your needs**, then you can go ahead and develop your own Substreams. The [Develop Substreams](../develop/develop.md) section of the documentation covers everything you need to know about building a Substreams from scratch. At the most basic level, you should cover: - -- [Install the Substreams CLI](../../common/installing-the-cli.md) -- [Authentication](../../common/authentication.md) -- [Manifest & Modules](../../common/manifest-modules.md) -- [Protobuf defitions](../../develop/creating-protobuf-schemas.md) -- [Packages](../../common/packages.md) -- [Run a Substreams](../../common/running-substreams.md) -- [Choose how you want to consume the data](../../consume/consume.md) - -## Tutorials - -If you want to deep dive into the code, you can follow one or several of the Solana Tutorials available in the documentation. -- The [Explore Solana Tutorial](../../tutorials/solana/explore-solana/explore-solana.md) will teach you the most basic operations you can perform in a Solana Substreams. -- The [Solana Token Tracker Tutorial](../../tutorials/solana/token-tracker/token-tracker.md) will teach how to track an SPL token of your choice. -- The [NFT Trades Tutorial](../../tutorials/solana/top-ledger/nft-trades.md) will help you in extracting data from different NFT exchanges. -- The [DEX Trades Tutorial](../../tutorials/solana/top-ledger/dex-trades.md) will help you in extracting data from different decentralized exchanges. - -## The Solana Data Model - -A Substreams module is, essentially, a Rust function that extracts data from the blockchain. In order to specify which data you want to retrieve, Substreams gives you access to the abstraction of a full [Solana block](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L9). - -In the following example, a Solana block (`solana::Block`) is passed as a parameter. Then, a custom object defined by the user, `BlockMeta`, is emitted as output, containing some relevant fields (`slot`, `hash`, `parent_hash`): - -```rust -fn map_block_meta(block: solana::Block) -> Result { - Ok(BlockMeta { - slot: blk.slot, - hash: blk.blockhash, - parent_hash: blk.previous_blockhash, - }) -} -``` - -The [Block](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L9) object holds other important data, such as: -- [block.transactions_owned()](https://github.com/streamingfast/substreams-solana/blob/1f66cc3081f61ad1189dc814cb82096ae5ac4b3b/core/src/block_view.rs#L15) -- [block.rewards](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L64) -- [block.parent_slot](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L12) - -## The Account Address Tables - -In Solana, the _account_ concept plays a very important role. However, the original Solana data model imposes a restriction on the number of accounts that a single transaction can have. The [Account Address Tables](https://docs.solana.com/developing/lookup-tables) is a way to overcome this restriction, allowing developers to increase the number of accounts per transaction. - -The [resolved_accounts()](https://github.com/streamingfast/substreams-solana/blob/1f66cc3081f61ad1189dc814cb82096ae5ac4b3b/core/src/lib.rs#L9) method of the [ConfirmedTransaction](https://github.com/streamingfast/substreams-solana/blob/1f66cc3081f61ad1189dc814cb82096ae5ac4b3b/core/src/lib.rs#L8) object includes a ready-to-use array of accounts, including accounts from the Lookup Table. \ No newline at end of file diff --git a/docs/new/how-to-guides/intro-how-to-guides.md b/docs/new/how-to-guides/intro-how-to-guides.md new file mode 100644 index 00000000..0e12d66b --- /dev/null +++ b/docs/new/how-to-guides/intro-how-to-guides.md @@ -0,0 +1 @@ +This section contains How-To-Guides that will onboard you on different aspects of the Substreams and Substreams-powered Subgraphs development. \ No newline at end of file diff --git a/docs/new/how-to-guides/your-first-application/evm.md b/docs/new/how-to-guides/your-first-application/evm.md new file mode 100644 index 00000000..641167af --- /dev/null +++ b/docs/new/how-to-guides/your-first-application/evm.md @@ -0,0 +1,12 @@ +Check out the [Getting Started Guide](./intro-your-first-application.md) for more information on how to initialize your project. There are two options within `substreams init` to initialize your EVM Substreams: + +- `evm-minimal`: creates a simple Substreams that extracts raw data from the block (generates Rust code). +- `evm-events-calls`: creates a Substreams that extracts EVM events and calls filtered by one or several smart contract addresses. + +## EVM Foundational Modules + +The `evm-events-calls` codegen path relies on one of the [EVM Foundational Modules](https://github.com/streamingfast/substreams-foundational-modules/tree/develop/ethereum-common). A Foundational Module extracts the most relevant data from blockchain, so that you don't have to code it yourself. + +Specifically, the `evm-events-calls` path uses the [filtered_events_and_calls](https://github.com/streamingfast/substreams-foundational-modules/blob/develop/ethereum-common/substreams.yaml#L142) module from the EVM Foundational Modules to retrieve all the events filtered by specific smart contract addresses. + +
diff --git a/docs/new/how-to-guides/your-first-application/injective.md b/docs/new/how-to-guides/your-first-application/injective.md new file mode 100644 index 00000000..12cc2822 --- /dev/null +++ b/docs/new/how-to-guides/your-first-application/injective.md @@ -0,0 +1,12 @@ +Check out the [Getting Started Guide](./intro-your-first-application.md) for more information on how to initialize your project. There are two options within `substreams init` to initialize your Injective Substreams: + +- `injective-minimal`: creates a simple Substreams that extracts raw data from the block (generates Rust code). +- `injective-events`: creates a Substreams that extracts Injective events filtered by _type_ and/or _attributes_. + +## Injective Foundational Modules + +The `injective-events` codegen path relies on one of the [Injective Foundational Modules](https://github.com/streamingfast/substreams-foundational-modules/tree/develop/injective-common). A Foundational Module extracts the most relevant data from blockchain, so that you don't have to code it yourself. + +Specifically, the `injective-events` path uses the [filtered_events](https://github.com/streamingfast/substreams-foundational-modules/blob/develop/injective-common/substreams.yaml#L58) module from the Injective Foundational Modules to retrieve all the events filtered by event type and/or attributes. + +
\ No newline at end of file diff --git a/docs/new/how-to-guides/your-first-application/intro-your-first-application.md b/docs/new/how-to-guides/your-first-application/intro-your-first-application.md new file mode 100644 index 00000000..07e7d8a1 --- /dev/null +++ b/docs/new/how-to-guides/your-first-application/intro-your-first-application.md @@ -0,0 +1,29 @@ +Quickly scaffold your first project inside the [Substreams Development Environment](https://github.com/streamingfast/substreams-starter?tab=readme-ov-file) (”devcontainer”). Check out the Getting Started in the `README`to run remotely or clone the substreams-starter repository to run locally. Inside the devcontainer the `substreams init` command sets up a code-generated Substreams project, from which you can easily build either a subgraph or an SQL-based solution for handling data. + +{% hint style="info" %} +**Note:** Validate Docker and VS Code are up-to-date. +{% endhint %} + +Consult the relevant ecosystem guide to get started using real-time and historical indexed data: + +- Solana +- EVM +- Injective + +## Navigatin the Devcontainer + +When entering the devcontainer, you can either insert your own `substreams.yaml` file and run `substreams build` to generate the associated Protobuf files, or choose from two auto-generated code-paths: + +- **Minimal**: Creates a simple Substreams that extracts raw data from the block. +- **Non-Minimal**: Extracts filtered data specific to the network and relies on the cache and Protobufs provided by the Foundational Modules. + +Complete your Substreams project to be fully queryable, either through a Subgraph or directly from your SQL database, ensuring seamless access to the data for analysis and application use: + +- `substreams codegen subgraph`: The generated project follows the standard subgraph structure. By default, the `schema.graphql` and `mappings.ts` files respectively include only a required input ID and the basic code to create one. It's up to you to decide what entities to create based on the data extracted by Substreams. For technical details on how to configure a Subgraph sink, [click here](https://substreams.streamingfast.io/documentation/consume/subgraph). +- `substreams codegen sql`: For technical details on how to configure a SQL sink, [click here](https://substreams.streamingfast.io/documentation/consume/sql). + +If your plan is to deploy a Subgraph you may choose to either run the `graph-node` locally with the `deploy-local` command or deploy to the Subgraph Studio by checking out the `deploy` command in the `package.json`. + +{% hint style="info" %} +**Note:** When running local, make sure to verify that all containers are running properly in the Docker tab and that there’s no errors in the logs. +{% endhint %} \ No newline at end of file diff --git a/docs/new/how-to-guides/your-first-application/solana.md b/docs/new/how-to-guides/your-first-application/solana.md new file mode 100644 index 00000000..497ac754 --- /dev/null +++ b/docs/new/how-to-guides/your-first-application/solana.md @@ -0,0 +1,12 @@ +Check out the [Getting Started Guide](./intro-your-first-application.md) for more information on how to initialize your project. There are two options within `substreams init` to initialize your Solana Substreams: + +- `sol-minimal`: Creates a simple Substreams that extracts raw data from the block. +- `sol-transactions`: Creates a Substreams that extracts Solana transactions filtered by one or more Program IDs and Account IDs. + +## Solana Foundational Modules + +The `sol-transactions` path of the codegen, which filters the transactions, relies on the [Solana Foundational Modules](https://github.com/streamingfast/substreams-foundational-modules/tree/develop/solana-common). A Foundational Module extracts the most relevant data from blockchain, so that you don't have to code it yourself. + +Specifically, the `sol-transactions` path uses the [filtered_transactions_without_votes](https://github.com/streamingfast/substreams-foundational-modules/blob/develop/solana-common/substreams-v0.2.0.yaml#L49) module, which accepts a regex (regular expression) as input to filter the transactions. + +
\ No newline at end of file diff --git a/docs/new/references/command-line-interface.md b/docs/new/references/command-line-interface.md index c7ab0519..08fe4db9 100644 --- a/docs/new/references/command-line-interface.md +++ b/docs/new/references/command-line-interface.md @@ -21,6 +21,27 @@ The `substreams` CLI exposes many commands to developers enabling a range of fea If you choose to not use it, make sure that you are in a directory that contains a substreams.yaml file. Otherwise, you will get a usage error back. {% endhint %} +### **`init`** + +The `init` command allows you to initialize a Substreams project for several blockchains. It is a conversational-like command: you will be asked several questions and a project with the specicied features will be created for you. + +The options included in the `init` command will evolve over time, but every blockchain should, at least, contain one option. + +```bash +substreams init +``` + +### **`build`** + +The `build` command: +- Generates the necessary Protobufs specified in the `substreams.yaml` file. +- Compiles the Rust code. +- Creates a Substreams package file (`.spkg`). + +```bash +substreams build +``` + ### **`run`** The `run` command connects to a Substreams endpoint and begins processing data. @@ -111,7 +132,7 @@ When you run a `gui` session, a file called `replay.log` gets written with the c You can reload the data without hitting the server again using `--replay`. The data is immediately reloaded in the GUI, ready for more inspection. -### `pack` +### `pack` **(DEPRECATED: use `build` instead) The `pack` command builds a shippable, importable package from a `substreams.yaml` manifest file. @@ -220,39 +241,32 @@ modules { ``` {% endcode %} -### Help +### **`codegen`** -To view a list of available commands and brief explanations in the `substreams` CLI, run the `substreams` command in a terminal passing the `-h` flag. You can use this help reference at any time. +The `codegen` command generates a code for a specific sink taking as a Substreams module as input. There are two code generators available: `subgraph` and `sql`. + +- Subgraph + +Generates a Substreams-powered Subgraph project from the Substreams package found in the current folder. -{% code title="help option" overflow="wrap" %} ```bash -substreams -h +substreams codegen subgraph ``` -{% endcode %} -The output of the `help` command will print a message resembling: +- SQL + +Generates a SQL-based Substreams project from the Substreams package found in the current folder. -{% code title="help output" overflow="wrap" %} ```bash -Usage: - substreams [command] - -Available Commands: - completion Generate the autocompletion script for the specified shell - decode - graph Generate mermaid-js graph document - help Help about any command - info Display package modules and docs - inspect Display low-level package structure - pack Build an .spkg out of a .yaml manifest - protogen Generate Rust bindings from a package - run Stream modules from a given package on a remote endpoint - tools Developer tools related to substreams - -Flags: - -h, --help help for substreams - -v, --version version for substreams - -Use "substreams [command] --help" for more information about a command. +substreams codegen sql ``` -{% endcode %} + +### Help + +To view a list of available commands and brief explanations in the `substreams` CLI, run the `substreams` command in a terminal passing the `-h` flag. You can use this help reference at any time. + +{% code title="help option" overflow="wrap" %} +```bash +substreams -h +``` +{% endcode %} \ No newline at end of file