Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Doc] Use absolute path when installing functions or using localrun #877

Merged
merged 5 commits into from
Apr 3, 2024
Merged

[Doc] Use absolute path when installing functions or using localrun #877

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a5741ce
Use absolute path when installing functions
lhotari Apr 2, 2024
a1e8ba3
Fix other relative paths
lhotari Apr 2, 2024
8bd0e69
Address a few more locations
lhotari Apr 2, 2024
85f615d
Fix even more relative paths and docs
lhotari Apr 2, 2024
32b9dc3
Address review commend: remove mandatory sentence for classname
lhotari Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/admin-api-functions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ pulsar-admin functions create \
--inputs test-input-topic \
--output persistent://public/default/test-output-topic \
--classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
--jar /examples/api-examples.jar
--jar $PWD/examples/api-examples.jar
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is $PWD a safe assumption as to the proper location of the artifact, or should we use a more explicit nomenclature such as path/to/examples/api-examples.jar ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think $PWD gives a good hint that it's an absolute path. At least I have found it useful with pulsar-admin when I have the files in the current directory and I want to reference the files with an absolute path. I guess there are also other ways. Perhaps it could be explained to the user that $PWD is the current working directory. They will also find out if they try it or google it. :)
In this particular case, the original /examples/api-examples.jar doesn't seem to make much sense. That's why I added the $PWD prefix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /path/to/file.jar type of examples have the problem that for the user, it's really a lot of extra steps to find out the absolute path for the file arguments. That's why I prefer $PWD/file.jar since it's simple and if someone doesn't already know it, they have the chance to learn it.

$PWD is for a unix shell, but the examples are already assuming a unix shell since the syntax is all unix like.

```

</TabItem>
Expand Down
2 changes: 1 addition & 1 deletion docs/admin-api-schemas.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ To extract (provide) a schema via a topic, use the following method.
Use the `extract` subcommand.

```bash
pulsar-admin schemas extract --classname <class-name> --jar <jar-path> --type <type-name>
pulsar-admin schemas extract --classname <class-name> --jar <absolute-jar-path> --type <type-name>
```

</TabItem>
Expand Down
2 changes: 1 addition & 1 deletion docs/deploy-bare-metal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ Create an ExclamationFunction `exclamation`.

```bash
bin/pulsar-admin functions create \
--jar examples/api-examples.jar \
--jar $PWD/examples/api-examples.jar \
--classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
--inputs persistent://public/default/exclamation-input \
--output persistent://public/default/exclamation-output \
Expand Down
6 changes: 3 additions & 3 deletions docs/functions-cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,9 @@ You can configure a function by using a predefined YAML file. The following tabl
| fqfn | String | `--fqfn` | The Fully Qualified Function Name (FQFN) of a function. |
| windowConfig | [WindowConfig](#windowconfig) | N/A | N/A |
| timeoutMs | Long | `--timeout-ms` | The message timeout (in milliseconds). |
| jar | String | `--jar` | The path of the JAR file for a function (written in Java). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| py | String | `--py` | The path of the main Python/Python wheel file for a function (written in Python). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| go | String | `--go` | Path to the main Go executable binary for the function (written in Go). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| jar | String | `--jar` | The absolute path of the JAR file for a function (written in Java). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| py | String | `--py` | The absolute path of the main Python/Python wheel file for a function (written in Python). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| go | String | `--go` | The absolute path of the main Go executable binary for the function (written in Go). It also supports URL paths that workers can download the package from, including HTTP, HTTPS, file (file protocol assuming that file already exists on worker host), and function (package URL from packages management service). |
| cleanupSubscription | Boolean | `--cleanup-subscription` | Whether the subscriptions that a function creates or uses should be deleted or not when the function is deleted. The default value is `true` |
| customRuntimeOptions | String | `--custom-runtime-options` | A string that encodes options to customize the runtime. |
| maxPendingAsyncRequests | Int | `--max-message-retries` | The max number of pending async requests per instance to avoid a large number of concurrent requests. |
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-deploy-arguments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Take the `create` command for example. The following function has default values

```bash
bin/pulsar-admin functions create \
--jar my-pulsar-functions.jar \
--jar $PWD/my-pulsar-functions.jar \
--classname org.example.MyFunction \
--inputs my-function-input-topic1,my-function-input-topic2
```
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-deploy-cluster-parallelism.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,6 @@ For an existing function, you can adjust the parallelism by using the `update` c

```bash
bin/pulsar-admin functions update \
--function-config-file function-config.yaml
--function-config-file $PWD/function-config.yaml
```

2 changes: 1 addition & 1 deletion docs/functions-deploy-cluster-resource.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ For example, the following command allocates 8 cores, 8GB of RAM, and 10GB of di

```bash
bin/pulsar-admin functions create \
--jar target/my-functions.jar \
--jar $PWD/target/my-functions.jar \
--classname org.example.functions.MyFunction \
--cpu 8 \
--ram 8589934592 \
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-deploy-localrun.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The following is an example of how to use the `localrun` command.

```bash
bin/pulsar-admin functions localrun \
--py myfunc.py \
--py $PWD/myfunc.py \
--classname myfunc.SomeFunction \
--inputs persistent://public/default/input-1 \
--output persistent://public/default/output-1
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-develop-admin-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ To enable your function to get access to the Pulsar admin client, you need to se

```bash
bin/pulsar-admin functions localrun \
--jar my-functions.jar \
--jar $PWD/my-functions.jar \
--classname my.package.CursorManagementFunction \
--web-service-url http://pulsar-web-service:8080 \
# Other function configs
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-develop-log.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ To enable your function to produce logs, you need to specify a log topic when cr

```bash
bin/pulsar-admin functions create \
--jar my-functions.jar \
--jar $PWD/my-functions.jar \
--classname my.package.LoggingFunction \
--log-topic persistent://public/default/logging-function-logs \
# Other function configs
Expand Down
2 changes: 1 addition & 1 deletion docs/functions-develop-tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To write a function for word count, complete the following steps.

```bash
bin/pulsar-admin functions create \
--jar target/my-jar-with-dependencies.jar \
--jar $PWD/target/my-jar-with-dependencies.jar \
--classname org.example.functions.WordCountFunction \
--tenant public \
--namespace default \
Expand Down
11 changes: 2 additions & 9 deletions docs/functions-package-go.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,11 @@ To package a Go function, complete the following steps.
go build <your Go function filename>.go
```

3. Copy the Go function file to the Pulsar image.

```bash
docker exec -it [CONTAINER ID] /bin/bash
docker cp <your go function path> CONTAINER ID:/pulsar
```

4. Run the Go function with the following command.
3. Run the Go function with the following command.

```bash
bin/pulsar-admin functions localrun \
--go [your go function path]
--go [absolute path to your go function]
--inputs [input topics] \
--output [output topic] \
--tenant [default:public] \
Expand Down
21 changes: 4 additions & 17 deletions docs/functions-package-java.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,19 +93,12 @@ To package a Java function as JAR, complete the following steps.

After the Java function is packaged, a `target` directory is created automatically. Open the `target` directory to check if there is a JAR package similar to `java-function-1.0-SNAPSHOT.jar`.

3. Copy the packaged jar file to the Pulsar image.

```bash
docker exec -it [CONTAINER ID] /bin/bash
docker cp <path of java-function-1.0-SNAPSHOT.jar> CONTAINER ID:/pulsar
```

4. Run the Java function using the following command.
3. Run the Java function using the following command.

```bash
./bin/pulsar-admin functions localrun \
--classname org.example.test.ExclamationFunction \
--jar java-function-1.0-SNAPSHOT.jar \
--jar $PWD/target/java-function-1.0-SNAPSHOT.jar \
--inputs persistent://public/default/my-topic-1 \
--output persistent://public/default/test-1 \
--tenant public \
Expand Down Expand Up @@ -183,17 +176,11 @@ To package a Java function as NAR, complete the following steps.

After the Java function is packaged, a `target` directory is created automatically. Open the `target` directory to check if there is a NAR package similar to `java-function-1.0-SNAPSHOT.nar`.

3. Copy the packaged NAR file to the Pulsar image.

```bash
docker cp <path of java-function-1.0-SNAPSHOT.nar> CONTAINER ID:/pulsar
```

4. Run the Java function using the following command.
3. Run the Java function using the following command.

```bash
./bin/pulsar-admin functions localrun \
--jar java-function-1.0-SNAPSHOT.nar \
--jar $PWD/target/java-function-1.0-SNAPSHOT.nar \
--inputs persistent://public/default/my-topic-1 \
--output persistent://public/default/test-1 \
--tenant public \
Expand Down
4 changes: 2 additions & 2 deletions docs/functions-package-python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ To package a Python function into **one Python file**, complete the following st
```bash
./bin/pulsar-admin functions localrun \
--classname <Python Function file name>.<Python Function class name> \
--py <path of Python Function file> \
--py <absolute path of Python Function file> \
--inputs persistent://public/default/my-topic-1 \
--output persistent://public/default/test-1 \
--tenant public \
Expand Down Expand Up @@ -109,7 +109,7 @@ To package a Python function into a **ZIP file**, complete the following steps.
```shell
./bin/pulsar-admin functions localrun \
--classname exclamation \
--py <path of ZIP file> \
--py <absolute path of ZIP file> \
--inputs persistent://public/default/in-topic \
--output persistent://public/default/out-topic \
--tenant public \
Expand Down
12 changes: 6 additions & 6 deletions docs/functions-quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ Before starting functions, you need to [start Pulsar](#start-standalone-pulsar)

```bash
bin/pulsar-admin functions create \
--function-config-file examples/example-function-config.yaml \
--jar examples/api-examples.jar
--function-config-file $PWD/examples/example-function-config.yaml \
--jar $PWD/examples/api-examples.jar
```

**Output**
Expand Down Expand Up @@ -295,8 +295,8 @@ The following example provides instructions to start a stateful function to vali

```bash
bin/pulsar-admin functions create \
--function-config-file examples/example-stateful-function-config.yaml \
--jar examples/api-examples.jar
--function-config-file $PWD/examples/example-stateful-function-config.yaml \
--jar $PWD/examples/api-examples.jar
```

**Output**
Expand Down Expand Up @@ -480,8 +480,8 @@ The following example provides instructions to start a window function to calcul

```bash
bin/pulsar-admin functions create \
--function-config-file examples/example-window-function-config.yaml \
--jar examples/api-examples.jar
--function-config-file $PWD/examples/example-window-function-config.yaml \
--jar $PWD/examples/api-examples.jar
```

**Output**
Expand Down
2 changes: 1 addition & 1 deletion docs/io-canal-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ Here is an example of storing MySQL data using the configuration file as above.
docker exec -it pulsar-standalone /bin/bash
wget https://archive.apache.org/dist/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar -P connectors
./bin/pulsar-admin source localrun \
--archive ./connectors/pulsar-io-canal-2.3.0.nar \
--archive $PWD/connectors/pulsar-io-canal-2.3.0.nar \
--classname org.apache.pulsar.io.canal.CanalStringSource \
--tenant public \
--namespace default \
Expand Down
14 changes: 7 additions & 7 deletions docs/io-cdc-debezium.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The configuration of the Debezium source connector has the following properties.
| `database.history` | true | null | The name of the database history class. |
| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.** |
| `database.history.pulsar.service.url` | false| null | Pulsar cluster service URL for history topic. <br /><br />**Note**: If `database.history.pulsar.service.url` is not set, then the database history Pulsar client will use the same client settings as those of the source connector, such as `client_auth_plugin` and `client_auth_params`.|
| `pulsar.service.url` | true | null | Pulsar cluster service URL for the offset topic used in Debezium. You can use the `bin/pulsar-admin --admin-url http://pulsar:8080 sources localrun --source-config-file configs/pg-pulsar-config.yaml` command to point to the target Pulsar cluster.|
| `pulsar.service.url` | true | null | Pulsar cluster service URL for the offset topic used in Debezium. You can use the `bin/pulsar-admin --admin-url http://pulsar:8080 sources localrun --source-config-file $PWD/configs/pg-pulsar-config.yaml` command to point to the target Pulsar cluster.|
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). |
| `mongodb.name` | true | null | A unique name that identifies the connector and/or MongoDB replica set or shared cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster. |
Expand Down Expand Up @@ -138,7 +138,7 @@ This example shows how to change the data of a MySQL table using the Pulsar Debe

```bash
bin/pulsar-admin source localrun \
--archive connectors/pulsar-io-debezium-mysql-@pulsar:[email protected] \
--archive $PWD/connectors/pulsar-io-debezium-mysql-@pulsar:[email protected] \
--name debezium-mysql-source \
--tenant public \
--namespace default \
Expand All @@ -149,7 +149,7 @@ This example shows how to change the data of a MySQL table using the Pulsar Debe

```bash
bin/pulsar-admin source localrun \
--source-config-file debezium-mysql-source-config.yaml
--source-config-file $PWD/debezium-mysql-source-config.yaml
```

4. Subscribe to the topic _sub-products_ for the table _inventory.products_.
Expand Down Expand Up @@ -289,7 +289,7 @@ This example shows how to change the data of a PostgreSQL table using the Pulsar

```bash
bin/pulsar-admin source localrun \
--archive connectors/pulsar-io-debezium-postgres-@pulsar:[email protected] \
--archive $PWD/connectors/pulsar-io-debezium-postgres-@pulsar:[email protected] \
--name debezium-postgres-source \
--tenant public \
--namespace default \
Expand All @@ -300,7 +300,7 @@ This example shows how to change the data of a PostgreSQL table using the Pulsar

```bash
bin/pulsar-admin source localrun \
--source-config-file debezium-postgres-source-config.yaml
--source-config-file $PWD/debezium-postgres-source-config.yaml
```

4. Subscribe to the topic _sub-products_ for the _inventory.products_ table.
Expand Down Expand Up @@ -431,7 +431,7 @@ This example shows how to change the data of a MongoDB table using the Pulsar De

```bash
bin/pulsar-admin source localrun \
--archive connectors/pulsar-io-debezium-mongodb-@pulsar:[email protected] \
--archive $PWD/connectors/pulsar-io-debezium-mongodb-@pulsar:[email protected] \
--name debezium-mongodb-source \
--tenant public \
--namespace default \
Expand All @@ -442,7 +442,7 @@ This example shows how to change the data of a MongoDB table using the Pulsar De

```bash
bin/pulsar-admin source localrun \
--source-config-file debezium-mongodb-source-config.yaml
--source-config-file $PWD/debezium-mongodb-source-config.yaml
```

4. Subscribe to the topic _sub-products_ for the _inventory.products_ table.
Expand Down
Loading
Loading