Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the operation steps of RabbitMQ source connector. #839

Merged
merged 26 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
9babbca
The sample code of transaction is missing ".build()"
Mar 4, 2024
652c71d
Update admin-api-schemas.md
zhaoyajun2009 Mar 4, 2024
7a4ab54
Update txn-use.md
zhaoyajun2009 Mar 4, 2024
9d4dfbd
Add the opration steps of RabbitMQ source connector.
Mar 11, 2024
126b50a
Add the operation steps of RabbitMQ source connector.
Mar 11, 2024
d492cb7
Add the operation steps of RabbitMQ source connector.
Mar 11, 2024
221fadc
Merge branch 'rabbitmq-connector' of https://github.com/zhaoyajun2009…
Mar 11, 2024
c3c63f1
Add the operation steps of RabbitMQ source connector.
Mar 11, 2024
1a2702a
Update admin-api-schemas.md
zhaoyajun2009 Mar 11, 2024
4a8e9d5
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
74a648d
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
e271643
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
f0281bf
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
5919ec7
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
72dc674
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
81213c6
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
2dd4c07
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
077420b
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
6299fce
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
e5663b9
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
9551955
Update docs/io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
d31918b
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
ecb5fed
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
e6fb0e7
Merge branch 'main' into rabbitmq-connector
visortelle Mar 12, 2024
b2708e9
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
dfc6f2c
Update io-rabbitmq-source.md
zhaoyajun2009 Mar 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions docs/io-rabbitmq-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,104 @@ Before using the RabbitMQ source connector, you need to create a configuration f
passive: "false"
```

## Usage


### Standalone mode

This example describes how to use the RabbitMQ source connector to feed data from RabbitMQ and write data to Pulsar topics in the standalone mode.

#### Prerequisites

- There is a RabbitMQ server with some history messages in the queue.

#### Steps

1. Get a Pulsar package and start Pulsar in standalone mode.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/apache-pulsar-@pulsar:[email protected]
tar xvfz apache-pulsar-@pulsar:[email protected]
cd apache-pulsar-@pulsar:version@
bin/pulsar standalone
```

2. Download the [nar package](https://archive.apache.org/dist/pulsar/) corresponding to Pulsar's version and copy the following file to Pulsar's directory.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/connectors/pulsar-io-rabbitmq-@pulsar:[email protected]
cp pulsar-io-rabbitmq-@pulsar:[email protected] ./connectors
```

3. Messages published to a topic lacking at least one durable subscription are automatically marked as ready for deletion by default. We can set a retention policy at the namespace level to prevent this.

```bash
./bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default
```


4. Prepare a configuration file with name is _rabbitmq-source-queue-name.yaml_.
```yaml
configs:
host: "localhost"
port: 5672
virtualHost: "/"
username: "guest"
password: "guest"
queueName: "test-queue"
connectionName: "test-connection"
requestedChannelMax: 0
requestedFrameMax: 0
connectionTimeout: 60000
handshakeTimeout: 10000
requestedHeartbeat: 60
prefetchCount: 0
prefetchGlobal: "false"
passive: "false"
```

Copy the configuration file to Pulsar‘s conf directory.
```bash
cp rabbitmq-source-queue-name.yaml ./conf
```

5. Open a new terminal window and start the connector in local run mode.

```bash
./bin/pulsar-admin source localrun \
--source-config-file conf/rabbitmq-source-queue-name.yaml \
--archive connectors/pulsar-io-rabbitmq-@pulsar:[email protected] \
--name rabbitmq-source \
--destination-topic-name pulsar-rabbitmq-test-topic \
--broker-service-url pulsar://{ip}:{port}
```

6. Open a new terminal window and check the topic is created automatically.

```bash
./bin/pulsar-admin topics list public/default \
```

This topic is created automatically as follows:

```bash
persistent://public/default/pulsar-rabbitmq-test-topic-partition-0
```

7. Consume this topic.

```bash
./bin/pulsar-client consume persistent://public/default/pulsar-rabbitmq-test-topic-partition-0 -s s1 -n 0 -p Earliest
```

The following information appears on the consumer terminal window.

```bash
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 0
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 1

... ...

```
101 changes: 101 additions & 0 deletions versioned_docs/version-2.10.x/io-rabbitmq-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,104 @@ Before using the RabbitMQ source connector, you need to create a configuration f

```

## Usage


### Standalone mode

This example describes how to use the RabbitMQ source connector to feed data from RabbitMQ and write data to Pulsar topics in the standalone mode.

#### Prerequisites

- There is a RabbitMQ server with some history messages in the queue.

#### Steps

1. Get a Pulsar package and start Pulsar in standalone mode.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/apache-pulsar-@pulsar:[email protected]
tar xvfz apache-pulsar-@pulsar:[email protected]
cd apache-pulsar-@pulsar:version@
bin/pulsar standalone
```

2. Download the [nar package](https://archive.apache.org/dist/pulsar/) corresponding to Pulsar's version and copy the following file to Pulsar's directory.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/connectors/pulsar-io-rabbitmq-@pulsar:[email protected]
cp pulsar-io-rabbitmq-@pulsar:[email protected] ./connectors
```

3. Messages published to a topic lacking at least one durable subscription are automatically marked as ready for deletion by default. We can set a retention policy at the namespace level to prevent this.

```bash
./bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default
```


4. Prepare a configuration file with name is _rabbitmq-source-queue-name.yaml_.
```yaml
configs:
host: "localhost"
port: 5672
virtualHost: "/"
username: "guest"
password: "guest"
queueName: "test-queue"
connectionName: "test-connection"
requestedChannelMax: 0
requestedFrameMax: 0
connectionTimeout: 60000
handshakeTimeout: 10000
requestedHeartbeat: 60
prefetchCount: 0
prefetchGlobal: "false"
passive: "false"
```

Copy the configuration file to Pulsar‘s conf directory.
```bash
cp rabbitmq-source-queue-name.yaml ./conf
```

5. Open a new terminal window and start the connector in local run mode.

```bash
./bin/pulsar-admin source localrun \
--source-config-file conf/rabbitmq-source-queue-name.yaml \
--archive connectors/pulsar-io-rabbitmq-@pulsar:[email protected] \
--name rabbitmq-source \
--destination-topic-name pulsar-rabbitmq-test-topic \
--broker-service-url pulsar://{ip}:{port}
```

6. Open a new terminal window and check the topic is created automatically.

```bash
./bin/pulsar-admin topics list public/default \
```

This topic is created automatically as follows:

```bash
persistent://public/default/pulsar-rabbitmq-test-topic-partition-0
```

7. Consume this topic.

```bash
./bin/pulsar-client consume persistent://public/default/pulsar-rabbitmq-test-topic-partition-0 -s s1 -n 0 -p Earliest
```

The following information appears on the consumer terminal window.

```bash
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 0
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 1

... ...

```
101 changes: 101 additions & 0 deletions versioned_docs/version-2.11.x/io-rabbitmq-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,104 @@ Before using the RabbitMQ source connector, you need to create a configuration f
passive: "false"
```

## Usage


### Standalone mode

This example describes how to use the RabbitMQ source connector to feed data from RabbitMQ and write data to Pulsar topics in the standalone mode.

#### Prerequisites

- There is a RabbitMQ server with some history messages in the queue.

#### Steps

1. Get a Pulsar package and start Pulsar in standalone mode.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/apache-pulsar-@pulsar:[email protected]
tar xvfz apache-pulsar-@pulsar:[email protected]
cd apache-pulsar-@pulsar:version@
bin/pulsar standalone
```

2. Download the [nar package](https://archive.apache.org/dist/pulsar/) corresponding to Pulsar's version and copy the following file to Pulsar's directory.

```bash
wget https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/connectors/pulsar-io-rabbitmq-@pulsar:[email protected]
cp pulsar-io-rabbitmq-@pulsar:[email protected] ./connectors
```

3. Messages published to a topic lacking at least one durable subscription are automatically marked as ready for deletion by default. We can set a retention policy at the namespace level to prevent this.

```bash
./bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default
```


4. Prepare a configuration file with name is _rabbitmq-source-queue-name.yaml_.
```yaml
configs:
host: "localhost"
port: 5672
virtualHost: "/"
username: "guest"
password: "guest"
queueName: "test-queue"
connectionName: "test-connection"
requestedChannelMax: 0
requestedFrameMax: 0
connectionTimeout: 60000
handshakeTimeout: 10000
requestedHeartbeat: 60
prefetchCount: 0
prefetchGlobal: "false"
passive: "false"
```

Copy the configuration file to Pulsar‘s conf directory.
```bash
cp rabbitmq-source-queue-name.yaml ./conf
```

5. Open a new terminal window and start the connector in local run mode.

```bash
./bin/pulsar-admin source localrun \
--source-config-file conf/rabbitmq-source-queue-name.yaml \
--archive connectors/pulsar-io-rabbitmq-@pulsar:[email protected] \
--name rabbitmq-source \
--destination-topic-name pulsar-rabbitmq-test-topic \
--broker-service-url pulsar://{ip}:{port}
```

6. Open a new terminal window and check the topic is created automatically.

```bash
./bin/pulsar-admin topics list public/default \
```

This topic is created automatically as follows:

```bash
persistent://public/default/pulsar-rabbitmq-test-topic-partition-0
```

7. Consume this topic.

```bash
./bin/pulsar-client consume persistent://public/default/pulsar-rabbitmq-test-topic-partition-0 -s s1 -n 0 -p Earliest
```

The following information appears on the consumer terminal window.

```bash
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 0
----- got message -----
key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 1

... ...

```
Loading