Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Rust server and client for get_simple example #8

Merged
merged 8 commits into from
Mar 11, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions http/get_simple/rs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

/target
Cargo.lock
27 changes: 27 additions & 0 deletions http/get_simple/rs/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[workspace]
resolver = "2"
members = ["client", "server"]

[workspace.dependencies]
arrow-array = "50.0.0"
arrow-ipc = "50.0.0"
arrow-schema = "50.0.0"
tracing = "0.1.40"
tracing-subscriber = "0.3.18"
26 changes: 26 additions & 0 deletions http/get_simple/rs/client/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[package]
name = "client"
version = "0.1.0"
edition = "2021"

[dependencies]
arrow-ipc.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
32 changes: 32 additions & 0 deletions http/get_simple/rs/client/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# HTTP GET Arrow Data: Simple Rust Client Example

This directory contains a minimal example of an HTTP client implemented in Rust. The client:

1. Sends an HTTP GET request to a server.
2. Receives an HTTP 200 response from the server, with the response body containing an Arrow IPC stream of record batches.
3. Adds the record batches to a list as they are received.

To run this example, first start one of the server examples in the parent directory, then:

```sh
cargo r --release
```
mbrobbel marked this conversation as resolved.
Show resolved Hide resolved
106 changes: 106 additions & 0 deletions http/get_simple/rs/client/src/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

use arrow_ipc::reader::StreamReader;
use std::{
io::{BufRead, BufReader, Read, Write},
net::{IpAddr, Ipv4Addr, SocketAddr, TcpStream},
ianmcook marked this conversation as resolved.
Show resolved Hide resolved
};
use tracing::{error, info, info_span};
use tracing_subscriber::fmt::format::FmtSpan;

fn main() {
// Configure tracing subscriber.
tracing_subscriber::fmt()
.with_span_events(FmtSpan::CLOSE)
.init();

info_span!("get_simple").in_scope(|| {
// Connect to server.
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 8000);
match TcpStream::connect(addr) {
Ok(mut stream) => {
info_span!("Reading Arrow IPC stream", %addr).in_scope(|| {
info!("Connected");

// Send request.
stream
.write_all(format!("GET / HTTP/1.1\r\nHost: {addr}\r\n\r\n").as_bytes())
.unwrap();

// Ignore response header.
let mut reader = BufReader::new(&mut stream);
let mut chunked = false;
loop {
let mut line = String::default();
reader.read_line(&mut line).unwrap();
if let Some(("transfer-encoding", "chunked")) = line
.to_lowercase()
.split_once(':')
.map(|(key, value)| (key.trim(), value.trim()))
{
chunked = true;
}
if line == "\r\n" {
break;
}
}
Comment on lines +47 to +61
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels a bit sad we are manually parsing headers and such. Makes the example a bit harder to follow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100% agreed. I started with hyper but moved to std because of apache/arrow-rs#1207.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Of all the examples here, this is the only one that directly implements low-level HTTP details instead of depending on a library to do that.

But the main objective of these get_simple examples is only to ensure that the IPC implementations in the mainstream Arrow libraries can function and interoperate over HTTP. This achieves that objective.


// Read Arrow IPC stream
let batches: Vec<_> = if chunked {
let mut buffer = Vec::default();
loop {
// Chunk size
let mut line = String::default();
reader.read_line(&mut line).unwrap();
let chunk_size = u64::from_str_radix(line.trim(), 16).unwrap();

if chunk_size == 0 {
// Terminating chunk
break;
} else {
// Append chunk to buffer
let mut chunk_reader = reader.take(chunk_size);
chunk_reader.read_to_end(&mut buffer).unwrap();
// Terminating CR-LF sequence
reader = chunk_reader.into_inner();
reader.read_line(&mut String::default()).unwrap();
}
}
StreamReader::try_new_unbuffered(buffer.as_slice(), None)
.unwrap()
.flat_map(Result::ok)
.collect()
} else {
StreamReader::try_new_unbuffered(reader, None)
.unwrap()
.flat_map(Result::ok)
.collect()
};

info!(
batches = batches.len(),
rows = batches.iter().map(|rb| rb.num_rows()).sum::<usize>()
);
});
}
Err(error) => {
error!(%error, "Connection failed")
}
}
})
}
31 changes: 31 additions & 0 deletions http/get_simple/rs/server/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[package]
name = "server"
version = "0.1.0"
edition = "2021"

[dependencies]
arrow-array.workspace = true
arrow-ipc.workspace = true
arrow-schema.workspace = true
once_cell = "1.19.0"
rand = "0.8.5"
rayon = "1.9.0"
tracing.workspace = true
tracing-subscriber.workspace = true
32 changes: 32 additions & 0 deletions http/get_simple/rs/server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# HTTP GET Arrow Data: Simple Rust Server Example

This directory contains a minimal example of an HTTP server implemented in Rust. The server:

1. Creates a list of record batches and populates it with synthesized data.
2. Listens for HTTP requests from clients.
3. Upon receiving a request, sends an HTTP 200 response with the body containing an Arrow IPC stream of record batches.

To run this example:

```sh
cargo r --release
```
mbrobbel marked this conversation as resolved.
Show resolved Hide resolved
Loading