Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: single reqwest::Client for HTTP #1071

Merged
merged 4 commits into from
Jan 7, 2025

Conversation

de-sh
Copy link
Contributor

@de-sh de-sh commented Jan 6, 2025

Fixes #XXXX.

Description

Use a single HTTP client to perform all normal outgoing HTTP requests from the parseable server.

Connection timeouts in 3s during setup, 10s per request and every 90s when not used/idling. Max 32 idle connections are allowed per host. gzip/brotli compression is enabled and can be used based on server(to which parseable connects) preference, rustls is used to perform TLS encryption (instead of native-tls) and only HTTP/1.1 connections are allowed (no HTTP/2)

This is different from the default client which has no timeouts, default pooling, no compression, and would support multiple TLS backends and HTTP versions.


This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

@@ -73,6 +73,8 @@ relative-path = { version = "1.7", features = ["serde"] }
reqwest = { version = "0.11.27", default-features = false, features = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice to have the latest version 0.12.x if there is no blocker issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will take this up in #1075

Copy link
Contributor Author

@de-sh de-sh Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actix_http:latest and reqwest:v0.11+ are incompatible because they depend on different versions of http(ref1, ref2).

@coveralls
Copy link

coveralls commented Jan 6, 2025

Pull Request Test Coverage Report for Build 12651787773

Details

  • 0 of 33 (0.0%) changed or added relevant lines in 7 files are covered.
  • 4 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.002%) to 11.924%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/handlers/http/mod.rs 0 1 0.0%
src/metrics/prom_utils.rs 0 1 0.0%
src/analytics.rs 0 2 0.0%
src/audit.rs 0 2 0.0%
src/handlers/http/cluster/utils.rs 0 3 0.0%
src/handlers/http/cluster/mod.rs 0 11 0.0%
src/lib.rs 0 13 0.0%
Files with Coverage Reduction New Missed Lines %
src/handlers/http/cluster/mod.rs 4 0.0%
Totals Coverage Status
Change from base Build 12650241806: 0.002%
Covered Lines: 2312
Relevant Lines: 19389

💛 - Coveralls

Copy link
Contributor

@nikhilsinhaparseable nikhilsinhaparseable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@nitisht
Copy link
Member

nitisht commented Jan 7, 2025

@de-sh can you resolve the conflicts pls

@nitisht nitisht merged commit 9122158 into parseablehq:main Jan 7, 2025
9 checks passed
@de-sh de-sh deleted the single-client branch January 9, 2025 10:13
parmesant pushed a commit to parmesant/parseable that referenced this pull request Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants