Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connection reset by peer on AWS S3 object store. #10478

Open
Smotrov opened this issue May 13, 2024 · 3 comments
Open

Connection reset by peer on AWS S3 object store. #10478

Smotrov opened this issue May 13, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@Smotrov
Copy link

Smotrov commented May 13, 2024

Describe the bug

I'm trying to read 6GB table of compressed NDJSON data from S3. The data is compressed with ZStd with about x100 compression ratio. Files are stored HIVE partitioned and have an average size of 25Mb.

Whatever I do it can not read the data and ends up with error Error: Custom { kind: Other, error: External(External(ArrowError(ExternalError(IoError(Custom { kind: Other, error: ObjectStore(Generic { store: "S3", source: reqwest::Error { kind: Body, source: hyper::Error(Body, Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" }) } }) })), None))) }

To Reproduce

Try to read huge Zstd comprised NDJSON from S3

Expected behavior

Would be grate if it can read the data.

Additional context

I use following object store configuration

fn create_object_store() -> Result<Arc<dyn object_store::ObjectStore>> {
    // AWS configuration

    let client_options =
        ClientOptions::default()
        .with_timeout(std::time::Duration::from_secs(60 * 60 * 24 * 7))
        .with_allow_http(true)
        .with_allow_http2();

    // Set up S3 object store

    let retry_config =     RetryConfig {
        backoff: BackoffConfig::default(),
        max_retries: 10,
        retry_timeout: std::time::Duration::from_secs(10),
    };


    let s3 = AmazonS3Builder::new()
        .with_bucket_name(BUCKET_NAME)
        .with_region(REGION)
        .with_access_key_id(env::var("AWS_ACCESS_KEY_ID").unwrap())
        .with_secret_access_key(env::var("AWS_SECRET_ACCESS_KEY").unwrap())
        .with_client_options(client_options)
        .with_retry(retry_config)
        .build()?;

    Ok(Arc::new(s3))
}

Listing option is following

    // Define the partitioned Listing Table
    let listing_options = ListingOptions::new(file_format)
        .with_table_partition_cols(part)
        .with_target_partitions(1)
        .with_file_extension(".ndjson.zst");

    ctx.register_listing_table(
        "data",
        format!("s3://{BUCKET_NAME}/data_lake/data_warehouse/"),
        listing_options,
        Some(schema),
        None,
    )
    .await?;
Here is a tail of debug output 
GlobalLimitExec: skip=0, fetch=10
SortExec: TopK(fetch=10), expr=[dep@0 ASC NULLS LAST], preserve_partitioning=[false]
  AggregateExec: mode=FinalPartitioned, gby=[dep@0 as dep], aggr=[]
    AggregateExec: mode=Partial, gby=[dep@0 as dep], aggr=[]
      ProjectionExec: expr=[substr(OriginalRequest@0, 9, 3) as dep]
        JsonExec: file_groups={1 group: [[data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_0vKzsrkfADbgpfaRqrRBvH4IERFrgWLR.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_2MafWEf7BWqFlK9kHJV6DL3u4DxjRpzf.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_57xEEEf5EoOQnGKF6cci9lRAYd6t9bKM.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_D8EsUpB0M7RBXArLdBkRZfvTwXtMUxBX.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_G3oTIZoB1Q4OIWLKuGfSJ6QKN647tVTy.ndjson.zst, ...]]}, projection=[OriginalRequest]

  
2024-05-13T07:16:39.061495Z DEBUG datafusion::physical_planner: Optimized physical plan:
GlobalLimitExec: skip=0, fetch=10
SortPreservingMergeExec: [dep@0 ASC NULLS LAST], fetch=10
  SortExec: TopK(fetch=10), expr=[dep@0 ASC NULLS LAST], preserve_partitioning=[true]
    AggregateExec: mode=FinalPartitioned, gby=[dep@0 as dep], aggr=[]
      CoalesceBatchesExec: target_batch_size=8192
        RepartitionExec: partitioning=Hash([dep@0], 10), input_partitions=10
          AggregateExec: mode=Partial, gby=[dep@0 as dep], aggr=[]
            ProjectionExec: expr=[substr(OriginalRequest@0, 9, 3) as dep]
              RepartitionExec: partitioning=RoundRobinBatch(10), input_partitions=1
                JsonExec: file_groups={1 group: [[data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_0vKzsrkfADbgpfaRqrRBvH4IERFrgWLR.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_2MafWEf7BWqFlK9kHJV6DL3u4DxjRpzf.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_57xEEEf5EoOQnGKF6cci9lRAYd6t9bKM.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_D8EsUpB0M7RBXArLdBkRZfvTwXtMUxBX.ndjson.zst, data_lake/data_warehouse/partition=robotic/pos=FR/pcc=8W2M/year=2024/month=05/day=09/hour=00/robotic_FR_8W2M_G3oTIZoB1Q4OIWLKuGfSJ6QKN647tVTy.ndjson.zst, ...]]}, projection=[OriginalRequest]

  
2024-05-13T07:16:39.061728Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062834Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062842Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062851Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062859Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062906Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062918Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062926Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062928Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062934Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062945Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062953Z DEBUG datafusion_physical_plan::sorts::sort_preserving_merge: Done setting up sender-receiver for SortPreservingMergeExec::execute    
2024-05-13T07:16:39.062960Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062963Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062964Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062970Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062975Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062977Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062984Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062971Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.062991Z DEBUG datafusion_physical_plan::aggregates::row_hash: Creating GroupedHashAggregateStream    
2024-05-13T07:16:39.063137Z DEBUG datafusion_physical_plan::sorts::sort_preserving_merge: Got stream result from SortPreservingMergeStream::new_from_receivers    
2024-05-13T07:16:39.063151Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:39.063207Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:16:39.197359Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:16:39.197387Z DEBUG hyper::proto::h1::conn: incoming body is content-length (25145569 bytes)
2024-05-13T07:16:39.199315Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:39.199468Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:16:39.365396Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:16:39.365409Z DEBUG hyper::proto::h1::conn: incoming body is content-length (29051098 bytes)
2024-05-13T07:16:43.339217Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.347246Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.347388Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.347427Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.348904Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.352269Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.352316Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.355565Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.355701Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.364136Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.399221Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.403241Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.407352Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.410921Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.416350Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.416511Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.419501Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.419528Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.424806Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.424816Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.429860Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.432870Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.450166Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.455917Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.462219Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.462250Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.467743Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.477334Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.483580Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.483652Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.485179Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.485186Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.485282Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.488639Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.488674Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.488694Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.494034Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.494089Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.494121Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:43.500552Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.974830Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.974845Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.974853Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.974945Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978768Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978799Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978832Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978890Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978899Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978910Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.978911Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.979017Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.979052Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.979068Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981598Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981629Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981646Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981676Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981701Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981704Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.981728Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.982167Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.982205Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.982235Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985265Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985312Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985360Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985364Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985381Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985395Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.985402Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.987945Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.987971Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.987981Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988702Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988735Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988761Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988799Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988829Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988853Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988906Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.988988Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989611Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989648Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989697Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989754Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989790Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989807Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989815Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.989819Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990169Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990199Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990246Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990281Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990305Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.990319Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991704Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991752Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991769Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991777Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991782Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991822Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991821Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.991880Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.992158Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.992195Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.992445Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.993418Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.993972Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.994434Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.994452Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.994458Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.995136Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.995167Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.995182Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.995922Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.997097Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.997679Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.998442Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:44.999229Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001139Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001154Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001157Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001177Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001225Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001663Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.001681Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.002210Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.002613Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.003085Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.004249Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.011255Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.015981Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.016006Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.019135Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.020274Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:16:45.020303Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:45.021245Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.035830Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:45.055773Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:16:46.270566Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:46.270772Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:16:46.425520Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:16:46.425535Z DEBUG hyper::proto::h1::conn: incoming body is content-length (23990302 bytes)
2024-05-13T07:16:54.180726Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:16:54.180764Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:56.285068Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:16:56.285245Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:16:56.420897Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:16:56.420911Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24613055 bytes)
2024-05-13T07:17:00.337078Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.337135Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.340042Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.340122Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.340268Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344103Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344167Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344107Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344196Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344200Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344214Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344871Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.344903Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.345421Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.345429Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.346994Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.347750Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.348424Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.349674Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.349692Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.350679Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.351925Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.353764Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.355346Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.355350Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.355362Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.356944Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.356974Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.362167Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.362189Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.362167Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.362201Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.362895Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.364660Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.368491Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.369062Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.372167Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.372197Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.385423Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.385420Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.394732Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.398144Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.398251Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.398278Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.400476Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.403156Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.403190Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.405954Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.408401Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.411854Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.429529Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.429552Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.434413Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.436303Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.439467Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.440587Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.441272Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.444670Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.447590Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.447614Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.449083Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.450379Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.452188Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.459346Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.459386Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.468300Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.474615Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.474625Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.478834Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.480674Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.484330Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.487461Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.487494Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.489949Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.489984Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.490005Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.492697Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.495226Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.495254Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.495259Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.505275Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.510968Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.527264Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.527264Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.594148Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:00.689667Z DEBUG hyper::client::client: client connection error: connection closed before message completed
2024-05-13T07:17:03.247674Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:03.247748Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:04.642161Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:04.642352Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:04.795309Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:04.795329Z DEBUG hyper::proto::h1::conn: incoming body is content-length (15075959 bytes)
2024-05-13T07:17:11.451370Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:11.451396Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:12.939055Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:12.939250Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:13.101945Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:13.101960Z DEBUG hyper::proto::h1::conn: incoming body is content-length (20645665 bytes)
2024-05-13T07:17:16.777778Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:16.777836Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:17.060840Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:17.061004Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:17.223377Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:17.223395Z DEBUG hyper::proto::h1::conn: incoming body is content-length (22006088 bytes)
2024-05-13T07:17:21.389714Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:21.389751Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:21.818463Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:21.818624Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:21.992174Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:21.992191Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17210606 bytes)
2024-05-13T07:17:27.612527Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:27.612554Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:28.452285Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:28.452464Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:28.662178Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:28.662193Z DEBUG hyper::proto::h1::conn: incoming body is content-length (20122883 bytes)
2024-05-13T07:17:32.917819Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:32.917892Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:33.341541Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:33.341696Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:33.496047Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:33.496061Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19839485 bytes)
2024-05-13T07:17:37.695079Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:37.695147Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:38.128650Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:38.128820Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:38.302062Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:38.302079Z DEBUG hyper::proto::h1::conn: incoming body is content-length (23493345 bytes)
2024-05-13T07:17:43.534360Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:43.534388Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:44.684875Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:44.685044Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:44.817628Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:44.817644Z DEBUG hyper::proto::h1::conn: incoming body is content-length (21772936 bytes)
2024-05-13T07:17:50.586556Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:50.586641Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:51.484463Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:51.484622Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:51.621528Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:51.621541Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24817527 bytes)
2024-05-13T07:17:56.428725Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:17:56.428786Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:56.877750Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:17:56.877913Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:17:57.080674Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:17:57.080687Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17849572 bytes)
2024-05-13T07:18:02.347299Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:02.347392Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:03.000283Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:03.000457Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:03.121178Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:03.121194Z DEBUG hyper::proto::h1::conn: incoming body is content-length (25759375 bytes)
2024-05-13T07:18:07.818483Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:07.818560Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:08.266402Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:08.266574Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:08.397167Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:08.397179Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17441581 bytes)
2024-05-13T07:18:14.425549Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:14.425662Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:15.608503Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:15.608670Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:15.736591Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:15.736605Z DEBUG hyper::proto::h1::conn: incoming body is content-length (20941855 bytes)
2024-05-13T07:18:20.370323Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:20.370404Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:20.795047Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:20.795178Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:20.950889Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:20.950899Z DEBUG hyper::proto::h1::conn: incoming body is content-length (15321498 bytes)
2024-05-13T07:18:25.763070Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:25.763113Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:26.192402Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:26.192569Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:26.321951Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:26.321962Z DEBUG hyper::proto::h1::conn: incoming body is content-length (15750268 bytes)
2024-05-13T07:18:29.694117Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:29.694146Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:29.966470Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:29.966622Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:30.122110Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:30.122126Z DEBUG hyper::proto::h1::conn: incoming body is content-length (20861623 bytes)
2024-05-13T07:18:34.402850Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:34.402877Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:34.810478Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:34.810634Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:34.972180Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:34.972194Z DEBUG hyper::proto::h1::conn: incoming body is content-length (14642929 bytes)
2024-05-13T07:18:39.710308Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:39.710339Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:40.135984Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:40.136141Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:40.262731Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:40.262746Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24244017 bytes)
2024-05-13T07:18:44.051494Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:44.051559Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:44.433984Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:44.434144Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:44.602055Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:44.602072Z DEBUG hyper::proto::h1::conn: incoming body is content-length (21476804 bytes)
2024-05-13T07:18:50.689745Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:50.689781Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:51.942204Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:51.942365Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:52.107298Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:52.107311Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24872681 bytes)
2024-05-13T07:18:57.263726Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:18:57.263815Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:57.760878Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:18:57.761047Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:18:57.895379Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:18:57.895396Z DEBUG hyper::proto::h1::conn: incoming body is content-length (23897435 bytes)
2024-05-13T07:19:03.328178Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:03.328247Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:03.833371Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:03.833585Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:04.010366Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:04.010379Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24232180 bytes)
2024-05-13T07:19:09.414719Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:09.414779Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:09.898340Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:09.898498Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:10.018387Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:10.018402Z DEBUG hyper::proto::h1::conn: incoming body is content-length (27209092 bytes)
2024-05-13T07:19:16.443689Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:16.443764Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:17.663020Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:17.663207Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:17.805127Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:17.805142Z DEBUG hyper::proto::h1::conn: incoming body is content-length (21028236 bytes)
2024-05-13T07:19:23.739590Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:23.739627Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:24.294926Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:24.295066Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:24.523839Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:24.523851Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24295227 bytes)
2024-05-13T07:19:29.217816Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:29.217886Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:29.645599Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:29.645757Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:29.807680Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:29.807697Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17131187 bytes)
2024-05-13T07:19:34.626480Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:34.626545Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:35.062090Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:35.062235Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:35.228726Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:35.228741Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17710900 bytes)
2024-05-13T07:19:39.449773Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:39.449842Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:39.857217Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:39.857360Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:40.044203Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:40.044218Z DEBUG hyper::proto::h1::conn: incoming body is content-length (12370072 bytes)
2024-05-13T07:19:44.610045Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:44.610095Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:45.054303Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:45.054463Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:45.208697Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:45.208709Z DEBUG hyper::proto::h1::conn: incoming body is content-length (16881838 bytes)
2024-05-13T07:19:48.070311Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:48.070379Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:48.328920Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:48.329082Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:48.454137Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:48.454151Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24618437 bytes)
2024-05-13T07:19:52.732994Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:52.733062Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:53.181193Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:19:53.181358Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:19:53.359456Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:19:53.359480Z DEBUG hyper::proto::h1::conn: incoming body is content-length (13607239 bytes)
2024-05-13T07:19:59.265366Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:19:59.265424Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:00.403747Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:00.403961Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:00.604866Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:00.604882Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19826594 bytes)
2024-05-13T07:20:03.338083Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:03.338112Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:03.607086Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:03.607264Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:03.751063Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:03.751079Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19856943 bytes)
2024-05-13T07:20:08.400953Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:08.400987Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:08.855041Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:08.855218Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:09.009937Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:09.009954Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19432867 bytes)
2024-05-13T07:20:13.244866Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:13.244895Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:13.669526Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:13.669698Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:13.851301Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:13.851318Z DEBUG hyper::proto::h1::conn: incoming body is content-length (20969674 bytes)
2024-05-13T07:20:18.808218Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:18.808248Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:19.515661Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:19.515819Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:19.679906Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:19.679923Z DEBUG hyper::proto::h1::conn: incoming body is content-length (21602069 bytes)
2024-05-13T07:20:24.352576Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:24.352604Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:24.991245Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:24.991403Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:25.144906Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:25.144921Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19108832 bytes)
2024-05-13T07:20:31.132202Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:31.132270Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:32.905708Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:32.905922Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:33.045752Z DEBUG hyper::proto::h1::io: parsed 16 headers
2024-05-13T07:20:33.045770Z DEBUG hyper::proto::h1::conn: incoming body is content-length (15058671 bytes)
2024-05-13T07:20:37.291684Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:37.291773Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:37.711302Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:37.711478Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:37.852531Z DEBUG hyper::proto::h1::io: parsed 16 headers
2024-05-13T07:20:37.852546Z DEBUG hyper::proto::h1::conn: incoming body is content-length (16046737 bytes)
2024-05-13T07:20:41.188036Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:41.188058Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:20:41.458919Z DEBUG reqwest::connect: starting new connection: https://s3.eu-west-1.amazonaws.com/    
2024-05-13T07:20:41.459059Z DEBUG hyper::client::connect::dns: resolving host="s3.eu-west-1.amazonaws.com"
2024-05-13T07:20:41.489989Z DEBUG hyper::client::connect::http: connecting to 52.218.41.251:443
2024-05-13T07:20:41.543851Z DEBUG hyper::client::connect::http: connected to 52.218.41.251:443
2024-05-13T07:20:41.543952Z DEBUG rustls::client::hs: Resuming session    
2024-05-13T07:20:41.597272Z DEBUG rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2024-05-13T07:20:41.597290Z DEBUG rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2024-05-13T07:20:41.597358Z DEBUG rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2024-05-13T07:20:41.597363Z DEBUG rustls::client::tls12: Server DNS name is DnsName("s3.eu-west-1.amazonaws.com")    
2024-05-13T07:20:41.652487Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:41.880295Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:41.880310Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17496199 bytes)
2024-05-13T07:20:45.473934Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:45.473955Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:20:45.870389Z DEBUG reqwest::connect: starting new connection: https://s3.eu-west-1.amazonaws.com/    
2024-05-13T07:20:45.870457Z DEBUG hyper::client::connect::dns: resolving host="s3.eu-west-1.amazonaws.com"
2024-05-13T07:20:45.873023Z DEBUG hyper::client::connect::http: connecting to 52.218.109.91:443
2024-05-13T07:20:45.928794Z DEBUG hyper::client::connect::http: connected to 52.218.109.91:443
2024-05-13T07:20:45.928846Z DEBUG rustls::client::hs: Resuming session    
2024-05-13T07:20:45.986910Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256    
2024-05-13T07:20:45.986930Z DEBUG rustls::client::tls13: Not resuming    
2024-05-13T07:20:45.986975Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]    
2024-05-13T07:20:45.986979Z DEBUG rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2024-05-13T07:20:45.991108Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:46.202955Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:46.202969Z DEBUG hyper::proto::h1::conn: incoming body is content-length (44585788 bytes)
2024-05-13T07:20:50.676718Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:20:50.676814Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:51.125979Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:20:51.126144Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:20:51.270109Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:20:51.270125Z DEBUG hyper::proto::h1::conn: incoming body is content-length (24015617 bytes)
2024-05-13T07:21:12.821493Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:21:12.821528Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:21:44.544977Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:21:45.334087Z DEBUG reqwest::connect: starting new connection: https://s3.eu-west-1.amazonaws.com/    
2024-05-13T07:21:45.335238Z DEBUG hyper::client::connect::dns: resolving host="s3.eu-west-1.amazonaws.com"
2024-05-13T07:21:45.367883Z DEBUG hyper::client::connect::http: connecting to 52.218.92.139:443
2024-05-13T07:21:45.421227Z DEBUG hyper::client::connect::http: connected to 52.218.92.139:443
2024-05-13T07:21:45.421427Z DEBUG rustls::client::hs: No cached session for DnsName("s3.eu-west-1.amazonaws.com")    
2024-05-13T07:21:45.421460Z DEBUG rustls::client::hs: Not resuming any session    
2024-05-13T07:21:45.474525Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256    
2024-05-13T07:21:45.474541Z DEBUG rustls::client::tls13: Not resuming    
2024-05-13T07:21:45.474850Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]    
2024-05-13T07:21:45.474857Z DEBUG rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2024-05-13T07:21:45.480265Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:21:45.649425Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:21:45.649436Z DEBUG hyper::proto::h1::conn: incoming body is content-length (33898312 bytes)
2024-05-13T07:21:50.901122Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:21:50.901186Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:21:51.310278Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:21:51.310457Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:21:51.484234Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:21:51.484251Z DEBUG hyper::proto::h1::conn: incoming body is content-length (95281559 bytes)
2024-05-13T07:21:59.591727Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:21:59.591796Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:22:00.219723Z DEBUG hyper::client::pool: reuse idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:22:00.219893Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:22:00.362546Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:22:00.362562Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17696976 bytes)
2024-05-13T07:22:43.950975Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-05-13T07:22:43.952763Z DEBUG hyper::client::pool: pooling idle connection for ("https", s3.eu-west-1.amazonaws.com)
2024-05-13T07:27:21.328987Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:27:22.315684Z DEBUG reqwest::connect: starting new connection: https://s3.eu-west-1.amazonaws.com/    
2024-05-13T07:27:22.316264Z DEBUG hyper::client::connect::dns: resolving host="s3.eu-west-1.amazonaws.com"
2024-05-13T07:27:22.351032Z DEBUG hyper::client::connect::http: connecting to 52.92.35.192:443
2024-05-13T07:27:22.408325Z DEBUG hyper::client::connect::http: connected to 52.92.35.192:443
2024-05-13T07:27:22.408352Z DEBUG rustls::client::hs: No cached session for DnsName("s3.eu-west-1.amazonaws.com")    
2024-05-13T07:27:22.408391Z DEBUG rustls::client::hs: Not resuming any session    
2024-05-13T07:27:22.464832Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256    
2024-05-13T07:27:22.464848Z DEBUG rustls::client::tls13: Not resuming    
2024-05-13T07:27:22.464898Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]    
2024-05-13T07:27:22.464905Z DEBUG rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2024-05-13T07:27:22.467413Z DEBUG hyper::proto::h1::io: flushed 664 bytes
2024-05-13T07:27:22.639022Z DEBUG hyper::proto::h1::io: parsed 15 headers
2024-05-13T07:27:22.639044Z DEBUG hyper::proto::h1::conn: incoming body is content-length (19513329 bytes)
2024-05-13T07:27:23.407605Z DEBUG hyper::proto::h1::conn: incoming body decode error: Connection reset by peer (os error 54)
2024-05-13T07:27:23.407947Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-05-13T07:27:23.407976Z DEBUG hyper::proto::h1::conn: error shutting down IO: Broken pipe (os error 32)
2024-05-13T07:27:23.407994Z DEBUG hyper::client::client: client connection error: error shutting down connection: Broken pipe (os error 32)
2024-05-13T07:27:23.416153Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
Error: Custom { kind: Other, error: External(External(ArrowError(ExternalError(IoError(Custom { kind: Other, error: ObjectStore(Generic { store: "S3", source: reqwest::Error { kind: Body, source: hyper::Error(Body, Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" }) } }) })), None))) }
➜  df_play git:(main) ✗
@Smotrov Smotrov added the bug Something isn't working label May 13, 2024
@Maxsparrow
Copy link

We've been hitting this issue too, the object store doesn't retry these errors even if you set retries:

Error: External(ParquetError(General("AsyncChunkReader::get_bytes error: Generic S3 error: Error after 0 retries in 621.914125ms, max_retries:120, retry_timeout:600s, source:error sending request for url(...): connection error: Connection reset by peer (os error 54)")))

I believe it may be fixed by apache/arrow-rs#5609 which is in object store release 0.10.0

@alamb - will the next DataFusion release include an object store bump?

@Smotrov
Copy link
Author

Smotrov commented May 19, 2024 •
edited
Loading

We've been hitting this issue too, the object store doesn't retry these errors even if you set retries:

In my case retry doesn't help. It is listed in the code above.

@Smotrov
Copy link
Author

Smotrov commented May 19, 2024

I believe it may be fixed by apache/arrow-rs#5609 which is in object store release 0.10.0

#10116

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Maxsparrow @Smotrov