Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Repository redirects cause tiny-maven-proxy to hang #10

Open
mbafford opened this issue Feb 20, 2020 · 4 comments
Open

Repository redirects cause tiny-maven-proxy to hang #10

mbafford opened this issue Feb 20, 2020 · 4 comments

Comments

@mbafford
Copy link

It looks like tiny-maven-proxy has a bug which causes the request to hang for 30+ minutes (indefinitely?) when the repository returns a 301/302 redirect.

Using the latest JAR build linked from this repository:

java -jar ./tiny-maven-proxy.jar --maven.proxy.debug true --log.console true --mirror https://oss.sonatype.org/content/repositories/releases/
START WITH URLS
{"msg":"config","debug":true,"level":30,"download.threads":"24","pid":94942,"dir":"/var/folders/vb/v36bb1js3dbfhkj9l7_2n3440000gn/T/maven","eventThreads":"3","hostname":"agentsmith.local","v":0,"name":"startup","time":"2020-02-20T14:05:15.345Z","mirroring":["https://oss.sonatype.org/content/repositories/releases/"],"workers":"6"}
TinyMavenProxy 1.5 on port  5956 serving:
                            Repo:          https://oss.sonatype.org/content/repositories/releases/
Settings:
download.threads            24
workers                     6
eventThreads                3

Called with curl:

$ curl -f -v -O http://10.10.11.10:5956/BAD.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.10.11.10...
* TCP_NODELAY set
* Connected to 10.10.11.10 (10.10.11.10) port 5956 (#0)
> GET /BAD.jar HTTP/1.1
> Host: 10.10.11.10:5956
> User-Agent: curl/7.54.0
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:34:12 --:--:--     0^C

Logs for those 34 minutes:

  defer and download  BAD.jar
attempt https://oss.sonatype.org/content/repositories/releases/BAD.jar
state Connecting https://oss.sonatype.org/content/repositories/releases/BAD.jar
state Connected https://oss.sonatype.org/content/repositories/releases/BAD.jar
state SendRequest https://oss.sonatype.org/content/repositories/releases/BAD.jar
state AwaitingResponse https://oss.sonatype.org/content/repositories/releases/BAD.jar
state Redirect https://oss.sonatype.org/content/repositories/releases/BAD.jar
state Connecting https://oss.sonatype.org/content/repositories/releases/BAD.jar

And then after killing the curl process:

Complete w/ remaining 1
state Cancelled https://oss.sonatype.org/content/repositories/releases/BAD.jar

The URL being fetched ( https://oss.sonatype.org/content/repositories/releases/BAD.jar ) returns a 302 redirect very quickly when called directly with curl:

$ curl -v 'https://oss.sonatype.org/content/repositories/releases/BAD.jar'
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 302
< server: awselb/2.0
< date: Thu, 20 Feb 2020 14:44:34 GMT
< content-type: text/html
< content-length: 126
< location: https://repo1.maven.org:443/content/repositories/releases/BAD.jar
<
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
</body>
</html>

This seems to be related to the repository returning a redirect. I can reproduce with https://repo1.maven.org/content/repositories/releases/, which 301 redirects to https://repo1.maven.org/maven2/BAD.jar in the above test.

However, a repository which doesn't do redirects fails quickly:

$ java -jar ./tiny-maven-proxy.jar --maven.proxy.debug true --log.console true --mirror https://repo1.maven.org/maven2/
START WITH URLS
{"msg":"config","debug":true,"level":30,"download.threads":"24","pid":95653,"dir":"/var/folders/vb/v36bb1js3dbfhkj9l7_2n3440000gn/T/maven","eventThreads":"3","hostname":"agentsmith.local","v":0,"name":"startup","time":"2020-02-20T14:47:29.647Z","mirroring":["https://repo1.maven.org/maven2/"],"workers":"6"}
TinyMavenProxy 1.5 on port  5956 serving:
                            Repo:          https://repo1.maven.org/maven2/
Settings:
download.threads            24
workers                     6
eventThreads                3

  defer and download  BAD.jar
attempt https://repo1.maven.org/maven2/BAD.jar
state Connecting https://repo1.maven.org/maven2/BAD.jar
state Connected https://repo1.maven.org/maven2/BAD.jar
state SendRequest https://repo1.maven.org/maven2/BAD.jar
state AwaitingResponse https://repo1.maven.org/maven2/BAD.jar
   download failed:  https://repo1.maven.org/maven2/BAD.jar, 404 Not Found
state Cancelled https://repo1.maven.org/maven2/BAD.jar
{"msg":"oneDownloadFailed","hostname":"agentsmith.local","u":"https://repo1.maven.org/maven2/BAD.jar","level":30,"v":0,"name":"download","pid":95653,"time":"2020-02-20T14:47:46.868Z","dlid":"k6uv0skq:0"}
 fail  404 Not Found
{"msg":"allDownloadsFailed","path":"BAD.jar","hostname":"agentsmith.local","u":"https://repo1.maven.org/maven2/BAD.jar","level":30,"v":0,"name":"download","pid":95653,"id":"eafyv:0","time":"2020-02-20T14:47:46.869Z","status":{"reason":"Not Found","code":404}}
state HeadersReceived https://repo1.maven.org/maven2/BAD.jar
Status https://repo1.maven.org/maven2/BAD.jar 404 Not Found
   download failed:  https://repo1.maven.org/maven2/BAD.jar, 404 Not Found
{"dur":611,"msg":"request","agent":"curl/7.54.0","address":"10.10.11.10","method":"GET","level":20,"pid":95653,"path":"BAD.jar","hostname":"agentsmith.local","v":0,"host":"10.10.11.10:5956","name":"requests","id":"eafyv:0","time":"2020-02-20T14:47:46.880Z","status":404}
Complete w/ remaining 0
@mbafford mbafford mentioned this issue Feb 20, 2020
Open
@alexklibisz
Copy link

@mbafford I'm also seeing this issue. Did you figure anything out? In general it would be nice to set a timeout. Maybe that can be done by putting a proxy like nginx in front of this and set the timeout there.

@mbafford
Copy link
Author

mbafford commented May 9, 2020

@alexklibisz Off-topic for this issue, so I'll keep it brief here (more comments at the link below). I found it easier to scratch my itch by writing a quick Python script (https://gist.github.com/mbafford/d25939a35f5066d46753db6bfba7684b) to do the mirroring from my local ~/.m2/repository

@alexklibisz
Copy link

@alexklibisz Off-topic for this issue, so I'll keep it brief here (more comments at the link below). I found it easier to scratch my itch by writing a quick Python script (https://gist.github.com/mbafford/d25939a35f5066d46753db6bfba7684b) to do the mirroring from my local ~/.m2/repository

Neat, thanks. We ended up going in a different direction. We set up a virtual repository in JFrog to merge all of our internal and external repos, deployed an nginx reverse proxy with caching in our cluster so that it proxied requests out to Jfrog, and pointed our builds at the nginx server. I did experiment with a pure nginx reverse proxy solution but it turns out to be very difficult to handle all of the response codes and possible errors with the different artifact repos.

@saramah
Copy link

saramah commented Jan 28, 2022

I am also seeing this issue with a 303.

curl -v https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
*snip*
> GET /m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom HTTP/1.1
> Host: plugins.gradle.org
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 303 See Other
< Date: Fri, 28 Jan 2022 23:39:44 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://plugins-artifacts.gradle.org/org.jlleitschuh.gradle/ktlint-gradle/9.4.1/86fac77c64e4229194669cde401e08c5fa4defd88a4fd3e05deff35b2d977a22/ktlint-gradle-9.4.1.pom
*snip*

tiny-maven-proxy output:

  defer and download  org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
attempt https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connected https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state SendRequest https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state AwaitingResponse https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Redirect https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
Complete w/ remaining 1
state Cancelled https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
  defer and download  org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
attempt https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connected https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state SendRequest https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state AwaitingResponse https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Redirect https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
Complete w/ remaining 1
state Cancelled https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
  defer and download  org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
attempt https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connected https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state SendRequest https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state AwaitingResponse https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Redirect https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
state Connecting https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom
Complete w/ remaining 1
state Cancelled https://plugins.gradle.org/m2/org/jlleitschuh/gradle/ktlint-gradle/9.4.1/ktlint-gradle-9.4.1.pom

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@saramah @mbafford @alexklibisz