Skip to content

Commit

Permalink
fix for #363
Browse files Browse the repository at this point in the history
We still need to set the load in the process because a demand for higher
crawl speed may require to increase the maximum load limit. However,
following the criticism in the bug, we do never reduce the load limit
again.
  • Loading branch information
Orbiter committed Oct 16, 2023
1 parent 5bc09af commit 03bf259
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions source/net/yacy/htroot/Crawler_p.java
Original file line number Diff line number Diff line change
Expand Up @@ -799,10 +799,14 @@ public static serverObjects respond(final RequestHeader header, final serverObje
wPPM = 30000;
}
final int newBusySleep = 60000 / wPPM; // for wantedPPM = 10: 6000; for wantedPPM = 1000: 60
final float loadprereq = wantedPPM <= 10 ? 1.0f : wantedPPM <= 100 ? 2.0f : wantedPPM >= 1000 ? 8.0f : 3.0f;

BusyThread thread;
// we must increase the load limit because a conservative load limit will prevent a high crawling speed
// however this must not cause that the load limit is reduced again because that may go against the users requirements
// in case they set the limit themself, see https://github.com/yacy/yacy_search_server/issues/363
float loadprereq = wantedPPM <= 10 ? 1.0f : wantedPPM <= 100 ? 2.0f : wantedPPM >= 1000 ? 8.0f : 3.0f;
loadprereq = Math.max(loadprereq, sb.getConfigFloat(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_LOADPREREQ, loadprereq));

BusyThread thread;
thread = sb.getThread(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL);
if ( thread != null ) {
sb.setConfig(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_BUSYSLEEP, thread.setBusySleep(newBusySleep));
Expand Down

0 comments on commit 03bf259

Please sign in to comment.