Skip to content

Commit

Permalink
refactor: change condition for pausing crawls during redirections
Browse files Browse the repository at this point in the history
- Change the condition for pausing crawls if it is not a redirection

Signed-off-by: Corentin Barreau <corentin@archive.org>
  • Loading branch information
CorentinB committed Aug 24, 2023
1 parent 0497d26 commit 0569f25
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion internal/pkg/crawl/capture.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func (c *Crawl) executeGET(item *frontier.Item, req *http.Request, isRedirection

// Temporarily pause crawls for individual hosts if they are over our configured maximum concurrent requests per domain.
// If the request is a redirection, we do not pause the crawl because we want to follow the redirection.
if isRedirection {
if !isRedirection {
for c.shouldPause(item.Host) {
time.Sleep(time.Millisecond * time.Duration(c.RateLimitDelay))
}
Expand Down

0 comments on commit 0569f25

Please sign in to comment.