Go Back
Related Support Articles
Trying to crawl a website fails with the error "An existing connection was forcibly closed by the remote host" How can I copy only changed files from a website? Why is WebCopy not scanning my site properly? When crawling a page, it fails with a "origin error" message (code 520) When attempting to open a project, you receive a "An item with the same key has already been added." error When viewing an offline website copied via WebCopy, clicking links displays raw HTML formatting tags When copying a website, WebCopy crashes with message "Access to the path 'Global\.net clr networking' is denied." When crawling a page, it fails with a "forbidden" error (code 403) When crawling a page, it fails with a "method not allowed" error (code 405)
Product Documentation
Forums
Roadmap
Known Issues
Suggest a Feature

When crawling a page, it fails with a "origin error" message (code 520)

Occasionally you might try to copy a website, only to get a 520 error code from each page.

What is status code 520?

HTTP status code 520 is not specified in any RFCs, but is used by Cloudflare's reverse proxies to signal an "unknown connection issue between CloudFlare and the origin web server" to a client in front of the proxy.

How can I work around it?

The most likely reason you are getting this message is due to the default user agent for your crawl project. By default, the user agent is the name and version of the Cyotek product doing the crawling, and the name and version of the crawling library itself. For WebCopy projects, this is generally CyotekWebCopy/1.0 CyotekWebCrawler/1.0. Notice that this user agent does not identify itself as a web browser.

Changing your user agent to that of an actual browser may help.

How do I change the user agent of my project?

  • Open the Project menu and choose User Agent
  • Select Use custom user agent
  • Either select one of the pre-defined agents, or enter your own custom string.
  • Click OK to save the changes

How can I check before crawling an entire site?

You can use the Test URI feature of WebCopy to determine if the URI you want to crawl is going to reject the user agent. Simply click Test URI from the toolbar, enter the URL of the site to test, select a predefined or custom user agent and click Test. WebCopy will try access the URL, and will notify you of any problems.