Some need to be crawled more frequently, some don't germany phone number list free need to be crawled at all. Therefore, we use a queue that decides in which order the URLs will be sent for crawling.
A common problem with this step is crawling too many similar and irrelevant URLs, which can lead people to see more spam and fewer unique referring domains.
What did we do?
To optimize the queue, we added filters that prioritize unique content and higher authority websites, as well as combat link farms. As a result, the system now finds more unique content and generates fewer duplicate link reports.
Some highlights of how our system works now:
To protect our queue from link farms we check if a large number of domains are coming from the same IP address. If we see too many domains from the same IP, their priority in the queue will be reduced, allowing us to scan more domains from different IPs and not get stuck on a link farm.
To protect websites and avoid polluting our reports with similar links, we check if there are too many URLs from the same domain. If we see too many URLs from the same domain, they will not all be scanned on the same day.
To ensure we crawl new pages as soon as possible, any URLs we haven't crawled previously will be given higher priority.
Each page has its own hash code that helps us prioritize crawling unique content.
We take into account the frequency with which new links are generated on the source page.
We take into account the authority score of a web page and a domain.
How the tail is improved
More than 10 different factors filter out unnecessary links.
More unique and high-quality pages, thanks to new quality control algorithms.
