Web Spider source code
2016-08-23
0 0 0
no vote
Other
Earn points
Main characteristics are:
Configurable: the number of threads, the thread waits for the time, connection timeout, to crawl a file type and priority, download directory, and so on.
The status bar displays the statistics: queued URL, download number, total number of bytes downloaded, such as CPU usage and available memory.
Preference of Creeper: to crawl a resource type to set different priorities.
Robustness: a dozen URL normalization strategy to eliminate redundant downloads, reptile traps to avoid the use of such strategies, a variety of strategies to resolve relative paths, an
Configurable: the number of threads, the thread waits for the time, connection timeout, to crawl a file type and priority, download directory, and so on.
The status bar displays the statistics: queued URL, download number, total number of bytes downloaded, such as CPU usage and available memory.
Preference of Creeper: to crawl a resource type to set different priorities.
Robustness: a dozen URL normalization strategy to eliminate redundant downloads, reptile traps to avoid the use of such strategies, a variety of strategies to resolve relative paths, an
c#
爬虫
网络
源代码
Related Source Codes
No. 186: DX0110- Source code for community propert
0
0
no vote
No. 219: DX0149- Source code for community propert
0
0
no vote
Verification code identification
0
0
no vote
CSV data analysis tool
0
0
no vote
Source code of hospital medical record information
0
0
no vote
No comment