| Package | Description |
|---|---|
| us.codecraft.webmagic |
Main class "Spider" and models.
|
| us.codecraft.webmagic.scheduler |
Scheduler is the part of url management.
|
| Modifier and Type | Field and Description |
|---|---|
protected Scheduler |
Spider.scheduler |
| Modifier and Type | Method and Description |
|---|---|
Scheduler |
Spider.getScheduler() |
| Modifier and Type | Method and Description |
|---|---|
Spider |
Spider.scheduler(Scheduler scheduler)
Deprecated.
|
Spider |
Spider.setScheduler(Scheduler scheduler)
set scheduler for Spider
|
| Modifier and Type | Interface and Description |
|---|---|
interface |
MonitorableScheduler
The scheduler whose requests can be counted for monitor.
|
| Modifier and Type | Class and Description |
|---|---|
class |
DuplicateRemovedScheduler
Remove duplicate urls and only push urls which are not duplicate.
|
class |
PriorityScheduler
Priority scheduler.
|
class |
QueueScheduler
Basic Scheduler implementation.
Store urls to fetch in LinkedBlockingQueue and remove duplicate urls by HashMap. |
Copyright © 2017. All rights reserved.