Spider (webmagic-core 0.5.2 API)

java.lang.Object
- us.codecraft.webmagic.Spider

所有已实现的接口:

Runnable, Task
```
public class Spider
extends Object
implements Runnable, Task
```
Entrance of a crawler.
A spider contains four modules: Downloader, Scheduler, PageProcessor and Pipeline.
Every module is a field of Spider.
The modules are defined in interface.
You can customize a spider with various implementations of them.
Examples:

A simple crawler:
Spider.create(new SimplePageProcessor("http://my.oschina.net/", "http://my.oschina.net/*blog/*")).run();

Store results to files by FilePipeline:
Spider.create(new SimplePageProcessor("http://my.oschina.net/", "http://my.oschina.net/*blog/*"))
.pipeline(new FilePipeline("/data/temp/webmagic/")).run();

Use FileCacheQueueScheduler to store urls and cursor in files, so that a Spider can resume the status when shutdown.
Spider.create(new SimplePageProcessor("http://my.oschina.net/", "http://my.oschina.net/*blog/*"))
.scheduler(new FileCacheQueueScheduler("/data/temp/webmagic/cache/")).run();

从以下版本开始:

0.1.0

作者:

code4crafter@gmail.com

另请参阅:
Downloader, Scheduler, PageProcessor, Pipeline

嵌套类概要

嵌套类
限定符和类型类和说明

static class Spider.Status

嵌套类
限定符和类型	类和说明
`static class`	`Spider.Status`

字段概要

字段
限定符和类型	字段和说明
`protected boolean`	`destroyWhenExit`
`protected Downloader`	`downloader`
`protected ExecutorService`	`executorService`
`protected boolean`	`exitWhenComplete`
`protected org.slf4j.Logger`	`logger`
`protected PageProcessor`	`pageProcessor`
`protected List<Pipeline>`	`pipelines`
`protected Scheduler`	`scheduler`
`protected Site`	`site`
`protected boolean`	`spawnUrl`
`protected List<Request>`	`startRequests`
`protected AtomicInteger`	`stat`
`protected static int`	`STAT_INIT`
`protected static int`	`STAT_RUNNING`
`protected static int`	`STAT_STOPPED`
`protected int`	`threadNum`
`protected CountableThreadPool`	`threadPool`
`protected String`	`uuid`

构造器概要

构造器
构造器和说明

Spider(PageProcessor pageProcessor)
create a spider with pageProcessor.

构造器
构造器和说明
`Spider(PageProcessor pageProcessor)` create a spider with pageProcessor.

方法概要

方法
限定符和类型	方法和说明
`Spider`	`addPipeline(Pipeline pipeline)` add a pipeline for Spider
`Spider`	`addRequest(Request... requests)` Add urls with information to crawl.
`Spider`	`addUrl(String... urls)` Add urls to crawl.
`protected void`	`checkIfRunning()`
`Spider`	`clearPipeline()` clear the pipelines set
`void`	`close()`
`static Spider`	`create(PageProcessor pageProcessor)` create a spider with pageProcessor.
`Spider`	`downloader(Downloader downloader)` 已过时。
`protected void`	`extractAndAddRequests(Page page, boolean spawnUrl)`
`<T> T`	`get(String url)`
`<T> List<T>`	`getAll(Collection<String> urls)` Download urls synchronizing.
`protected CollectorPipeline`	`getCollectorPipeline()`
`long`	`getPageCount()` Get page count downloaded by spider.
`Scheduler`	`getScheduler()`
`Site`	`getSite()` site of a task
`List<SpiderListener>`	`getSpiderListeners()`
`Date`	`getStartTime()`
`Spider.Status`	`getStatus()` Get running status by spider.
`int`	`getThreadAlive()` Get thread count which is running
`String`	`getUUID()` unique id for a task.
`protected void`	`initComponent()`
`boolean`	`isExitWhenComplete()`
`boolean`	`isSpawnUrl()`
`protected void`	`onError(Request request)`
`protected void`	`onSuccess(Request request)`
`Spider`	`pipeline(Pipeline pipeline)` 已过时。
`protected void`	`processRequest(Request request)`
`void`	`run()`
`void`	`runAsync()`
`Spider`	`scheduler(Scheduler scheduler)` set scheduler for Spider
`Spider`	`setDownloader(Downloader downloader)` set the downloader of spider
`void`	`setEmptySleepTime(int emptySleepTime)` Set wait time when no url is polled.
`Spider`	`setExecutorService(ExecutorService executorService)`
`Spider`	`setExitWhenComplete(boolean exitWhenComplete)` Exit when complete.
`Spider`	`setPipelines(List<Pipeline> pipelines)` set pipelines for Spider
`Spider`	`setScheduler(Scheduler scheduler)` set scheduler for Spider
`Spider`	`setSpawnUrl(boolean spawnUrl)` Whether add urls extracted to download.
`Spider`	`setSpiderListeners(List<SpiderListener> spiderListeners)`
`Spider`	`setUUID(String uuid)` Set an uuid for spider.
`protected void`	`sleep(int time)`
`void`	`start()`
`Spider`	`startRequest(List<Request> startRequests)` Set startUrls of Spider.
`Spider`	`startUrls(List<String> startUrls)` Set startUrls of Spider.
`void`	`stop()`
`void`	`test(String... urls)` Process specific urls without url discovering.
`Spider`	`thread(ExecutorService executorService, int threadNum)` start with more than one threads
`Spider`	`thread(int threadNum)` start with more than one threads

从类继承的方法 java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- 字段详细资料
  - downloader
```
protected Downloader downloader
```
  - pipelines
```
protected List<Pipeline> pipelines
```
  - pageProcessor
```
protected PageProcessor pageProcessor
```
  - startRequests
```
protected List<Request> startRequests
```
  - site
```
protected Site site
```
  - uuid
```
protected String uuid
```
  - scheduler
```
protected Scheduler scheduler
```
  - logger
```
protected org.slf4j.Logger logger
```
  - threadPool
```
protected CountableThreadPool threadPool
```
  - executorService
```
protected ExecutorService executorService
```
  - threadNum
```
protected int threadNum
```
  - stat
```
protected AtomicInteger stat
```
  - exitWhenComplete
```
protected boolean exitWhenComplete
```
  - STAT_INIT
```
protected static final int STAT_INIT
```
    另请参阅:
    常量字段值
  - STAT_RUNNING
```
protected static final int STAT_RUNNING
```
    另请参阅:
    常量字段值
  - STAT_STOPPED
```
protected static final int STAT_STOPPED
```
    另请参阅:
    常量字段值
  - spawnUrl
```
protected boolean spawnUrl
```
  - destroyWhenExit
```
protected boolean destroyWhenExit
```
- 构造器详细资料
  - Spider
```
public Spider(PageProcessor pageProcessor)
```
    create a spider with pageProcessor.
    
    参数:
    pageProcessor -
- 方法详细资料
  - create
```
public static Spider create(PageProcessor pageProcessor)
```
    create a spider with pageProcessor.
    
    参数:
    pageProcessor -
    
    返回:
    new spider
    另请参阅:
    PageProcessor
  - startUrls
```
public Spider startUrls(List<String> startUrls)
```
    Set startUrls of Spider.
    Prior to startUrls of Site.
    
    参数:
    startUrls -
    
    返回:
    this
  - startRequest
```
public Spider startRequest(List<Request> startRequests)
```
    Set startUrls of Spider.
    Prior to startUrls of Site.
    
    参数:
    startRequests -
    
    返回:
    this
  - setUUID
```
public Spider setUUID(String uuid)
```
    Set an uuid for spider.
    Default uuid is domain of site.
    
    参数:
    uuid -
    
    返回:
    this
  - scheduler
```
public Spider scheduler(Scheduler scheduler)
```
    set scheduler for Spider
    
    参数:
    scheduler -
    
    返回:
    this
    另请参阅:
    setScheduler(us.codecraft.webmagic.scheduler.Scheduler)
  - setScheduler
```
public Spider setScheduler(Scheduler scheduler)
```
    set scheduler for Spider
    
    参数:
    scheduler -
    
    返回:
    this
    从以下版本开始:
    
    0.2.1
    
    另请参阅:
    Scheduler
  - pipeline
```
public Spider pipeline(Pipeline pipeline)
```
    已过时。
    
    add a pipeline for Spider
    
    参数:
    pipeline -
    
    返回:
    this
    另请参阅:
    addPipeline(us.codecraft.webmagic.pipeline.Pipeline)
  - addPipeline
```
public Spider addPipeline(Pipeline pipeline)
```
    add a pipeline for Spider
    
    参数:
    pipeline -
    
    返回:
    this
    从以下版本开始:
    
    0.2.1
    
    另请参阅:
    Pipeline
  - setPipelines
```
public Spider setPipelines(List<Pipeline> pipelines)
```
    set pipelines for Spider
    
    参数:
    pipelines -
    
    返回:
    this
    从以下版本开始:
    
    0.4.1
    
    另请参阅:
    Pipeline
  - clearPipeline
```
public Spider clearPipeline()
```
    clear the pipelines set
    
    返回:
    this
  - downloader
```
public Spider downloader(Downloader downloader)
```
    已过时。
    
    set the downloader of spider
    
    参数:
    downloader -
    
    返回:
    this
    另请参阅:
    setDownloader(us.codecraft.webmagic.downloader.Downloader)
  - setDownloader
```
public Spider setDownloader(Downloader downloader)
```
    set the downloader of spider
    
    参数:
    downloader -
    
    返回:
    this
    另请参阅:
    Downloader
  - initComponent
```
protected void initComponent()
```
  - run
```
public void run()
```
    指定者:
    
    run 在接口中 Runnable
  - onError
```
protected void onError(Request request)
```
  - onSuccess
```
protected void onSuccess(Request request)
```
  - close
```
public void close()
```
  - test
```
public void test(String... urls)
```
    Process specific urls without url discovering.
    
    参数:
    urls - urls to process
  - processRequest
```
protected void processRequest(Request request)
```
  - sleep
```
protected void sleep(int time)
```
  - extractAndAddRequests
```
protected void extractAndAddRequests(Page page,
                         boolean spawnUrl)
```
  - checkIfRunning
```
protected void checkIfRunning()
```
  - runAsync
```
public void runAsync()
```
  - addUrl
```
public Spider addUrl(String... urls)
```
    Add urls to crawl.
    
    参数:
    urls -
    
    返回:
  - getAll
```
public <T> List<T> getAll(Collection<String> urls)
```
    Download urls synchronizing.
    
    参数:
    urls -
    
    返回:
  - getCollectorPipeline
```
protected CollectorPipeline getCollectorPipeline()
```
  - get
```
public <T> T get(String url)
```
  - addRequest
```
public Spider addRequest(Request... requests)
```
    Add urls with information to crawl.
    
    参数:
    requests -
    
    返回:
  - start
```
public void start()
```
  - stop
```
public void stop()
```
  - thread
```
public Spider thread(int threadNum)
```
    start with more than one threads
    
    参数:
    threadNum -
    
    返回:
    this
  - thread
```
public Spider thread(ExecutorService executorService,
            int threadNum)
```
    start with more than one threads
    
    参数:
    threadNum -
    
    返回:
    this
  - isExitWhenComplete
```
public boolean isExitWhenComplete()
```
  - setExitWhenComplete
```
public Spider setExitWhenComplete(boolean exitWhenComplete)
```
    Exit when complete.
    True: exit when all url of the site is downloaded.
    False: not exit until call stop() manually.
    
    参数:
    exitWhenComplete -
    
    返回:
  - isSpawnUrl
```
public boolean isSpawnUrl()
```
  - getPageCount
```
public long getPageCount()
```
    Get page count downloaded by spider.
    
    返回:
    total downloaded page count
    从以下版本开始:
    
    0.4.1
  - getStatus
```
public Spider.Status getStatus()
```
    Get running status by spider.
    
    返回:
    running status
    从以下版本开始:
    
    0.4.1
    
    另请参阅:
    Spider.Status
  - getThreadAlive
```
public int getThreadAlive()
```
    Get thread count which is running
    
    返回:
    thread count which is running
    从以下版本开始:
    
    0.4.1
  - setSpawnUrl
```
public Spider setSpawnUrl(boolean spawnUrl)
```
    Whether add urls extracted to download.
    Add urls to download when it is true, and just download seed urls when it is false.
    DO NOT set it unless you know what it means!
    
    参数:
    spawnUrl -
    
    返回:
    从以下版本开始:
    
    0.4.0
  - getUUID
```
public String getUUID()
```
    从接口复制的说明: Task
    
    unique id for a task.
    
    指定者:
    
    getUUID 在接口中 Task
    
    返回:
    uuid
  - setExecutorService
```
public Spider setExecutorService(ExecutorService executorService)
```
  - getSite
```
public Site getSite()
```
    从接口复制的说明: Task
    
    site of a task
    
    指定者:
    
    getSite 在接口中 Task
    
    返回:
    site
  - getSpiderListeners
```
public List<SpiderListener> getSpiderListeners()
```
  - setSpiderListeners
```
public Spider setSpiderListeners(List<SpiderListener> spiderListeners)
```
  - getStartTime
```
public Date getStartTime()
```
  - getScheduler
```
public Scheduler getScheduler()
```
  - setEmptySleepTime
```
public void setEmptySleepTime(int emptySleepTime)
```
    Set wait time when no url is polled.
    
    参数:
    emptySleepTime - In MILLISECONDS.

类 Spider

嵌套类概要

字段概要

构造器概要

方法概要

从类继承的方法 java.lang.Object

字段详细资料

downloader

pipelines

pageProcessor

startRequests

site

uuid

scheduler

logger

threadPool

executorService

threadNum

stat

exitWhenComplete

STAT_INIT

STAT_RUNNING

STAT_STOPPED

spawnUrl

destroyWhenExit

构造器详细资料

Spider

方法详细资料

create

startUrls

startRequest

setUUID

scheduler

setScheduler

pipeline

addPipeline

setPipelines

clearPipeline

downloader

setDownloader

initComponent

run

onError

onSuccess

close

test

processRequest

sleep

extractAndAddRequests

checkIfRunning

runAsync

addUrl

getAll

getCollectorPipeline

get

addRequest

start

stop

thread

thread

isExitWhenComplete

setExitWhenComplete

isSpawnUrl

getPageCount

getStatus

getThreadAlive

setSpawnUrl

getUUID

setExecutorService

getSite

getSpiderListeners

setSpiderListeners

getStartTime

getScheduler

setEmptySleepTime