us.codecraft.webmagic

类 Site

java.lang.Object

us.codecraft.webmagic.Site

public class Site
extends Object

Object contains setting for crawler.

从以下版本开始:

0.1.0

作者:

code4crafter@gmail.com

另请参阅:

PageProcessor

嵌套类概要

嵌套类

限定符和类型	类和说明
static interface	Site.HeaderConst 已过时。

构造器概要

构造器

构造器和说明
Site()

方法概要

方法

限定符和类型	方法和说明
Site	addCookie(String name, String value) Add a cookie with domain getDomain()
Site	addCookie(String domain, String name, String value) Add a cookie with specific domain.
Site	addHeader(String key, String value) Put an Http header for downloader.
Site	addStartRequest(Request startRequest) 已过时。
Site	addStartUrl(String startUrl) 已过时。
boolean	equals(Object o)
Set<Integer>	getAcceptStatCode() get acceptStatCode
Map<String,Map<String,String>>	getAllCookies() get cookies of all domains
String	getCharset() get charset set manually
Map<String,String>	getCookies() get cookies
int	getCycleRetryTimes() When cycleRetryTimes is more than 0, it will add back to scheduler and try download again.
String	getDomain() get domain
Map<String,String>	getHeaders()
org.apache.http.HttpHost	getHttpProxy()
org.apache.http.HttpHost	getHttpProxyFromPool()
ProxyPool	getHttpProxyPool()
int	getRetryTimes() Get retry times immediately when download fail, 0 by default.
int	getSleepTime() Get the interval between the processing of two pages.
List<Request>	getStartRequests()
List<String>	getStartUrls() 已过时。
int	getTimeOut()
String	getUserAgent() get user agent
int	hashCode()
boolean	isUseGzip()
static Site	me() new a Site
void	returnHttpProxyToPool(org.apache.http.HttpHost proxy, int statusCode)
Site	setAcceptStatCode(Set<Integer> acceptStatCode) Set acceptStatCode.
Site	setCharset(String charset) Set charset of page manually.
Site	setCycleRetryTimes(int cycleRetryTimes) Set cycleRetryTimes times when download fail, 0 by default.
Site	setDomain(String domain) set the domain of site.
Site	setHttpProxy(org.apache.http.HttpHost httpProxy) set up httpProxy for this site
Site	setHttpProxyPool(List<String[]> httpProxyList) Set httpProxyPool, String[0]:ip, String[1]:port
Site	setProxyReuseInterval(int reuseInterval)
Site	setRetryTimes(int retryTimes) Set retry times when download fail, 0 by default.
Site	setSleepTime(int sleepTime) Set the interval between the processing of two pages.
Site	setTimeOut(int timeOut) set timeout for downloader in ms
Site	setUseGzip(boolean useGzip) Whether use gzip.
Site	setUserAgent(String userAgent) set user agent
String	toString()
Task	toTask()

从类继承的方法 java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

构造器详细资料

Site

public Site()

方法详细资料

me

public static Site me()

new a Site

返回:

new site

addCookie

public Site addCookie(String name,
             String value)

Add a cookie with domain getDomain()

参数:

name -

value -

返回:

this

addCookie

public Site addCookie(String domain,
             String name,
             String value)

Add a cookie with specific domain.

参数:

domain -

name -

value -

返回:

setUserAgent

public Site setUserAgent(String userAgent)

set user agent

参数:

userAgent - userAgent

返回:

this

getCookies

public Map<String,String> getCookies()

get cookies

返回:

get cookies

getAllCookies

public Map<String,Map<String,String>> getAllCookies()

get cookies of all domains

返回:

get cookies

getUserAgent

public String getUserAgent()

get user agent

返回:

user agent

getDomain

public String getDomain()

get domain

返回:

get domain

setDomain

public Site setDomain(String domain)

set the domain of site.

参数:

domain -

返回:

this

setCharset

public Site setCharset(String charset)

Set charset of page manually.
When charset is not set or set to null, it can be auto detected by Http header.

参数:

charset -

返回:

this

getCharset

public String getCharset()

get charset set manually

返回:

charset

getTimeOut

public int getTimeOut()

setTimeOut

public Site setTimeOut(int timeOut)

set timeout for downloader in ms

参数:

timeOut -

setAcceptStatCode

public Site setAcceptStatCode(Set<Integer> acceptStatCode)

Set acceptStatCode.
When status code of http response is in acceptStatCodes, it will be processed.
{200} by default.
It is not necessarily to be set.

参数:

acceptStatCode -

返回:

this

getAcceptStatCode

public Set<Integer> getAcceptStatCode()

get acceptStatCode

返回:

acceptStatCode

getStartUrls

@Deprecated
public List<String> getStartUrls()

已过时。

get start urls

返回:

start urls

另请参阅:

getStartRequests()

getStartRequests

public List<Request> getStartRequests()

addStartUrl

public Site addStartUrl(String startUrl)

已过时。

Add a url to start url.
Because urls are more a Spider's property than Site, move it to Spider.addUrl(String...)}

参数:

startUrl -

返回:

this

另请参阅:

Spider.addUrl(String...)

addStartRequest

public Site addStartRequest(Request startRequest)

已过时。

Add a url to start url.
Because urls are more a Spider's property than Site, move it to Spider.addRequest(Request...)}

参数:

startRequest -

返回:

this

另请参阅:

Spider.addRequest(Request...)

setSleepTime

public Site setSleepTime(int sleepTime)

Set the interval between the processing of two pages.
Time unit is micro seconds.

参数:

sleepTime -

返回:

this

getSleepTime

public int getSleepTime()

Get the interval between the processing of two pages.
Time unit is micro seconds.

返回:

the interval between the processing of two pages,

getRetryTimes

public int getRetryTimes()

Get retry times immediately when download fail, 0 by default.

返回:

retry times when download fail

getHeaders

public Map<String,String> getHeaders()

addHeader

public Site addHeader(String key,
             String value)

Put an Http header for downloader.
Use addCookie(String, String) for cookie and setUserAgent(String) for user-agent.

参数:

key - key of http header, there are some keys constant in Site.HeaderConst

value - value of header

返回:

setRetryTimes

public Site setRetryTimes(int retryTimes)

Set retry times when download fail, 0 by default.

返回:

this

getCycleRetryTimes

public int getCycleRetryTimes()

When cycleRetryTimes is more than 0, it will add back to scheduler and try download again.

返回:

retry times when download fail

setCycleRetryTimes

public Site setCycleRetryTimes(int cycleRetryTimes)

Set cycleRetryTimes times when download fail, 0 by default. Only work in RedisScheduler.

返回:

this

getHttpProxy

public org.apache.http.HttpHost getHttpProxy()

setHttpProxy

public Site setHttpProxy(org.apache.http.HttpHost httpProxy)

set up httpProxy for this site

参数:

httpProxy -

返回:

isUseGzip

public boolean isUseGzip()

setUseGzip

public Site setUseGzip(boolean useGzip)

Whether use gzip.
Default is true, you can set it to false to disable gzip.

参数:

useGzip -

返回:

toTask

public Task toTask()

equals

public boolean equals(Object o)

覆盖:

equals 在类中 Object

hashCode

public int hashCode()

覆盖:

hashCode 在类中 Object

toString

public String toString()

覆盖:

toString 在类中 Object

setHttpProxyPool

public Site setHttpProxyPool(List<String[]> httpProxyList)

Set httpProxyPool, String[0]:ip, String[1]:port

返回:

this

getHttpProxyPool

public ProxyPool getHttpProxyPool()

getHttpProxyFromPool

public org.apache.http.HttpHost getHttpProxyFromPool()

returnHttpProxyToPool

public void returnHttpProxyToPool(org.apache.http.HttpHost proxy,
                         int statusCode)

setProxyReuseInterval

public Site setProxyReuseInterval(int reuseInterval)