us.codecraft.webmagic.downloader

类 HttpClientDownloader

java.lang.Object

us.codecraft.webmagic.downloader.AbstractDownloader

us.codecraft.webmagic.downloader.HttpClientDownloader

所有已实现的接口:

Downloader

@ThreadSafe
public class HttpClientDownloader
extends AbstractDownloader

The http downloader based on HttpClient.

从以下版本开始:

0.1.0

作者:

code4crafter@gmail.com

构造器概要

构造器

构造器和说明
HttpClientDownloader()

方法概要

方法

限定符和类型	方法和说明
Page	download(Request request, Task task) Downloads web pages and store in Page object.
protected String	getContent(String charset, org.apache.http.HttpResponse httpResponse)
protected String	getHtmlCharset(org.apache.http.HttpResponse httpResponse, byte[] contentBytes)
protected org.apache.http.client.methods.HttpUriRequest	getHttpUriRequest(Request request, Site site, Map<String,String> headers)
protected Page	handleResponse(Request request, String charset, org.apache.http.HttpResponse httpResponse, Task task)
protected org.apache.http.client.methods.RequestBuilder	selectRequestMethod(Request request)
void	setThread(int thread) Tell the downloader how many threads the spider used.
protected boolean	statusAccept(Set<Integer> acceptStatCode, int statusCode)

从类继承的方法 us.codecraft.webmagic.downloader.AbstractDownloader

addToCycleRetry, download, download, onError, onSuccess

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

HttpClientDownloader

public HttpClientDownloader()

方法详细资料

download

public Page download(Request request,
            Task task)

从接口复制的说明: Downloader

Downloads web pages and store in Page object.

返回:

page

setThread

public void setThread(int thread)

从接口复制的说明: Downloader

Tell the downloader how many threads the spider used.

参数:

thread - number of threads

statusAccept

protected boolean statusAccept(Set<Integer> acceptStatCode,
                   int statusCode)

getHttpUriRequest

protected org.apache.http.client.methods.HttpUriRequest getHttpUriRequest(Request request,
                                                              Site site,
                                                              Map<String,String> headers)

selectRequestMethod

protected org.apache.http.client.methods.RequestBuilder selectRequestMethod(Request request)

handleResponse

protected Page handleResponse(Request request,
                  String charset,
                  org.apache.http.HttpResponse httpResponse,
                  Task task)
                       throws IOException

抛出:

IOException

getContent

protected String getContent(String charset,
                org.apache.http.HttpResponse httpResponse)
                     throws IOException

抛出:

IOException

getHtmlCharset

protected String getHtmlCharset(org.apache.http.HttpResponse httpResponse,
                    byte[] contentBytes)
                         throws IOException

抛出:

IOException