Request HTTP URLs in a complex world — basic and digest authentication, redirections, cookies, timeout and more.
npm install urllib --save
import { request } from 'urllib';
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);
const { request } = require('urllib');
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);
You should create a urllib instance first.
import { HttpClient } from 'urllib';
const httpclient = new HttpClient();
httpclient.on('response', (info) => {
error: err,
ctx: args.ctx,
req: {
url: url,
options: options,
size: requestSize,
},
res: res
});
const { data, res } = await httpclient.request('https://nodejs.org');
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);
- url String | Object - The URL to request, either a String or a Object that return by url.parse.
- options Object - Optional
- method String - Request method, defaults to
GET
. Could beGET
,POST
,DELETE
orPUT
. Alias 'type'. - data Object - Data to be sent. Will be stringify automatically.
- content String | Buffer - Manually set the content of payload. If set,
data
will be ignored. - stream stream.Readable - Stream to be pipe to the remote. If set,
data
andcontent
will be ignored. - writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and
callback
will be called withdata
setnull
after finished writing. - files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with
multipart/form-data
format, base onformstream
. Ifmethod
not set, will usePOST
method by default. - contentType String - Type of request data. Could be
json
(Notes: not useapplication/json
here). If it'sjson
, will auto setContent-Type: application/json
header. - dataType String - Type of response data. Could be
text
orjson
. If it'stext
, thecallback
eddata
would be a String. If it'sjson
, thedata
of callback would be a parsed JSON Object and will auto setAccept: application/json
header. Defaultcallback
eddata
would be aBuffer
. - fixJSONCtlChars Boolean - Fix the control characters (U 0000 through U 001F) before JSON parse response. Default is
false
. - headers Object - Request headers.
- timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Defaults to
exports.TIMEOUT
, both are 5s. You can usetimeout: 5000
to tell urllib use same timeout on two phase or set them seperately such astimeout: [3000, 5000]
, which will set connecting timeout to 3s and response 5s. - auth String -
username:password
used in HTTP Basic Authorization. - digestAuth String -
username:password
used in HTTP Digest Authorization. - agent http.Agent - HTTP Agent object.
Set
false
if you does not use agent. - httpsAgent https.Agent - HTTPS Agent object.
Set
false
if you does not use agent. - followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
- maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
- formatRedirectUrl Function - Format the redirect url by your self. Default is
url.resolve(from, to)
. - beforeRequest Function - Before request hook, you can change every thing here.
- streaming Boolean - let you get the
res
object when request connected, defaultfalse
. aliascustomResponse
- gzip Boolean - Accept
gzip, br
response content and auto decode it, default isfalse
. - timing Boolean - Enable timing or not, default is
false
.
- method String - Request method, defaults to
When making a request:
await request('https://example.com', {
method: 'GET',
data: {
'a': 'hello',
'b': 'world',
}
});
For GET
request, data
will be stringify to query string, e.g. http://example.com/?a=hello&b=world
.
For others like POST
, PATCH
or PUT
request,
in defaults, the data
will be stringify into application/x-www-form-urlencoded
format
if content-type
header is not set.
If content-type
is application/json
, the data
will be JSON.stringify
to JSON data format.
options.content
is useful when you wish to construct the request body by yourself,
for example making a content-type: application/json
request.
Notes that if you want to send a JSON body, you should stringify it yourself:
await request('https://example.com', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
content: JSON.stringify({
a: 'hello',
b: 'world',
})
});
It would make a HTTP request like:
POST / HTTP/1.1
host: example.com
content-type: application/json
{
"a": "hello",
"b": "world"
}
This exmaple can use options.data
with application/json
content type:
await request('https://example.com', {
method: 'POST',
headers: {
'content-type': 'application/json'
},
data: {
a: 'hello',
b: 'world',
}
});
Upload a file with a hello
field.
await request('https://example.com/upload', {
method: 'POST',
files: __filename,
data: {
hello: 'hello urllib',
},
});
Upload multi files with a hello
field.
await request('https://example.com/upload', {
method: 'POST',
files: [
__filename,
fs.createReadStream(__filename),
Buffer.from('mock file content'),
],
data: {
hello: 'hello urllib with multi files',
},
});
Custom file field name with uploadfile
.
await request('https://example.com/upload', {
method: 'POST',
files: {
uploadfile: __filename,
},
});
Uploads a file with formstream:
import formstream from 'formstream';
const form = formstream();
form.file('file', __filename);
form.field('hello', '你好urllib');
await request('https://example.com/upload', {
method: 'POST',
headers: form.headers(),
stream: form,
});
Response is normal object, it contains:
status
orstatusCode
: response status code.-1
meaning some network error likeENOTFOUND
-2
meaning ConnectionTimeoutError
headers
: response http headers, default is{}
size
: response sizeaborted
: response was aborted or notrt
: total request and response time in ms.timing
: timing object if timing enable.remoteAddress
: http server ip addressremotePort
: http server ip portsocketHandledRequests
: socket already handled request countsocketHandledResponses
: socket already handled response count
NODE_DEBUG=urllib:* npm test
export from undici
import { strict as assert } from 'assert';
import { MockAgent, setGlobalDispatcher, request } from 'urllib';
const mockAgent = new MockAgent();
setGlobalDispatcher(mockAgent);
const mockPool = mockAgent.get('http://localhost:7001');
mockPool.intercept({
path: '/foo',
method: 'POST',
}).reply(400, {
message: 'mock 400 bad request',
});
const response = await request('http://localhost:7001/foo', {
method: 'POST',
dataType: 'json',
});
assert.equal(response.status, 400);
assert.deepEqual(response.data, { message: 'mock 400 bad request' });
Tests | Samples | Result | Tolerance | Difference with slowest |
---|---|---|---|---|
http - no keepalive | 15 | 6.38 req/sec | ± 2.44 % | - |
http - keepalive | 10 | 6.77 req/sec | ± 2.35 % | 6.13 % |
urllib2 - request | 45 | 40.13 req/sec | ± 2.88 % | 528.66 % |
urllib3 - request | 10 | 58.51 req/sec | ± 2.52 % | 816.64 % |
undici - pipeline | 5 | 59.12 req/sec | ± 2.47 % | 826.18 % |
undici - fetch | 15 | 60.42 req/sec | ± 3.00 % | 846.60 % |
undici - dispatch | 5 | 60.58 req/sec | ± 1.39 % | 848.99 % |
undici - stream | 5 | 61.30 req/sec | ± 1.31 % | 860.39 % |
undici - request | 5 | 61.74 req/sec | ± 2.03 % | 867.20 % |
Tests | Samples | Result | Tolerance | Difference with slowest |
---|---|---|---|---|
urllib2 - request | 51 | 1465.40 req/sec | ± 14.40 % | - |
undici - fetch | 40 | 3121.10 req/sec | ± 2.82 % | 112.99 % |
http - no keepalive | 45 | 3355.42 req/sec | ± 2.84 % | 128.98 % |
http - keepalive | 51 | 5179.55 req/sec | ± 36.61 % | 253.46 % |
urllib3 - request | 30 | 7045.86 req/sec | ± 2.93 % | 380.82 % |
undici - pipeline | 50 | 8306.92 req/sec | ± 2.99 % | 466.87 % |
undici - request | 51 | 9552.59 req/sec | ± 13.13 % | 551.88 % |
undici - stream | 45 | 12523.45 req/sec | ± 2.97 % | 754.61 % |
undici - dispatch | 51 | 12970.18 req/sec | ± 3.15 % | 785.10 % |
This project follows the git-contributor spec, auto updated at Fri Sep 09 2022 21:45:49 GMT 0800
.