send-stream
send-stream
is a library for streaming files from the file system or any other source of data.
It supports partial responses (Ranges including multipart), conditional-GET negotiation (If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since) and precompressed content encoding negociation.
It also have high test coverage, typescript typings and has multiple examples using Fastify, Koa, Express or pure NodeJS http/http2 modules.
Installation
This is a Node.js module available through the
npm registry. Installation is done using the
npm install
command:
npm install send-stream
Getting start
Serve all files from a directory (also serve index.html from directories on trailing slash) with Fastify, Koa or Express
See examples for more advanced usages.
Using Fastify (v3.x.x):
const path = require('path');
const fastify = require('fastify');
const { FileSystemStorage } = require('send-stream');
const app = fastify({ exposeHeadRoutes: true });
const storage = new FileSystemStorage(
path.join(__dirname, 'assets'),
{ onDirectory: 'serve-index' }
);
app.get('*', async (request, reply) => {
const result = await storage.prepareResponse(
request.url,
request.raw
);
if (result.statusCode === 404) {
reply.callNotFound(); // let fastify handle 404
return;
}
await result.send(reply.raw);
});
app.listen(3000)
.then(() => {
console.info('listening on http://localhost:3000');
});
Using Koa (v2.x.x):
const path = require('path');
const Koa = require('koa');
const { FileSystemStorage } = require('send-stream');
const app = new Koa();
const storage = new FileSystemStorage(
path.join(__dirname, 'assets'),
{ onDirectory: 'serve-index' }
);
app.use(async (ctx, next) => {
let result = await storage.prepareResponse(
ctx.request.path,
ctx.req
);
if (result.statusCode === 404) {
await next(); // let koa handle 404
return;
}
ctx.status = result.statusCode;
ctx.set(result.headers);
ctx.body = result.stream;
});
app.listen(3000, () => {
console.info('listening on http://localhost:3000');
});
Using Express (v4.x.x):
const path = require("path");
const express = require("express");
const { FileSystemStorage } = require('send-stream');
const app = express();
const storage = new FileSystemStorage(
path.join(__dirname, 'assets'),
{ onDirectory: 'serve-index' }
);
app.get('*', async (req, res, next) => {
try {
let result = await storage.prepareResponse(
req.url,
req
);
if (result.statusCode === 404) {
next(); // let express handle 404
return;
}
await result.send(res);
} catch (err) {
next(err);
}
});
app.listen(3000, () => {
console.info('listening on http://localhost:3000');
});
API
new FileSystemStorage(root, [options])
Create a new FileSystemStorage
which is a stream storage giving access to the files inside the given root folder.
-
The
root
parameter is a the absolute path from which the storage takes the files. -
The
options
parameter let you add some addition options:
mimeTypeLookup
In order to return the content type, the storage will try to guess the mime type thanks to the mime-types
module
(see mime-types module documentation).
This option override the mime-types lookup function which will be used for this purpose.
Example:
new FileSystemStorage(
directory,
{
mimeTypeLookup: filename => {
if (filename.endsWith('.abc')) {
return 'text/abc';
}
return undefined;
}
}
)
mimeTypeDefaultCharset
In order to return the content type, the storage will try to guess the mime type charset thanks to the mime-types
module (see mime-types module documentation).
This option override the mime-types charset function which will be used for this purpose.
Example:
new FileSystemStorage(
directory,
{
mimeTypeDefaultCharset: mimeType => {
if (mimeType === 'text/abc') {
return 'UTF-8';
}
return undefined;
}
}
)
dynamicCompression
Enable dynamic compression of file content.
This can be a boolean or a list of encodings ordered by priority, ['br', 'gzip']
if true
is used.
Activating this option will automatically compress content as brotli or gzip
if the content is detected as compressible and supported by the client.
Note that this is highly recommended to use this option only if you can not use pre-compressed files with the contentEncodingMappings option.
Also when dynamic compression is active, Content-Length
header will be removed
and range requests will be disabled as content length is unknown
Defaults to false
Example:
new FileSystemStorage(
directory,
{
dynamicCompression: true
}
)
mimeTypeCompressible
Function used to determine if a type is compressible (for dynamic compression only)
compressible
module will be used by default
Example:
new FileSystemStorage(
directory,
{
dynamicCompression: true,
mimeTypeCompressible: mimeType => mimeType === 'text/plain'
}
)
dynamicCompressionMinLength
Sets the minimum length of a response that will be dynamically compressed (only when the length is known)
Defaults to 20
Example:
new FileSystemStorage(
directory,
{
dynamicCompression: true,
dynamicCompressionMinLength: 100
}
)
defaultMimeType
Configures the default content type (without charset) that will be used if the content type is unknown.
undefined
by default
Example:
new FileSystemStorage(
directory,
{ defaultMimeType: 'application/octet-stream' }
)
maxRanges
Configure how many ranges can be accessed in one request.
Defaults to 200
Setting it to 1
will disable multipart/byteranges
Setting it to 0
or less will disable range requests
Example:
new FileSystemStorage(
directory,
{ maxRanges: 10 }
)
weakEtags
The storage will generate strong etags by default, when set to true the storage will generate weak etags instead.
false
by default
Example:
new FileSystemStorage(
directory,
{ weakEtags: true }
)
contentEncodingMappings
Configure content encoding file mappings.
You can use this option to serve pre-compressed files. (see this this example to precompress files with NodeJS)
This is a list of objects containing the following properties:
-
matcher
: a regular expression used to detect which files have a pre-compressed version -
encodings
: a list of objets with aname
property which is the encoding name andpath
property which is a string (or function returning a string) containing the path of the compressed file (the regular expression groups frommatcher
will be replaced in the string).
undefined
by default
Example:
new FileSystemStorage(
directory,
{
contentEncodingMappings: [
{
matcher: /^(. \.(?:html|js|css))$/,
encodings: [
{ name: 'br', path: '$1.br' },
{ name: 'gzip', path: '$1.gz' }
]
}
]
}
)
ignorePattern
The storage will ignore files which have any parts of the path matching this pattern.
false
will disable it
Defaults to /^\./
(files/folders beginning with a dot)
Example:
new FileSystemStorage(
directory,
{ ignorePattern: /^myPrivateFolder$/ }
)
fsModule
Let you override the fs
module that will be used to retrieve files.
Example:
const memfs = require('memfs');
memfs.fs.writeFileSync('/hello.txt', 'world');
new FileSystemStorage(
directory,
{ fsModule: memfs }
)
onDirectory
Determine what should happen on directory requests (trailing slash)
-
false
to return an error -
'list-files'
to list the files of directories -
'serve-index'
to serve the index.html file of directories
Defaults to false
Note that you can customize the html template used for 'list-files'
by overiding getDirectoryListing
method.
Example:
new FileSystemStorage(
directory,
{ onDirectory: 'list-files' }
)
storage.prepareResponse(path, req, [options])
Create asynchronously a new StreamResponse
for the given path relative to root ready to be sent to a server response.
Note that you can use storage.send(path, req, res, [options])
instead if you don't need to treat the response before sending.
-
The
path
parameter is a urlencoded path (urlencoded) or an array of path parts (should always start with '').Query params will be ignored if present.
For example,
'/my directory/index.html?param=value'
is the equivalent of['', 'my directory', 'index.html']
. -
The
req
is the related request, it can be ahttp.IncomingMessage
or ahttp2.Http2ServerRequest
or ahttp2.IncomingHttpHeaders
. -
The
options
parameter let you add some addition options:
cacheControl
Custom cache-control header value, overrides storage value (public, max-age=0
by default)
false
to remove header
Example:
await storage.prepareResponse(
req.url,
req,
{ cacheControl: 'public, max-age=31536000' }
)
lastModified
Custom last-modified header value, overrides storage value (defaults to mtimeMs converted to UTC)
false
to remove header
Example:
await storage.prepareResponse(
req.url,
req,
{ lastModified: 'Wed, 21 Oct 2015 07:28:00 GMT' }
)
etag
Custom etag header value, overrides storage value (defaults to size mtimeMs content encoding)
false
to remove header
Example:
await storage.prepareResponse(
req.url,
req,
{ etag: '"123"' }
)
mimeType
Custom mime type for content-type header value, overrides storage value (defaults to storage content type)
false
to remove header
Example:
await storage.prepareResponse(
req.url,
req,
{ mimeType: 'text/plain' }
)
mimeTypeCharset
Custom content-type charset value, overrides storage value (defaults to storage content type charset mapping)
false
to remove charset
Example:
await storage.prepareResponse(
req.url,
req,
{ mimeTypeCharset: 'UTF-8' }
)
contentDispositionType
Custom content-disposition header type value, overrides storage value
false
to remove header
Example:
await storage.prepareResponse(
req.url,
req,
{ contentDispositionType: 'attachment' }
)
contentDispositionFilename
Custom content-disposition header filename value, overrides storage value
false
to remove filename from header
Example:
await storage.prepareResponse(
req.url,
req,
{ contentDispositionFilename: 'file.txt' }
)
statusCode
Defines the statusCode that will be used in response (instead of 200/206) Setting this will disable conditional GET and partial responses
undefined
by default
Example:
await storage.prepareResponse(
req.url,
req,
{ statusCode: 404 }
)
allowedMethods
By default GET and HEAD are the only allowed http methods, set this parameter to change allowed methods
['GET', 'HEAD']
by default
Example:
await storage.prepareResponse(
req.url,
req,
{ allowedMethods: ['POST'] }
)
storage.send(path, req, res, [options])
The parameters are the same as for storage.prepareResponse(path, req, [options])
and streamResponse.send(res, [options])
and the result is the same as if you call the two methods consecutively.
You may prefer to use this method if you don't need to use the stream response.
streamResponse.statusCode
The status code that match the required resource.
For example, it can be 200 if the file is found, 206 for a range request, 404 if the file does not exists, ...
streamResponse.headers
The headers that match the required resource.
streamResponse.error
If an error occured while reading (e.g. if the file is not found), a 404 status is returned and this property will contains the error which have been returned by the storage. See errors
streamResponse.send(res, [options])
Send the current response through the response in parameter
-
The
res
parameter is the related response, it can be ahttp.ServerResponse
or ahttp2.Http2ServerResponse
or ahttp2.ServerHttp2Stream
. -
The
options
parameter let you add some addition options:
ignorePrematureClose
Ignore premature close errors, if any error occurs while sending, it will be ignored
Defaults to true
(since clients can stop transfer anytime, forwarding errors can be annoying)
streamResponse.dispose()
Disposes of resources within the stream response object
Errors
StorageError
All errors inherits from this one.
The following property is available:
-
reference
: the storage reference linked to the error
MethodNotAllowedStorageError
(extends StorageError)
The HTTP method is not allowed (405 HTTP code)
PreconditionFailedStorageError
(extends StorageError)
The precondition set in headers have failed (412 HTTP code)
RangeNotSatisfiableStorageError
(extends StorageError)
The ranges set in headers are not statisfiable (416 HTTP code)
FileSystemStorageError
(extends StorageError)
All errors from FileSystemStorage inherits from this one.
The following additional property is available:
-
pathParts
: the path parts linked to the error
MalformedPathError
(extends FileSystemStorageError)
The path cannot be parsed for some reason.
NotNormalizedPathError
(extends FileSystemStorageError)
Storages only accept normalized paths for security reasons.
For example, '/../index.html'
access will be refused.
The following additional property is available:
-
normalizedPath
: the encoded normalized path (you can redirect to it if you want to)
ForbiddenCharacterError
(extends FileSystemStorageError)
Storages refuse some forbidden characters like encoded slashes.
ConsecutiveSlashesError
(extends FileSystemStorageError)
Storages refuse pathes like '/dir//index.html'
because it should not contain two consecutive slashes.
TrailingSlashError
(extends FileSystemStorageError)
Storages refuse pathes like '/dir/'
because it is probably pointing to a directory.
IgnoredFileError
(extends FileSystemStorageError)
Storages can ignore some files/folders composing the path (see ignorePattern).
IsDirectoryError
(extends FileSystemStorageError)
Storages refuse pathes matching directories.
The following additional property is available:
-
resolvedPath
: the resolved file system path
DoesNotExistError
(extends FileSystemStorageError)
When the file can not be found.
The following additional property is available:
-
resolvedPath
: the resolved file system path
Other source of data
You can extend the class Storage
if you want to stream files from something else than the file system.
Take a look at this MongoDB/GridFS storage example to have an example.
You can also take a look to this this Buffer storage example or this JSON storage example to serve data directly from your current route.
If you want to implement something nearer from file system you can also take a look at the GenericFileSystemStorage
class. Take a look at this memory cache storage example to have an example.
Examples
See examples folder in this repository for full examples
Serve files
const storage = new FileSystemStorage(directory);
...
let result = await storage.prepareResponse(
req.url,
req
);
await result.send(res);
Serve files with directory index.html
const storage = new FileSystemStorage(
directory,
{ onDirectory: 'serve-index' }
);
...
let result = await storage.prepareResponse(
req.url,
req
);
await result.send(res);
Serve files with directory listing
const storage = new FileSystemStorage(
directory,
{ onDirectory: 'list-files' }
);
...
let result = await storage.prepareResponse(
req.url,
req
);
await result.send(res);
Serve files and add CORS headers
const storage = new FileSystemStorage(directory);
...
let result = await storage.prepareResponse(
req.url,
req
);
result.headers['Access-Control-Allow-Origin'] = '*';
result.headers['Access-Control-Allow-Headers'] = 'Origin, X-Requested-With, Content-Type, Accept, Range';
await result.send(res);
Serve one file specifically
const storage = new FileSystemStorage(directory);
...
let result = await storage.prepareResponse(
'/index.html',
req
);
await result.send(res);
Serve index.html instead of 404 for history.pushState applications
const storage = new FileSystemStorage(directory);
...
let result = await storage.prepareResponse(
req.url,
req
);
// if path is not found then rewrite to root index.html
if (result.error instanceof FileSystemStorageError) {
result = await storage.prepareResponse(
'/index.html',
req
);
}
await result.send(res);
Serve files and add CSP (Content-Security-Policy) header when content is html
const storage = new FileSystemStorage(directory);
...
let result = await storage.prepareResponse(
req.url,
req
);
if (
!result.error
&& result.storageInfo?.mimeType === 'text/html'
) {
result.headers['Content-Security-Policy'] = "script-src 'self'";
// you can also add some other security headers:
// result.headers['X-Frame-Options'] = "SAMEORIGIN";
// result.headers['Referrer-Policy'] = "no-referrer";
// result.headers['Feature-Policy'] = "...";
}
await result.send(res);