file-box

FileBox is a virtual container for packing a file data into it for future read, and easily transport between servers with the least payload, no mater than where it is (local path, remote url, or cloud storage).

Currently the FileBox supports almost all kinds of the data input/output methods/formats:

File Type	Pack Method	Unpack Method	Description
Local File	`fromFile()`	`toFile()`	Local file in file system
Remote URL	`fromUrl()`	`toUrl()`(TBW)	Remote file in a HTTP/HTTPS URL
Buffer	`fromBuffer()`	`toBuffer()`	JavaScript Buffer
Stream	`fromStream()`	`toStream()`	JavaScript Stream
Base64	`fromBase64()`	`toBase64()`	Base64 data
DataURL	`fromDataURL()`	`toDataURL()`	DataURL data
QRCode	`fromQRCode()`	`toQRCode()`	QR Code Image Decode/Encode
UUID	`fromUuid()`	`toUuid()`	UUID by loader/saver helper functions
JSON	`fromJSON()`	`toJSON()`	Serialize/Deserialize FileBox

Examples

The following example demos:

Load local file
Save URL to local file
Convert buffer to stream
Pack from Base64 then Unpack to DataURL

import { FileBox } from 'file-box'

/**
 * 0. Load local file
 */
const fileBox0 = FileBox.fromFile('/tmp/file.jpg')

/**
 * 1. Save URL to File
 */
const fileBox1 = FileBox.fromUrl(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
  'logo.jpg',
)
fileBox1.toFile('/tmp/file-box-logo.jpg')

/**
 * 2. Convert Buffer to Stream
 */
import fs from 'fs'
const fileBox2 = FileBox.fromBuffer(
  Buffer.from('world'),
  'hello.txt',
)
const writeStream = fs.createWriteStream('/tmp/hello.txt')
fileBox2.pipe(writeStream)

/**
 * 3. Pack Base64, Unpack to DataURL
 */
const fileBox3 = FileBox.fromBase64('d29ybGQK', 'hello.txt')
fileBox3.toDataURL()
        .then(console.log)
// Output: data:text/plain;base64,d29ybGQK

Known Issues

TypeError [ERR_UNESCAPED_CHARACTERS]: Request path contains unescaped characters #56

API Reference

API docs on Paka

1. Load File in to Box

1.1 `fromFile(filePath: string, name?: string, md5?: string): FileBox`

Alias: fromLocal()

About optional arguments:

name: filename, if not passed, will be parsed from file path, url, etc.

md5: file md5 code, only for checking purposes and will not be computed from file.

const fileBox = FileBox.fromLocal('/tmp/test.txt')

1.2 `fromUrl(url: string, options?: {headers?: HTTP.OutgoingHttpHeaders, name?: string, size?: string, md5?: string}): FileBox`

Alias: fromRemote()

const fileBox = FileBox.fromUrl(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
  {name: 'logo.jpg'},
)

1.3 `fromStream(stream: Readable, name?: string, md5?: string): FileBox`

Will be named 'stream.dat' if no name is provided.

const fileBox = FileBox.fromStream(res, '/tmp/download.zip')

1.4 `fromBuffer(buffer: Buffer, name?: string, md5?: string): FileBox`

Will be named 'buffer.dat' if no name is provided.

const fileBox = FileBox.fromBuffer(buf, '/tmp/download.zip')

1.5 `FileBox.fromBase64(base64: string, name?: string, md5?: string): FileBox`

Decoded a base64 encoded file data. Will be named 'base64.dat' if no name is provided.

const fileBox = FileBox.fromBase64('d29ybGQK', 'hello.txt')
fileBox.toFile()

1.6 `FileBox.fromDataURL(dataUrl: string, name?: string, md5?: string): FileBox`

Decoded a DataURL data. Will be named 'data-url.dat' if no name is provided.

const fileBox = FileBox.fromDataURL('data:text/plain;base64,d29ybGQK', 'hello.txt')
fileBox.toFile()

1.7 `FileBox.fromJSON()`

Restore a FileBox.toJSON() text string back to a FileBox instance.

const restoredFileBox = FileBox.fromJSON(jsonText)

1.8 `FileBox.fromQRCode(qrCodeValue: string, md5?: string)`

Get a FileBox instance that represent a QR Code value.

const fileBox = FileBox.fromQRCode('https://github.com')
fileBox.toFile('qrcode.png')

1.9 `FileBox.fromUuid(uuid: string, options?: {name?: string, size?: string, md5?: string})`

Load a FileBox from a UUID. Will be named ${uuid}.dat if no name is provided.

See: FileBox.setUuidLoader()

2. Get File out from Box

2.1 `toFile(name?: string): Promise<void>`

Save file to current work path(cwd) of the local file system with the default name.

if name specified with a full path, then will use the speficied file name instead.

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
await fileBox.toFile('/tmp/logo.jpg')

2.2 `toStream(): Readable`

Get the stream of file data.

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const readableStream = fileBox.toStream()

2.3 `pipe(destination: Writable): Promise<void>`

Pipe to a writable stream.

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const writableStream = fs.createWritable('/tmp/logo.jpg')
fileBox.pipe(writableStream)

2.4 `toBase64(): Promise<string>`

Get the base64 data of file.

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const base64Text = await fileBox.toBase64()
console.log(base64Text) // Output: the base64 encoded data of the file

2.5 `toJSON(): string`

Get the JSON.stringify-ed text.

Not Implement Yet: Working In Progress...

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
const jsonText1 = fileBox.toJSON()
const jsonText2 = JSON.stringify(fileBox)
assert(jsonText1 === jsonText2)

console.log(jsonText1) // Output: the stringified data of the fileBox

const restoredFileBox = fileBox.fromJSON(jsonText1)
restoredFileBox.toFile('/tmp/file-box-logo.jpg')

2.6 `toDataURL(): Promise<string>`

Get the DataURL of the file.

const fileBox = FileBox.fromFile('tests/fixtures/hello.txt')
const dataUrl = await fileBox.toDataURL()
console.log(dataUrl) // Output: data:text/plain;base64,d29ybGQK'

2.7 `toBuffer(): Promise<Buffer>`

Get the Buffer of the file.

const fileBox = FileBox.fromFile('tests/fixtures/hello.txt')
const buffer = await fileBox.toBuffer()
console.log(buffer.toString()) // Output: world

2.8 `toQRCode(): Promise<string>`

Decode the QR Code value from the file.

const fileBox = FileBox.fromFile('qrcode.jpg')
const qrCodeValue = await fileBox.toQRCode()
console.log(`QR Code decoded value is: "${qrCodeValue}"`)
// Output: QR Code decoded value is: "https://github.com"

2.9 `toUuid(): Promise<string>`

Save the FileBox to a UUID file and return the UUID.

See: FileBox.setUuidSaver()

2.10 `toJSON(): string`

Encode a FileBox instance to JSON string so that we can transfer the FileBox on the wire.

const fileBox = FileBox.fromBase64('RmlsZUJveEJhc2U2NAo=')

const jsonText = JSON.stringify(fileBox)
// the above code equals to the following line of code:
// const jsonText = fileBox.toJSON()

// we will get the serialized data for this FileBox:
console.log(jsonText)
// Output: {"name":"qrcode.png","metadata":{},"boxType":1,"base64":"RmlsZUJveEJhc2U2NAo="}

// restore our fleBox:
// const newFileBox = FileBox.fromJSON(jsonText)

Limitation

Because we want to enable the JSON.stringify(fileBox), which will call fileBox.toJSON(), so the toJSON() can not be async, which means we can only support limited FileBoxType(s):

FileBoxType.Base64
FileBoxType.Url
FileBoxType.QRCode

For other types like FileBoxType.File, FileBoxType.Buffer, FileBoxType.Stream, etc, we need to transform them to FileBoxType.Base64 before we call toJSON:

const fileBoxLazy = FileBox.fromFile('./test.txt')
const base64 = await fileBoxLazy.toBase64()

const fileBox = FileBox.fromBase64(base64, 'test.txt')
// fileBox will be serializable because it do not need async operations

const jsonText = JSON.stringify(fileBox)
console.log(jsonText)

3. Misc

3.1 `name`

File name of the file in the box

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
console.log(fileBox.name) // Output: file-box-logo.jpg

3.2 `md5`

File md5 of the file in the box. The value is set by user, not computed from file.

const fileBox = FileBox.fromUrl(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
  {md5: 'computed-md5-string'}
)
console.log(fileBox.md5) // Output: computed-md5-string

3.3 `metadata: Metadata { [key: string]: any }`

Metadata for the file in the box. This value can only be assigned once, and will be immutable afterwards, all following assign or modify actions on metadata will throw errors

const fileBox = FileBox.fromRemote(
  'https://huan.github.io/file-box/images/file-box-logo.jpg',
)
fileBox.metadata = {
  author      : 'huan',
  githubRepo  : 'https://github.com/huan/file-box',
}

console.log(fileBox.metadata)       // Output: { author: 'huan', githubRepo: 'https://github.com/huan/file-box' }
fileBox.metadata.author = 'Tank'  // Will throw exception

3.4 `version(): string`

Version of the FileBox

3.5 `toJSON(): string`

Serialize FileBox metadata to JSON.

3.6 `ready(): Promise<void>`

Update the necessary internal data and make everything ready for use.

3.7 `syncRemoteName(): Promise<void>`

Sync the filename with the HTTP Response Header

HTTP Header Example:

Content-Disposition: attachment; filename="filename.ext"

3.8 `type: FileBoxType`

Return the type of the current FileBox instance.

The currently supported types are defined at file-box-type.ts as the following demonstrated:

enum FileBoxType {
  Unknown = 0,
  Base64  = 1,
  Url     = 2,
  QRCode  = 3,
  Buffer  = 4,
  File    = 5,
  Stream  = 6,
  Uuid    = 7,
}

3.9 `FileBox.setUuidLoader(loader: UuidLoader): void`

Required by static method FileBox.fromUuid()

class FileBoxUuid extends FileBox {}

const loader: UuidLoader = async (uuid: string) => {
  const stream = new PassThrough()
  stream.end('hello, world!')
  return stream
})

FileBoxUuid.setUuidLoader(loader)
const fileBox = FileBoxUuid.fromUuid('12345678-1234-1234-1234-123456789012', 'test.txt')
await fileBox.toFile('test.txt')

The UuidLoader is a function that takes a UUID and return a readable stream.

type UuidLoader = (this: FileBox, uuid: string) => Readable

3.10 `FileBox.setUuidSaver(saver: UuidSaver): void`

Required by instance method fileBox.toUuid()

class FileBoxUuid extends FileBox {}

const saver: UuidSaver = async (stream: Readable) => {
  // save the stream and get uuid
  return '12345678-1234-1234-1234-123456789012'
})

FileBoxUuid.setUuidSaver(saver)

const fileBox = FileBoxUuid.fromFile('test.txt')
const uuid = await fileBox.toUuid()

The UuidSaver is a function that takes a readable stream and return a UUID promise.

type UuidSaver = (this: FileBox, stream: Readable) => Promise<string>

3.11 `size`

The file box size in bytes. (-1 means unknown)

It is not the size of the target (boxed) file itself.

For example:

const fileBox = FileBox.fromUrl('http://example.com/image.png')
console.log(fileBox.size)
// > 20 <- this is the length of the URL string

3.12 `remoteSize`

The remote file size in bytes. (-1 or undefined means unknown)

For example:

const fileBox = FileBox.fromUrl('http://example.com/image.png')
await fileBox.ready()
console.log(fileBox.remoteSize)
// > 102400 <- this is the size of the remote image.png

Features

Present A File by Abstracting It's Meta Information that supports Reading & toJSON() API.
Follow DOM File/BLOB Interface
Present a file that could be: Local, Remote, Stream
Lazy load
Serializable
Can be Transfered from server to server, server to browser.

SCHEMAS

Url

Node.js Documentation > URL Strings and URL Objects

┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│                                            href                                             │
├──────────┬──┬─────────────────────┬─────────────────────┬───────────────────────────┬───────┤
│ protocol │  │        auth         │        host         │           path            │ hash  │
│          │  │                     ├──────────────┬──────┼──────────┬────────────────┤       │
│          │  │                     │   hostname   │ port │ pathname │     search     │       │
│          │  │                     │              │      │          ├─┬──────────────┤       │
│          │  │                     │              │      │          │ │    query     │       │
"  https:   //    user   :   pass   @ sub.host.com : 8080   /p/a/t/h  ?  query=string   #hash "
│          │  │          │          │   hostname   │ port │          │                │       │
│          │  │          │          ├──────────────┴──────┤          │                │       │
│ protocol │  │ username │ password │        host         │          │                │       │
├──────────┴──┼──────────┴──────────┼─────────────────────┤          │                │       │
│   origin    │                     │       origin        │ pathname │     search     │ hash  │
├─────────────┴─────────────────────┴─────────────────────┴──────────┴────────────────┴───────┤
│                                            href                                             │
└─────────────────────────────────────────────────────────────────────────────────────────────┘

Path

Node.js Documentation > path.parse(path)

┌─────────────────────┬────────────┐
│          dir        │    base    │
├──────┬              ├──────┬─────┤
│ root │              │ name │ ext │
"  /    home/user/dir / file  .txt "
└──────┴──────────────┴──────┴─────┘

History

main v1.5 (Jan 18, 2022)

fileBox.md5 can be set by user. This filed is for the receiver of the filebox to check, and it is not computed from the file.

main v1.4 (Nov 14, 2021)

fileBox.size will be serialized to/from JSON, and present the Content-Length of the file. (-1 means unknown)
mimeType has been renamed to mediaType, and added to the FileBoxInterface

v1.0 (Oct 20, 2021)

Suppert ES Module. (#54)
Add UUID boxType support: FileBox.fromUuid() and FileBox.toUuid()
Add size property to return the size of the file. (-1 means unknown)
Add remoteSize property to present the remote size of the file (if applicable, -1 means unknown)
Add UniformResourceNameRegistry class for providing a production-ready basic UUID management tool.
Add FileBoxInterface, FileBox.isInstance(), and FileBox.isInterface()

Breaking changes:

toJSON format renamed boxType to type
type() has been changed to type
version() has been changed to version

v0.16 master

Throw error when consume a stream twice to prevent data lost. (#50)

v0.14 (Oct 2020)

Add fileBox.type() to return the FileBoxType of a FileBox. (wechaty/wechaty#1918)
Change Readable to stream.Readable for better compatibility. (Jun 27, 2020)
Add chunkerTransformStream to toStream (#44)

v0.12 (Feb 2020)

Add support to JSON.stringify() (#25):

FileBox.fromJSON() - Static method for deserialization
fileBox.toJSON() - Instance method for serialization

v0.10 (Jan 2020)

Add support to QR Code: FileBox.fromQRCode() and FileBox.toQRCode()
Start using @chatie/tsconfig

v0.8 (Jun 2018)

Add two new factory methods: fromBase64(), fromDataURL()
Add toBuffer(), toBase64() and toDataURL() to get the Buffer and BASE64 encoded file data
Add metadata property to store additional information. (#3)

v0.4 (May 2018)

Add headers option for fromRemote() method

v0.2 (Apr 2018)

Initial version.

Thanks

This module is inspired by https://github.com/gulpjs/vinyl and DefinitelyTyped/DefinitelyTyped#12368 when I need a virtual File module for my Chatie project.

Author

Huan Li (LinkedIn), Microsoft AI MVP, huan@chatie.io

Copyright & License

Docs released under Creative Commons
Code released under the Apache-2.0 License

Name		Name	Last commit message	Last commit date
Latest commit History 426 Commits
.github/workflows		.github/workflows
.vscode		.vscode
docs/images		docs/images
scripts		scripts
src		src
tests		tests
.editorconfig		.editorconfig
.eslintrc.cjs		.eslintrc.cjs
.gitignore		.gitignore
.travis.yml		.travis.yml
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
package.json		package.json
tsconfig.cjs.json		tsconfig.cjs.json
tsconfig.json		tsconfig.json

License

huan/file-box

Folders and files

Latest commit

History

Repository files navigation

file-box

Examples

Known Issues

API Reference

1. Load File in to Box

1.1 fromFile(filePath: string, name?: string, md5?: string): FileBox

1.2 fromUrl(url: string, options?: {headers?: HTTP.OutgoingHttpHeaders, name?: string, size?: string, md5?: string}): FileBox

1.3 fromStream(stream: Readable, name?: string, md5?: string): FileBox

1.4 fromBuffer(buffer: Buffer, name?: string, md5?: string): FileBox

1.5 FileBox.fromBase64(base64: string, name?: string, md5?: string): FileBox

1.6 FileBox.fromDataURL(dataUrl: string, name?: string, md5?: string): FileBox

1.7 FileBox.fromJSON()

1.8 FileBox.fromQRCode(qrCodeValue: string, md5?: string)

1.9 FileBox.fromUuid(uuid: string, options?: {name?: string, size?: string, md5?: string})

2. Get File out from Box

2.1 toFile(name?: string): Promise<void>

2.2 toStream(): Readable

2.3 pipe(destination: Writable): Promise<void>

2.4 toBase64(): Promise<string>

2.5 toJSON(): string

2.6 toDataURL(): Promise<string>

2.7 toBuffer(): Promise<Buffer>

2.8 toQRCode(): Promise<string>

2.9 toUuid(): Promise<string>

2.10 toJSON(): string

Limitation

3. Misc

3.1 name

3.2 md5

3.3 metadata: Metadata { [key: string]: any }

3.4 version(): string

3.5 toJSON(): string

3.6 ready(): Promise<void>

3.7 syncRemoteName(): Promise<void>

3.8 type: FileBoxType

3.9 FileBox.setUuidLoader(loader: UuidLoader): void

3.10 FileBox.setUuidSaver(saver: UuidSaver): void

3.11 size

3.12 remoteSize

Features

SCHEMAS

Url

Path

History

main v1.5 (Jan 18, 2022)

main v1.4 (Nov 14, 2021)

v1.0 (Oct 20, 2021)

v0.16 master

v0.14 (Oct 2020)

v0.12 (Feb 2020)

v0.10 (Jan 2020)

v0.8 (Jun 2018)

v0.4 (May 2018)

v0.2 (Apr 2018)

See Also

Thanks

Author

Copyright & License

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 1

Packages 0

Contributors 11

Languages

1.1 `fromFile(filePath: string, name?: string, md5?: string): FileBox`

1.2 `fromUrl(url: string, options?: {headers?: HTTP.OutgoingHttpHeaders, name?: string, size?: string, md5?: string}): FileBox`

1.3 `fromStream(stream: Readable, name?: string, md5?: string): FileBox`

1.4 `fromBuffer(buffer: Buffer, name?: string, md5?: string): FileBox`

1.5 `FileBox.fromBase64(base64: string, name?: string, md5?: string): FileBox`

1.6 `FileBox.fromDataURL(dataUrl: string, name?: string, md5?: string): FileBox`

1.7 `FileBox.fromJSON()`

1.8 `FileBox.fromQRCode(qrCodeValue: string, md5?: string)`

1.9 `FileBox.fromUuid(uuid: string, options?: {name?: string, size?: string, md5?: string})`

2.1 `toFile(name?: string): Promise<void>`

2.2 `toStream(): Readable`

2.3 `pipe(destination: Writable): Promise<void>`

2.4 `toBase64(): Promise<string>`

2.5 `toJSON(): string`

2.6 `toDataURL(): Promise<string>`

2.7 `toBuffer(): Promise<Buffer>`

2.8 `toQRCode(): Promise<string>`

2.9 `toUuid(): Promise<string>`

2.10 `toJSON(): string`

3.1 `name`

3.2 `md5`

3.3 `metadata: Metadata { [key: string]: any }`

3.4 `version(): string`

3.5 `toJSON(): string`

3.6 `ready(): Promise<void>`

3.7 `syncRemoteName(): Promise<void>`

3.8 `type: FileBoxType`

3.9 `FileBox.setUuidLoader(loader: UuidLoader): void`

3.10 `FileBox.setUuidSaver(saver: UuidSaver): void`

3.11 `size`

3.12 `remoteSize`

Packages