# @aweftjs/uploads

The uploads battery: a file from a page or a module, kept as bytes in a directory or an
S3-compatible bucket and as a record in the application's own store, served back at
`/files/<id>`. The rules an application varies (which types, how large, what a file must pass
before it is kept, who may read one, who may upload) are configuration, not a fork.
`@aweftjs/uploads/client` is the browser half that posts a file with progress;
`@aweftjs/uploads/s3` is the bucket adapter; the readers this package exports are for any
process.

## Quickstart

The server side is three modules, the store declarations they query, and one rule about order:

```ts
import { auth, paths as authPaths } from '@aweftjs/auth';
import { fromDirectory } from '@aweftjs/modules/node';
import { createServer } from '@aweftjs/server';
import { node } from '@aweftjs/server/node';
import { files } from '@aweftjs/static';
import { createStore, memoryDriver } from '@aweftjs/store';
import { paths as uploadPaths, uploads } from '@aweftjs/uploads';

const store = createStore({ driver: memoryDriver(), declare: { ...authPaths, ...uploadPaths } });

const server = createServer({
	// `uploads` before `files`: static/Files answers every URL it is asked, so it goes last.
	sources: [fromDirectory('./modules'), uploads, files, auth],
	store,
	gate: 'auth/Gate',
	listener: node({ port: 8080 }),
});

await server.start();
```

The page posts a file and keeps the URL it gets back wherever it keeps state:

```tsx
import { createUploads } from '@aweftjs/uploads/client';
import { FileDrop } from '@aweftjs/ui';

const uploads = createUploads();

const changePhoto = async (file: File): Promise<void> => {
	const record = await uploads.upload(file, { progress: (fraction) => bar.set(fraction) });
	profile.image = record.url;   // '/files/<id>', served by uploads/Serve
};

<FileDrop.Button label="Change photo" extensions={['image/*']} multiple={false} onDrop={([file]) => changePhoto(file)} />
```

That is the whole wiring. Under the auth battery's gate the post needs a signed-in user and
the file is readable by anyone with its URL; both are one word of configuration away.

## What it does

**`POST /api/uploads`** takes one file as the request body. `Content-Type` is the file's type
and `X-Upload-Name` its name, percent-encoded. The route refuses before storing a byte when the
type is not one the application takes (415), the declared length is over the cap (413), or
there is no `Content-Length` (411). It reads the first bytes and refuses a file whose bytes are
not its declared type (415) for the types it can tell (png, jpeg, gif, webp, pdf, mp4, webm,
ogg, mp3, wav, flac). Then the body streams into storage while it is counted and hashed: a body
that runs over the cap is cut and removed (413), one shorter than declared is removed (400).
Then the application's `accept`, then the record, then 201 with the record. A refusal reads and
discards what is still arriving, up to twice the cap, so the answer reaches a sender that is
still mid-body through a streaming proxy; the gate's 403 for an anonymous post reads nothing.
A sender whose socket fails mid-body is 400 `incomplete`; storage that fails is the module's
failure, 500 and reported, never a refusal. A name is kept to 255 characters, cut by code
point.

```json
{ "id": "k3jd8sQ2pL0aZx9C", "url": "/files/k3jd8sQ2pL0aZx9C", "user": "u1", "name": "cat.png",
  "type": "image/png", "size": 48213, "sha256": "…", "at": 1757800000000,
  "storage": { "adapter": "directory", "key": "k3jd8sQ2pL0aZx9C" }, "meta": null }
```

Every refusal is JSON `{ reasons: [{ code, message }] }`. Over `concurrent` uploads in flight
the route answers 429.

**`GET /files/<id>`** streams the bytes with the type from the record, the length, an `ETag`
that is the sha256 (304 on `If-None-Match`), `Cache-Control: public, max-age=31536000,
immutable` (an id never changes content), `X-Content-Type-Options: nosniff`,
`Content-Security-Policy: sandbox`, and `Content-Disposition: inline` with the name. `HEAD`
carries the headers and no body, and 404 when the bytes are gone, as GET does. A path that is
not `/files/<one segment>`, or an id with no record, is declined so the next module answers it;
another method on a live file is 405.

**From a module**, name `uploads/Files` in `deps`:

```ts
export const deps = ['uploads/Files'];

export default ({ imports }) => ({
	call: async (args, context) => {
		const csv = await exportOf(context.user);
		// The trusted path: no type rule, no cap, no accept. Bytes in, record out.
		const record = await imports.Files.put(new TextEncoder().encode(csv), { type: 'text/csv', name: 'export.csv', user: context.user });
		return record.url;
	},
});
```

`put(bytes | stream, { type, name?, size?, user?, meta? })` (a stream needs its `size`, and
`type` is one MIME type with no parameters), `get(id)`, `open(id)` for the record and a stream
over its bytes, and `remove(id)`, which takes the bytes first and the record second and answers
whether there was one.

**From any process**, the readers over the store:

```ts
import { records, upload } from '@aweftjs/uploads';

const one = await upload(store, id);
const hers = await records(store, { user, limit: 50 });     // newest first
const same = await records(store, { sha256 });              // the same bytes uploaded twice
```

## Configuring it

The way every battery module is configured: a same-named file in the application's own source
exporting `config`.

```ts
// modules/uploads/Files.ts
import { directory } from '@aweftjs/uploads';
import { s3 } from '@aweftjs/uploads/s3';

export const config = {
	storage: process.env.SPACES_BUCKET
		? s3({ endpoint: process.env.SPACES_ENDPOINT!, region: 'nyc3', bucket: process.env.SPACES_BUCKET, accessKey: process.env.SPACES_KEY!, secretKey: process.env.SPACES_SECRET! })
		: directory('var/uploads'),
	types: ['image/png', 'image/jpeg', 'image/webp', 'audio/mpeg', 'video/mp4'],
	maxBytes: { image: 5 * 1024 * 1024, audio: 25 * 1024 * 1024, video: 25 * 1024 * 1024 },
	accept: async (upload, context) => {
		if (!upload.type.startsWith('image/')) return;
		const verdict = await moderate(await upload.bytes(), upload.type);
		if (!verdict.ok) return { reasons: [{ code: 'moderation', message: verdict.reason }] };
	},
};
```

| module | setting | default | what it is |
|---|---|---|---|
| `uploads/Files` | `storage` | `directory('uploads')` under the working directory | where the bytes go: `directory(path)`, `s3(options)`, or an adapter of your own |
| | `types` | png, jpeg, gif, webp | the MIME types the route takes |
| | `maxBytes` | 10 MiB | one number, or a map by family: the part of the type before the `/` (`image`, `audio`, `video`, any other), and `default` for a family the map does not name |
| | `accept` | none | `(upload, context)`: nothing to accept, `{ reasons }` to refuse; `upload` is `{ id, name, type, size, sha256, user, bytes() }` |
| `uploads/Receive` | `public` | `false` | `true` lets anyone post under a gate that reads it |
| | `concurrent` | 16 | uploads held in flight at once |
| `uploads/Serve` | `public` | `true` | `false` makes every file need a signed-in reader |
| | `allow` | none | `(upload, context)`: nothing to allow, `{ reasons }` for 403 |

A value of the wrong type is refused at load with `invalid-config`.

`accept` runs after the bytes are in storage and before the record is written, so it reads
them back with `bytes()` and its refusal removes them. A throw out of it is 500, reported under
`uploads/Receive`, the bytes removed. Moderation, a quota, a manifest check: all here, none
ships.

**The bucket adapter.** `s3({ endpoint, region, bucket, accessKey, secretKey, prefix? })`
signs every request with Signature Version 4 and streams a put with its length and an unsigned
payload; path-style URLs, so it reaches a bucket on DigitalOcean Spaces, MinIO or AWS alike.
No dependency. `prefix` is letters, digits, `_`, `.`, `/` and `-`, such as `site1/`.

**A key** is letters, digits, `_` and `-`, at most 128 of them; `keyOf(key)` on the root
answers it or throws `invalid-key`, and every adapter applies it before a key becomes a path
or a URL. An adapter of your own is `{ name, put, open, head, remove }` over such keys;
`adapterChecks()` from `@aweftjs/testing` is the suite it passes, the key rule among them.

**Storage, the order rule.** `static/Files` answers every request it is asked, with a file or
with its 404 page, and never declines. Load order follows the order of `sources`, so list
`uploads` before `files`. Listed after, every `/files/<id>` is the 404 page, or the shell with
status 200 under `unknown: 'shell'`, and an `<img>` gets HTML.

## The store, and the record

`upload:<id>` holds `kind`, `user`, `name`, `type`, `size`, `sha256`, `at`, `storage` (the
adapter's name and the key) and `meta` (what `put` was handed, plain values only). It is an
ordinary document; the battery truncates its own documents' tails to nothing. `paths` declares
`kind`, `user`, `sha256`, `type` and `at`; `user` is the auth battery's declaration and the
same path.

An id is any text the key rule takes (letters, digits, `_`, `-`, at most 128), so objects
already in a bucket under such keys are recorded by a script that writes `upload:<oldkey>`
documents through the store with `storage: { adapter: 's3', key: '<oldkey>' }`, and
`uploads/Serve` finds them at `/files/<oldkey>`. An object under a key outside the rule (a
slash, a dot) is copied under one inside it first. A record whose key is outside the rule
names no bytes: it serves 404 and `remove` still takes it.

## What it never decides

Who may upload beyond the gate's word and `accept`. Who may read beyond `public` and `allow`.
Whether a file may be deleted over the wire: no delete route ships, because whose file it is
differs by application (a listing's image belongs to the listing, not the uploader); `remove`
is the tool and the route is yours. When a file expires: no sweep ships; the readers list by
user, hash and time, and your job calls `remove`. What a file means: it never indexes, tags,
dedupes, resizes, strips metadata, transcodes or thumbnails. It never reads a storage setting
from the environment, never lists a directory, never answers a range, never redirects to a
CDN, and never stores an address.

## Known limits

- A served file streams through the process. A bucket's own edge in front of `/files/` (an
  nginx location, a CDN) is the way to keep a busy site's media off the box; the route is then
  never reached.
- Nothing is cached in memory: a file is read from storage on every request that does not
  carry its ETag.
- The record and the bytes are two writes with no transaction across them. The bytes land
  first and the record last, so a crash between them leaves an object nothing names, which the
  bucket's own listing finds and nothing here removes.
- The sniff table knows eleven types; a declared type outside it is trusted as declared, and
  `nosniff` plus `sandbox` are what keep that from becoming a script on the origin.
- `fetch` reports no upload progress in Firefox or Safari, so the client half posts over
  `XMLHttpRequest`. A page that wants `fetch` posts the file itself: `POST /api/uploads` with
  the body, `Content-Type` and `X-Upload-Name`, credentials included.
- No range requests: a `<video>` that seeks pulls the whole file. A need for ranges is a note of
  its own.
- No resumable or chunked uploads: a file has to fit one request, and the listener's
  `maxPayload`, when set, bounds it too.
- A refusal is answered once the body has been drained, and the drain has no clock: a sender
  that stalls mid-body holds its `concurrent` slot until the listener's own request timeout
  ends it (Node's is five minutes).
- A `put` from a module runs none of the rules, on purpose; a module that wants them calls
  `receive` with a stream, the declared type and the size.

## API

Every export of `@aweftjs/uploads`, its signature as the compiler resolves it, and its block comment.

### `@aweftjs/uploads`

#### `Accept`

```ts
type Accept = (upload: Upload, context: unknown) => { readonly reasons: readonly Refusal[]; } | undefined | void | Promise<{ readonly reasons: readonly Refusal[]; } | undefined | void>;
```

The application's rule: nothing to accept, or reasons to refuse. A throw is a defect.

#### `Adapter`

```ts
interface Adapter { readonly name: string; put(key: string, stream: ReadableStream<Uint8Array>, options: PutOptions): Promise<void>; open(key: string): Promise<ReadableStream<Uint8Array> | undefined>; head(key: string): Promise<{ readonly size: number; } | undefined>; remove(key: string): Promise<void>; }
```

Where uploaded bytes are kept, by opaque key. `directory` and `s3` ship; an application
writes another and proves it with `adapterChecks()` from `@aweftjs/testing`.

A key is URL-safe text (`keyOf`), and an adapter refuses any other before it becomes a
path or a URL. `put` reads the stream to its end and keeps nothing when the stream errors.
`open` and `head` answer undefined for a key with no bytes. `remove` of a key with no bytes
is not an error.

#### `Allow`

```ts
type Allow = (upload: UploadRecord, context: unknown) => { readonly reasons: readonly Refusal[]; } | undefined | void | Promise<{ readonly reasons: readonly Refusal[]; } | undefined | void>;
```

The application's read rule for one file: nothing to allow, or reasons to refuse.

#### `Files`

```ts
interface Files { readonly adapter: Adapter; readonly types: readonly string[]; capFor(type: string): number; put(bytes: Uint8Array | ReadableStream<Uint8Array>, fields: PutFields): Promise<UploadRecord>; receive(stream: ReadableStream<Uint8Array>, fields: ReceiveFields, context: unknown): Promise<UploadRecord>; get(id: string): Promise<UploadRecord | undefined>; open(id: string): Promise<{ readonly record: UploadRecord; readonly stream: ReadableStream<Uint8Array>; } | undefined>; remove(id: string): Promise<boolean>; stop(): Promise<void>; }
```

The instance: the adapter and the rules, and what a module and the route call.

#### `Primitive`

```ts
type Primitive = string | number | boolean | null;
```

No block comment on this export.

#### `PutFields`

```ts
interface PutFields { readonly type: string; readonly name?: string | null | undefined; readonly size?: number | undefined; readonly user?: string | null | undefined; readonly meta?: Readonly<Record<string, unknown>> | undefined; }
```

What a module hands `put`.

#### `PutOptions`

```ts
interface PutOptions { readonly type: string; readonly size: number; }
```

What an adapter is told about the bytes it is handed.

#### `Receive`

```ts
interface Receive { readonly public: boolean; readonly routes: { readonly 'POST /api/uploads': (request: Request, context: unknown) => Promise<Response>; }; }
```

The instance: what the gate reads, and the route.

#### `ReceiveFields`

```ts
interface ReceiveFields { readonly type: string; readonly name: string | null; readonly size: number; }
```

What the route hands `receive`: the request's word for the file.

#### `Refused`

```ts
interface Refused extends Error { readonly reason: string; readonly reasons: readonly Refusal[]; }
```

A refusal thrown by `receive`, carrying the reasons the answer should say.

#### `Serve`

```ts
interface Serve { readonly public: boolean; request(request: Request, context: unknown): Promise<Response | undefined>; }
```

The instance: what the gate reads, and the hook the server walks to.

#### `Upload`

```ts
interface Upload { readonly id: string; readonly name: string | null; readonly type: string; readonly size: number; readonly sha256: string; readonly user: string | null; bytes(): Promise<Uint8Array>; }
```

What `accept` is handed: the file as it will be recorded, and a way to read its bytes.

#### `UploadFilter`

```ts
interface UploadFilter { readonly user?: string; readonly sha256?: string; readonly type?: string; readonly since?: number; readonly limit?: number; }
```

No block comment on this export.

#### `UploadRecord`

```ts
interface UploadRecord { readonly id: string; readonly url: string; readonly user: string | null; readonly name: string | null; readonly type: string; readonly size: number; readonly sha256: string; readonly at: number; readonly storage: { readonly adapter: string; readonly key: string; }; readonly meta: Readonly<Record<string, Primitive>> | null; }
```

An upload as plain data: what `put` and the route answer, and what the readers read back.

#### `directory`

```ts
directory: (path: string) => Adapter
```

Bytes kept as files in a directory, one per key.

**Params**

- `path`: the directory, resolved from the working directory and made when it is missing

**Returns** the adapter. A put goes to a temporary name in the same directory and is renamed over the key once every byte is written, so a reader never opens a partial file and a put that fails leaves nothing behind.

**Example**

```ts
// modules/uploads/Files.ts
export const config = { storage: directory('var/uploads') };
```

#### `keyOf`

```ts
keyOf: (key: string) => string
```

The key rule: letters, digits, `_` and `-`, at most 128 of them. An id the keeper mints is
sixteen of those, and an object recorded from a bucket by hand keeps whatever key it had, as
long as it is one of these.

**Throws** `invalid-key` for anything else, before it reaches a path or a URL.

#### `paths`

```ts
paths: Readonly<Record<string, readonly string[]>>
```

The paths the application declares on its store for the readers to query: `kind`, `user`,
`sha256`, `type` and `at` on upload records. `user` is the auth battery's declaration and
the same path, so spreading both declares it once.

**Example**

```ts
const store = createStore({ driver, declare: { ...paths } });
```

#### `records`

```ts
records: (store: Store, filter?: UploadFilter) => Promise<UploadRecord[]>
```

The uploads that match, newest first.

**Params**

- `store`: the application's store, with `paths` declared on it
- `filter`: `user`, `sha256`, `type`, `since`, `limit` (100 unless given)

**Returns** the records, newest first. Every record is opened to be read, so `limit` bounds the reads.

**Example**

```ts
const mine = await records(store, { user, limit: 20 });
const same = await records(store, { sha256 });
```

#### `upload`

```ts
upload: (store: Store, id: string) => Promise<UploadRecord | undefined>
```

One upload as plain data, or undefined when there is none by that id.

**Params**

- `store`: the application's store
- `id`: the upload's id, or its full document name `upload:<id>`

**Returns** the record, with `url` where `uploads/Serve` answers it.

**Example**

```ts
const record = await upload(store, 'k3jd8sQ2pL0aZx9C');
```

#### `uploads`

```ts
uploads: Source
```

The three modules, for `sources`: `uploads/Files` keeps the bytes and the records,
`uploads/Receive` answers `POST /api/uploads`, `uploads/Serve` answers `GET /files/<id>`
(design 262).

**Returns** the source. List it before `static`, whose module answers every request it is asked and would answer `/files/<id>` with the 404 page. The application's own source goes first, and a file there named `modules/uploads/Files.ts` exporting only `config` configures the keeper.

**Example**

```ts
const store = createStore({ driver, declare: { ...auth.paths, ...uploads.paths } });
const server = createServer({ sources: [own, uploads, files, auth], store, gate: 'auth/Gate', listener });
// modules/uploads/Files.ts, in `own`:
export const config = { storage: directory('var/uploads'), types: ['image/png', 'audio/mpeg'], maxBytes: { image: 5e6, audio: 25e6 } };
```

### `@aweftjs/uploads/client`

#### `FileLike`

```ts
interface FileLike { readonly size: number; readonly type: string; readonly name?: string | undefined; }
```

A file as the page holds it: a `Blob`, with the name a `File` carries.

#### `RequestLike`

```ts
interface RequestLike { open(method: string, url: string): void; setRequestHeader(name: string, value: string): void; send(body: unknown): void; abort(): void; readonly status: number; readonly responseText: string; withCredentials: boolean; onload: (() => void) | null; onerror: (() => void) | null; onabort: (() => void) | null; readonly upload: { onprogress: ((event: { readonly lengthComputable: boolean; readonly loaded: number; readonly total: number; }) => void) | null; }; }
```

The part of `XMLHttpRequest` this half uses, so a test hands its own.

#### `UploadError`

```ts
interface UploadError extends Error { readonly reason: 'refused' | 'network' | 'aborted'; readonly status: number; readonly reasons: readonly Refusal[]; }
```

A refusal from the route: its status, and the reasons it answered with.

#### `UploadOptions`

```ts
interface UploadOptions { readonly name?: string | undefined; readonly type?: string | undefined; readonly progress?: ((fraction: number) => void) | undefined; readonly signal?: AbortSignal | undefined; }
```

What one upload may carry beside the file.

#### `Uploads`

```ts
interface Uploads { upload(file: FileLike, options?: UploadOptions): Promise<UploadRecord>; url(id: string): string; }
```

No block comment on this export.

#### `UploadsOptions`

```ts
interface UploadsOptions { readonly origin?: string | undefined; readonly request?: (() => RequestLike) | undefined; }
```

No block comment on this export.

#### `createUploads`

```ts
createUploads: (options?: UploadsOptions) => Uploads
```

The page's way to post a file.

**Params**

- `options`: `origin` and the request seam, both optional

**Returns** `upload` and `url`.

**Example**

```ts
const uploads = createUploads();
const record = await uploads.upload(file, { progress: (f) => bar.set(f) });
avatar.set(record.url);   // '/files/<id>'
```

### `@aweftjs/uploads/s3`

#### `S3Options`

```ts
interface S3Options { readonly endpoint: string; readonly region: string; readonly bucket: string; readonly accessKey: string; readonly secretKey: string; readonly prefix?: string | undefined; }
```

No block comment on this export.

#### `s3`

```ts
s3: (options: S3Options) => Adapter
```

Bytes kept as objects in a bucket.

**Params**

- `options`: where the bucket is and how to sign for it

**Returns** the adapter. Every request is signed with Signature Version 4; a put streams its body with the payload unsigned, so the bytes are read once.

**Throws** `invalid-config` for a missing or empty option. At use, `storage-failed` with the status the bucket answered.

**Example**

```ts
// modules/uploads/Files.ts
export const config = {
	storage: s3({ endpoint: process.env.SPACES_ENDPOINT!, region: 'nyc3', bucket: 'app', accessKey: process.env.SPACES_KEY!, secretKey: process.env.SPACES_SECRET! }),
};
```

## Refusals

Every error this package raises carries a `reason` to switch on and a `fix` that says what to do. These are its reasons, from `errors.txt`.

| reason | fix |
|---|---|
| `aborted` | Nothing to fix; the page cancelled it. |
| `incomplete` | Send the whole body, or send Content-Length equal to what is sent. |
| `invalid-config` | Add a default to maxBytes, or an entry for that family. |
| `invalid-config` | Give accept a function of (upload, context) answering nothing or { reasons }, or leave it null. |
| `invalid-config` | Give allow a function of (upload, context) answering nothing or { reasons }, or leave it null. |
| `invalid-config` | Give concurrent the number of uploads to hold in flight at once, above zero. |
| `invalid-config` | Give directory the path of the directory to keep files in. |
| `invalid-config` | Give maxBytes a number of bytes, or a map by family such as { image: 5000000, default: 25000000 }. |
| `invalid-config` | Give prefix letters, digits, _ . / and -, such as "site1/", or leave it out. |
| `invalid-config` | Give s3 an endpoint URL, a region, a bucket, an accessKey and a secretKey, each a non-empty string. |
| `invalid-config` | Give storage an adapter: directory(path), s3(options), or one of your own. |
| `invalid-config` | Give types a list of MIME types, such as ["image/png", "audio/mpeg"]. |
| `invalid-config` | Set public to true for a route anyone may post to, or false for one that needs a signed-in user. |
| `invalid-config` | Set public to true for files anyone with the URL may read, or false for ones that need a signed-in user. |
| `invalid-key` | Use letters, digits, _ and -, at most 128 characters. |
| `invalid-put` | Give put a MIME type such as "image/png". |
| `invalid-put` | Give put the byte length beside a stream, or hand it the bytes. |
| `network` | Try again; check the origin and the route. |
| `no-store` | Pass store to createServer, or props: { store } to a loader you build yourself. |
| `refused` | Read status and reasons; each reason names what to change. |
| `refused` | Read the reasons; they are the application's own. |
| `storage-failed` | Check the endpoint, the bucket, the region and the keys the s3 adapter was given. |
| `too-large` | Send a smaller file, or raise maxBytes in uploads/Files's config. |
| `unsupported-type` | Send one of the types uploads/Files is configured with, and bytes that are that type. |
| `wrong-length` | Send Content-Length equal to the number of bytes in the body. |

## Recipes

The programs in the stack's gate that use this package, each a job someone would have.

- [`recipes/uploads`](/docs/recipes/uploads): A page uploads pictures under the gate and they paint from `/files/<id>`; each refusal reaches the page with its reason, a module makes a file of its own, and the static battery behind it never sees a file