Upload improvements #2

Merged
ferociousbyte merged 3 commits from main into stable 2026-11-19 10:25:51 +00:00
No description provided.
Measured against an S3 store (Hetzner, hel1, 2026-11-19, a file of 700 KB): the upload was a multipart upload of
one part next to its place (three requests), then asked for, copied and deleted (three more); the copy and the
delete take a tenth of a second and more each. An upload in one request that announces at most
storage:maxBufferedUploadBytes (8 MiB) is now held in memory, checked against its SHA-256 and written to its
place with one PUT. Nothing is stored of an upload that isn't what was announced; a larger one, or one of unknown
length, goes the way it went.

A download asked for the object's size and then for the object. The GET says both (IFileStore.OpenAsync), also
for a Range, whose answer names the size in its Content-Range; a range that starts past the end is refused by
the store, and only then is the size asked for. Against the store, one GET took 71 to 123 ms where HEAD and GET
took 114 to 161 (median of 16 objects, three passes).

The version a client names in If-None-Match is the one its token is for: 304 without asking the store. It was
asked for the size first.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A job says where the content it is about lies, so an upload can stay where it was uploaded to
Some checks failed
/ build (push) Failing after 40s
/ push (push) Has been skipped
bf7684b9a2
unilayer made an upload the asset's content by moving it to {uid}: in S3 a copy and a delete, about half a
second of a completion that takes little more (Hetzner, hel1, 2026-11-19). It now records where the upload
lies instead and moves nothing, so the content is no longer where this service took it to be. Job format 2:
data.content for optimize and files.delete, data.replaced for variants.cleanUp, which then removes the content
of the version before as well. Without them the content is at {uid}, as in format 1, which is still read.

A path a job names is a safe one and never in the snapshots; a job with another fails and touches nothing.

A result is written in its job's format: an instance of the format before reads no newer one, and this
service is updated before the instances are.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A range that starts at the end of an object is opened with the object's size, whatever the store answers
All checks were successful
/ build (push) Successful in 1m11s
/ push (push) Successful in 36s
ff0066b2f6
CI failed (task 18873): SeaweedFS answers a range that starts at the object's end with 206, no content and
"Content-Range: bytes 10-9/10", where Hetzner's store answers 416. That was read as no Content-Range at all and
the length of the empty part taken for the object's size, 0. The size is the Content-Range's whenever it names
one, and a part that names no bytes is nothing to read on from. A part without any word of the object's size
has it asked for.

Run against SeaweedFS 3.97 as CI starts it and against Hetzner (hel1), which I had left out before the push.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
unilayer/storage!2
No description provided.