Batched requests

[clostao]

No description provided.

[teor2345]

This will work, and once I implement the first stage of autonomys/subspace#3316 , it will re-use pieces most of the time.

But if an object crosses multiple pieces, the last piece will be downloaded twice (once at the end of the batch for the first piece, because that object needs data from both pieces, and again at the start of the batch for the last piece).

If you want to re-use even more pieces, you could batch groups with nearby piece indexes together. That way, you'll re-use the pieces in this situation as well:

1. There are objects in nearby pieces, and one object crosses multiple pieces
2. The first piece is re-used for all the objects in the batch in that piece
3. The object that crosses multiple pieces re-uses the first piece, and downloads the later pieces (up to 5)
4. The last piece is only downloaded once, and re-used for the next objects in the batch

You can actually combine as many nearby pieces as you like this way. So if you want the most download-efficient code, an alternative algorithm is:

1. Sort the mappings by piece index
2. Split the batch when the next object couldn't possibly share a piece with the last one

If the piece is already cached, the response will be almost instant, because it is just moving some data around, and doing one blake3 hash. (Or up to 4 hashes if the object crosses segments - but that's rare, and will only happen once per batch at most.)

Here is how you can work out if two objects could share a piece:

1. Blocks and objects are limited to 5 MB. So if the difference between the piece indexes in two mappings is greater than 5, they can't share any pieces - and you can split the batch there.
2. If you know the size of the object, you can calculate the number of pieces in the object using: (object_size + 100) / (2^15 * 31), and rounding up. Then split the batch if the difference between the two piece indexes is greater than the number of pieces in the object.

(The calculation is a bit complicated to account for segment padding and headers, and the unused bytes in the cryptographic scalars we use to generate parity pieces.)

[clostao]

I implemented this change in my last commit. As you can see I've implemented that no re-utilisation is to be tried in the case that objects' pieces are not consecutive.

This is because though objects at the protocol level are limitted to 5MB @autonomys/auto-dag-data limits to 64KB since it's the biggest a Bytes input can take for an extrinsic (that is what is used bySystem.remark) so for two objects to share a piece they have to be consecutive.

[teor2345]

Thanks! That seems like an annoying limitation 🙂

Hopefully my next PR will help re-use pieces more, if we're splitting into 64kB objects, then that's a lot of small objects in a single piece for a 1 MB file.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​oven/​bun-linux-aarch64-musl@​1.2.8 ⏵ 1.2.9	-19		-65	-2
	@​oven/​bun-linux-x64-musl@​1.2.8 ⏵ 1.2.9			-65	-2
	@​oven/​bun-linux-x64-musl-baseline@​1.2.8 ⏵ 1.2.9			-65	-2
	@​oven/​bun-darwin-aarch64@​1.2.8 ⏵ 1.2.9	-3		-63	-2
	@​oven/​bun-darwin-x64@​1.2.8 ⏵ 1.2.9	-13		-63	-2
	@​oven/​bun-linux-aarch64@​1.2.8 ⏵ 1.2.9	-2		-63	-2
	@​oven/​bun-linux-x64@​1.2.8 ⏵ 1.2.9			-63	-2
	@​oven/​bun-windows-x64@​1.2.8 ⏵ 1.2.9	-8		-63	-2
	@​oven/​bun-linux-x64-baseline@​1.2.8 ⏵ 1.2.9			-62	-2
	@​oven/​bun-darwin-x64-baseline@​1.2.8 ⏵ 1.2.9	-11		-62	-2
	@​oven/​bun-windows-x64-baseline@​1.2.8 ⏵ 1.2.9	-7		-62	-2
	isarray@​1.0.0 ⏵ 2.0.5			-20
	array-buffer-byte-length@​1.0.2
	which-collection@​1.0.2
	object.groupby@​1.0.3
	is-set@​2.0.3
	is-weakmap@​2.0.2
	is-map@​2.0.3
	define-properties@​1.2.1
See 89 more rows in the dashboard

View full report