use BufReader to read archive index files

[syphar]

I ran test with an index with 17k files:

• it took ~250ms to load the index
• after this change it only takes 25ms
• for the problematic crate ( big crates are slow to use with new archive-storage #1528 ) it still takes ~250ms

Since the average archive-size for docs is just around 600 I think we should merge and deploy this first.
Using mmap made it only slightly faster, which is why I didn't add the additional dependency.

The next optimization is probably not to load the whole index into memory but to answer the find-file requests while streaming the content of the index. But I would like to gather more data on archive-sizes first and benchmark it.

[jyn514]

for the problematic crate ( big crates are slow to use with new archive-storage #1528 ) it still takes ~250ms

What was it before this change?

The next optimization is probably not to load the whole index into memory but to answer the find-file requests while streaming the content of the index. But I would like to gather more data on archive-sizes first and benchmark it.

👍

[jyn514]

Can we take BufReader here instead, to avoid having a BufReader<BufReader>?

[syphar]

while changing this I saw an issue with that:
sometimes we are using load with a slice. In this case I also used an unnecessary BufReader.

Also, the BufWriter on save doesn't help, because we're saving into a in-memory buffer anyways...

I updated the code, even less now

[syphar]

for the problematic crate ( big crates are slow to use with new archive-storage #1528 ) it still takes ~250ms

What was it before this change?

roughly the same ratio, so over 20 seconds