Consider explaining the concurrency model in the README

[michaellenaghan]

Given the use of WASM it isn't clear what the concurrency model of the package is. Consider explaining the concurrency model in the README?

[ncruces]

From the Go point-of-view, Wasm is irrelevant. Consider it an implementation detail.

For the database/sql driver, concurrency is just like any other driver. The database/sql API is (for the most part) fully goroutine safe. But check out the docs for details.

For the lower level sqlite3 package, database connections can be used by a single goroutine at a time; so, like the SQLite multi-thread mode.

For how much concurrency you can get out of SQLite, it's mostly the same as SQLite itself: one writer or many readers, but not both (with rollback journal), one writer coexisting with many readers (in Wal mode). On BSD you get slightly less concurrency in rollback mode (readers need to leave sooner for writers to make progress). And with dot-locks multiple processes are not allowed.

For the implementation detail: we don't use Wasm threads. Each connection gets its own isolated (and single-threaded) sandbox.

[michaellenaghan]

Thank you, all very useful to know.

For the implementation detail: we don't use Wasm threads. Each connection gets its own isolated (and single-threaded) sandbox.

Does that imply that connections consume more memory than you might otherwise expect?

[ncruces]

Yes, nothing is shared. Any "inter-instance communication" goes through the VFS. Think of each instance as a different "virtualized" SQLite process.

[michaellenaghan]

Understood, thanks!

Should I close this, or let you do it? (The original suggestion was to document the concurrency model in the README. Until then, maybe this is that documentation?)

[ncruces]

Yes, it's OK to leave open. All this hopefully serves as documentation.

Writing documentation, design docs, examples, is hard, but important.

This kind of discussion is very relevant, but might not fit the README, it'd probably fit better in a wiki page. Even the VFS README might not be the best place for what's in there, as it's more an architectural design doc than a README.

I mostly left it there as (1) it's not the main README, and (2) people should be aware that I rewrote core parts of SQLite. The saying goes that tires are important because they're the only bit of your car that touches the road. Well, I rewrote the only bit of SQLite that touches the disk, and although I test it a lot I'm definitely not as rigorous as the original. I don't want to hide that message in just a design doc.

[michaellenaghan]

This kind of discussion is very relevant, but might not fit the README, it'd probably fit better in a wiki page.

Hmm, I'm not sure I agree. Let me see if I can figure out why.

What I've learned so far, I think, is that the package will have some CPU and memory cost due to WASM — but less than I might have assumed; that it will work correctly on all of the platforms I care about; that it will be roughly as concurrent as SQLite; and that it will work differently, from a memory perspective, than a multi-threaded SQLite. Those aren't just design details! I think people who are evaluating the package, and people who are using the package, should be able to learn those things quickly, without having to click through to other documents. Especially, maybe, because it's WASM-based — and WASM outside of the browser is very much unknown territory to most of us.

(Btw, I'm developing on macOS, and cross-compiling to FreeBSD. But I'm not using your package to avoid CGO. I'm using it because it is, by far, the best-maintained of the Go SQLite packages. That's more valuable to me than "some CPU and memory cost due to WASM.")

[ncruces]

Thanks for the compliment!
I meant that, IMO, the level of detail in the front page docs needs to be measured.
I added a bit more detail to the READMEs. But I'm not very inspired to write docs ATM.