semantics/documentation of Lwt_unix.fork

[gares]

As per fc633f8 fork is supposed to "cancel" the pending jobs. Unfortunately the doc does not say it does so in a delayed fashion. A comment hints that it is to give change to do an exec sys call (which would anyway "cancel" all pending jobs).

We were unlucky, the first call after fork in our code was triggering an error in a promise that we though would have been cancelled by fork. It is unclear to me if yielding there would guarantee this cannot happen...

CC @maximedenes

[aantron]

Part of the issue here is that we (well, I) failed to post the docs for Lwt 5.3.0. The docs for Lwt_unix.fork now say:

lwt/src/unix/lwt_unix.cppo.mli

Lines 182 to 202 in 20466b8

	val fork : unit -> int
	(** [fork ()] does the same as [Unix.fork]. You must use this
	function instead of [Unix.fork] when you want to use Lwt in the
	child process.
	Notes:
	- In the child process all pending [Lwt_unix] I/O jobs are abandoned.
	This may cause the child's copy of their associated promises to remain
	forever pending.
	- If you are going to use Lwt in the parent and the child, it is
	a good idea to call {!Lwt_io.flush_all} before callling
	{!fork} to avoid double-flush.
	- Otherwise, if you will not use Lwt in the child, call
	{!Lwt_main.Exit_hooks.remove_all} to avoid Lwt calling {!Lwt_main.run}
	during process exit.
	- None of the above is necessary if you intend to call [exec]. Indeed, in
	that case, it is not even necessary to use [Lwt_unix.fork]. You can use
	[Unix.fork].
	- To abandon some more promises, see
	{!Lwt_main.abandon_yielded_and_paused}. *)

i.e. jobs are not canceled, they are abandoned, and it tries to imply the differences between jobs and promises — the jobs are the actual pending or running system calls, and they are simply abandoned by the child (in truth, they just don't exist in the child, and Lwt removes them from its data structures). That causes the promsies that are "attached" to them to simply be "forgotten" by Lwt.

Would these docs have helped you? Is there some way we can further improve them? We will also post them :)

[gares]

Is the yield () >>= cancel_all still there, or you just cancel_all now? (cancel_all is pseudocode).

My impression was that fork did not abandon promises now, and our code was fork >>= fun pid -> if pid != 0 then dup2 null stdin, with a pending promise on read stdin. I'm pretty sure we saw that promise receive End_of_file after fork returned (I guess dup2 closes, hence the EOF is received by the child's copy of the promise).

If this scenario is possible, then the doc is still incorrect, since a user cannot assume that jobs are abandoned in the child as it starts.
Maybe we failed to understand exactly what was going on, it is not easy to debug these "races", but we could not see any other reason for the read-blocked on stdin promise in the child to get EOF.

[aantron]

fork doesn't abandon promises, it abandons the actual I/O jobs, i.e. the system threads and system calls that are being currently executed/monitored by Lwt. It then makes an effort to cancel their associated promises, as you describe:

lwt/src/unix/lwt_unix.cppo.ml

Lines 2293 to 2311 in 20466b8

	let fork () =
	match Unix.fork () with
	| 0 ->
	(* Reset threading. *)
	reset_after_fork ();
	(* Stop the old event for notifications. *)
	Lwt_engine.stop_event !event_notifications;
	(* Reinitialise the notification system. *)
	event_notifications := Lwt_engine.on_readable (init_notification ()) handle_notifications;
	(* Collect all pending jobs. *)
	let l = Lwt_sequence.fold_l (fun (_, f) l -> f :: l) jobs [] in
	(* Remove them all. *)
	Lwt_sequence.iter_node_l Lwt_sequence.remove jobs;
	(* And cancel them all. We yield first so that if the program
	do an exec just after, it won't be executed. *)
	Lwt.on_termination (Lwt_main.yield ()) (fun () -> List.iter (fun f -> f Lwt.Canceled) l);
	0
	| pid ->
	pid

Unfortunately the (very old) internal comments in the above code also conflate "jobs" and "promises." They probably predated when we started using the word "promise" in Lwt.

I don't immediately know what the reason for the End_of_file is. It's possible that it is a (incorrect?) translation of some other condition done elsewhere in the library.

If you intend to do exec (if I understood you correctly), are you able to use the bare Unix.fork? Lwt_unix.fork in general has proved to be very fragile and error-prone, and I doubt there is a fully correct implementation, at least with Lwt's current internal data structures. Lwt_unix.fork tries to make the child process inherit something like the parent's version of Lwt. If you don't need that, because you are doing exec, then I suggest to get whatever data from Lwt the child will need before exec, before running Unix.fork, not calling back into Lwt at all or using any binds or promises, doing whatever setup the child needs, and then doing the exec. This seems to have been much more stable in practice. If that works for you, perhaps that's what we need to add to the documentation :)