Escape from promiscuous mode with packet siphons

[FelixMcFelix]

This change hangs a callback off the primary MAC client on each underlay device, which gives OPTE primary access to any packets coming into a device in both the interrupt and poll-thread contexts. This allows us to leave promiscuous mode behind, but OPTE is now responsible for handing any unmatched packets back to the actual illumos pipeline.

This change does not unblock all the work that a more systemic fix such as IPD45 and the intended end-state of #62 would get us. We will be in a position where we are able to think about bathcing packets for Viona's benefit in the Rx path from the NIC, but not e.g. Tx or Loopback (as Viona is still hamstrung by the same-flow constraint in mac_tx). Moreover, when we get flows reworked we'll be in a far less responsible position for every service on the rack's traffic.

Companion change to (read: reliant on) https://github.com/oxidecomputer/stlouis/issues/744.

---

Quoting from a comment on that stlouis issue, since it's private:

Having deployed this on berlin (with #749 and #750 in tow so that we actually perform MTU boosting), instance performance numbers with a full control plane and viona in play are at:

	$N	Mean (Gbps)	std_dev	Median (Gbps)
Sled-to-sled (anti-affinity)	1	8.32	0.34	8.22
	2	10.23	0.42	10.30
	8	9.95	0.07	9.94
Sled-local (affinity)	1	14.15	0.10	14.10
	2	13.68	0.38	13.70
	8	13.23	0.11	13.30

Although imprecise with regards to modelling the old behaviour (masses of geneve packets aren't being dropped by the IP stack), running snoop on both cxgbe interfaces at the same time to force promisc mode gives us ~6.95/8.02/8.83 Gbps (-P1/2/8). We'll see how closely that hews to what we get on the next dogfood update.

The command run here is iperf3 -c 172.30.0.6 -Z -O2 -P $N, using alpine linux instances at 8 CPUs + 4GiB RAM, 10 trials each. It should be noted that sled-local cases shouldn't benefit from removing promisc, but are useful for comparison against an upper bound.

[pfmooney]

Comment about this whole block: What are we supposed to do if any of these unexpected conditions result in an error being emitted, since we're altering state as part of the attempted tear-down. take()-ing the underlay for example. If we emit EBUSY, then we won't have a path back to re-take()-ing it, right?

[FelixMcFelix]

Some of these errors are suggestive of programming errors, but we could do worse than to check all preconditions at the first instance and then proceed (the conditions we've setup, no ports && write lock held, should ensure nothing can violate those expectations). I'll revisit the logic here in the morning.