Discussion: implications of macOS's new Network.framework

[njsmith]

Network.framework is a new API for networking that Apple is starting to roll out in macOS 10.14+: https://developer.apple.com/documentation/network

It will be a while before we can count on this being available: Steam currently says ~5% of their users are still on 10.11, and ~7% on 10.12; I guess it will take a few years before "everyone" is on 10.14 or better.

Unfortunately the documentation is currently extremely incomplete. (Apparently the main source of info for now is WWDC talks.) But I had a long conversation with @Lukasa last night about what's going on here, so I wanted to write down what I learned.

What is this?

Network.framework provides a general API for doing low-ish level networking (TCP, UDP, TLS), with its own event loop and protocol stack systems. A NWConnection is a container that holds a stack of NWProtocols, which might be something like TLS on top of TCP, or more exotic things like TLS on top of some kind of multiplexing tunnel protocol on top of QUIC or who knows what. Event notification is based around Grand Central Dispatch. (I.e., callback based.)

The general idea is that Apple wants folks to treat Network.framework as a replacement for the BSD socket layer (!), and is even doing standardization work at the IETF in that direction: https://datatracker.ietf.org/wg/taps/about/

It has superpowers that BSD sockets don't, like built-in happy eyeballs (with special access to system-level information that lets it be cleverer than user-space happy eyeballs), better support for the weird and mutable network configurations you find in phones, etc. On macOS this is "nice to have", on iOS it's fairly important, and on watchOS it's absolutely mandatory (they simply don't let you use BSD sockets).

Currently, you cannot define custom "bottom level" NWProtocols – every NWConnection has to ultimately use one of the built-in transports like TCP. You can define custom framing protocols on top of this. So basically this means we can adapt a NWConnection to be a trio.abc.Stream, but we can't easily go the other way and use an NWConnection-based protocol stack on top of a trio.abc.Stream.

As part of this, their old TLS library – SecureTransport – is now deprecated. It's still getting security fixes for now and presumably for the next N years, but it will never get new features like TLS 1.3. Instead you're supposed to use the new TLS library that's built into Network.framework. Put together with the previous paragraph, this means that you can't use their new TLS library in a sans-IO way or over arbitrary transports; you can only run it over real-actual TCP or similar.

Implications for Trio's TLS stack

In principle, it would be nice to use Apple's new TLS library (see #1145), because it's the only non-deprecated way to get security updates automatically as part of operating system updates.

But, because there's currently no sans-IO interface, it can't fit into the SSLStream interface at all, and using it would break standard stuff like mocking out the network during tests. I guess people could use one TLS stack for testing and a totally different one in production, but that seems like a bad idea.

The alternative is to just keep using OpenSSL. It's annoying but totally viable to use OpenSSL + the native system trust store (you pull out the cert chain and pass it to SecEvaluateTrust), so really the only downside of sticking with OpenSSL is that users have to remember to upgrade it regularly, instead of having it automatically upgraded with the OS.

So very tentative conclusion: unless Apple expands their new TLS library's public APIs to allow a more sans-IO style usage, it's probably not worth trying to support it in Trio. And as far as I know they don't currently have any plans to do this.

Implications for Trio's lower-level I/O stack

Using Network.framework as a replacement for the BSD socket API is more compelling. It's doing things you genuinely can't do otherwise, and if we ever want to support iOS then it's probably necessary. (But there are a lot of other things that would need to happen before we could seriously consider supporting iOS. Folks like BeeWare are working on getting Python to work reliably on iOS, but as of July 2019 it's not really a viable platform for us to target.)

This would require two major changes:

Switching to the new event notification approach: Using Grand Central Dispatch (GCD) is sort of awkward, because right now we assume that every OS provides some kind of operation like "check for I/O, while blocking for up to X seconds". But GCD is all about hidden threads that invoke callbacks. At first I was thinking that this would require a significant restructuring of our I/O loop, but on further thought I think the simple way would be:

• Create a Dispatch Queue to use for notifications -- probably a background serial queue
• Create a Python queue.Queue
• Define notification callbacks that will run in the Dispatch Queue, and all they do is push a message to the queue.Queue
• Make our handle_io operation be basically q.get(timeout=...).

I'm not sure how much overhead this would add. I'm guessing it's not too bad? You do have to handoff between threads, but only once per scheduler batch, and the handoff should be pretty cheap.

This would also cause problems for our low-level APIs like wait_kevent: most kqueue features are exposed through GCD, but they're all wrapped up in "higher level" APIs, so you can't just take a kqueue event type and falgs and use them directly, you have to find the corresponding DispatchSource type and set it up etc. So this might need some adjustment. But these low-level APIs are, by definition, low-level exposure of non-portable OS functionality, so if the OS functionality changes then it makes sense for the API to change. I don't think this part would be a major blocker, if switching otherwise makes sense. (There's also the possibility of using a dedicated kqueue thread, sort of like how we currently have a dedicated IOCP thread on Windows.)

Exposing the higher-level Network.framework-based APIs: I'm guessing we can't totally hide this behind trio.socket... though, who knows, maybe. But it would be at about that level in our stack, and that's probably doable somehow. We really need some actual documentation on Network.framework to know what kind of abstraction layer we would need and where it would slot in.

Given the unknowns and the timescale here, I don't think there's anything to do immediately. And so long as we only support desktop/server macOS, not mobile iOS, it might not make sense to switch at all.

[Tronic]

Why is everyone doing background threads and callbacks (GCD, asyncio Protocols -- from C# originally, I suppose)? It is not native on the socket end and is only nice if your use case is to immediately react to any received data (e.g. simple echo protocols). A pretty typical case of having to talk with another connection (query a database?) just doesn't fit the pattern.

In the end you'll probably use queues that can then be used as if they were an async socket/stream, so you have in fact an abstraction in the framework and an equivalent de-abstraction in the application, and a hidden threadpool + two extra layers of buffering to make that work.

[njsmith]

@Tronic If you're stuck with C's stack layout, but need lots of concurrency without bloating memory, then scheduling callbacks across threads is about the best option available. asyncio protocols are inherited from twisted, which was designed to work on the Python of twenty years ago; GCD is written for C and Obj-C, which shares C's ABI and thus its stack layout... There are a lot of downsides for sure, but it's not so easy to shake loose the weight of decades of legacy systems.

[pquentin]

More bad news: I believe we can't use the TLS 1.3 SecureTransport support, see urllib3/urllib3#1674 for details.

[njsmith]

@pquentin Does SecureTransport even have any TLS 1.3 support? The rumor I heard was that Apple doesn't have any plans to support this at all.

[pquentin]

Hmmm. I thought it was supported, given that:

• TLS 1.3 beta support was announced at WWDC 2017 (see 1, 2), while the Network framework was only announced a year later at WWDC 2018, so TLS 1.3 must have been included in SecureTransport at some point
• There's a kTLSProtocol13 constant in SecureTransport

But the documentation at https://developer.apple.com/documentation/security/secure_transport currently states that TLS 1.2 is the last supported version. That surprised me, so I used the Wayback Machine. I found out that Apple documented TLS 1.3 in June 2018, and removed it this summer.

So I guess it's safe to assume that it's not supported. You are well informed!