I'm a big fan of asyncio, couroutines and
await syntax in
Python 3.5+. However, they come with some well documented downsides. Not least
among these is the
red/blue function problem.
Coroutines can call plain functions, but only a coroutine can await another
coroutine (unless you count running it by directly invoking the event loop).
Meanwhile, if a coroutine calls a plain function which triggers some sort of
blocking IO, the whole event loop is blocked.
I have found the red/blue divide particularly annoying in the context of web API clients. Take boto3, the popular client for AWS's various APIs. This only works in a synchronous context (unless you don't mind blocking the event loop for the duration of each request). Another project, aiobotocore, provides some of these capabilities in an async context, but doesn't support all services and operations.
Yet, many of the things API clients do have nothing directly to do with IO, async or otherwise. Preparing a HTTP request - determining what headers to use, constructing a request body, calculating signatures and other authentication details - is a purely functional, IO-free business. The same goes for interpreting a HTTP response.
A function or method in an API client is like a sandwich where constructing the request and interpreting the response are the two pieces of bread, and sending the request is the filling. We ought to be able to use the same bread for any type of sandwich, without caring if the filling is ham, cheese, or something else.
Ok, enough with the shaky metaphors. Time for some code.
The strategy I have stumbled upon is to write the pure request-construction / response-interpretation logic as a generator function, which yields in the middle. The generator yields an object providing details of the HTTP request it wishes to be sent, ceding control to "something else, I care not what" which executes the actual web request, and sends an object representing the response back into the generator. The generator then processes the response.
Here's a toy API client for The Internet Chuck Norris Database.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
I still haven't decided if this is a good strategy overall, although it certainly seems to achieve its immediate goal of allowing a large chunk of code to be shared between sync and async clients. Comments welcome.