1: What, Why, How Coroutines?

As noted in the previous post coroutines are a way of having a function “return and continue later”. This is briefer in python than C++ so we’ll start there. Note in the comments below I assume python 3 syntax, even though they date all the way back to the early 2.2 series.

Python’s simplest coroutines - PEP 255 Simple Generators

(For those that worry about such things – I’m not advovating for or against this style of generator/coroutine, just picking it because it’s the simplest option, and is still useful)

In python the simplest type looks like this:

def simplest():
    print("Hello")
    yield
    print("Hello Again")
    yield
    print("Bye then")

You can run this as follows:

>>> s = simplest()
>>> s.__next__()
Hello
>>> s.__next__()
Hello Again
>>> s.__next__()
Bye then
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Note:

• s = simplest() - creates a thing that can be run
• s.__next__() - gives it a bit of CPU time: emiting Hello
• s.__next__() - gives it more CPU time - note it picks up where it left off, emiting Hello Again
• s.__next__() - gives it more CPU time, emiting Bye then. However, this time it drops off the end of the bottom and raises a StopIteration exception.

This form of coroutine was added to python in python 2.2 in July 2001 to enable Iterators to be simplified, and is specifically called a simple generator there. Python’s moved on somewhat and this opens the door to all sorts of things. (indeed, this is the sort of coroutine that Kamaelia used). Of note in particular is the use of exceptions to signal the end of the generator/coroutine, rather than an error code/sentinel value.

Simple Generators in Python

The example I usually pick is the fibonnaci sequence. This is a simple mathematical sequence where the next value is the sum of the previous 2. As a result it goes:

1 1 2 3 5 8 13 21 34 ... etc

It’s notable in that it was originally to do with bunny populations and is Walter Bishop’s (Fringe) favourite sequence. It turns up a lot in nature.

If you want to create a function to print it ad-infinitum, the python for this is very simple:

def fibonnaci():
    a, b = 1,1
    while True:
        print("Fib:", a)
        a,b = b, a+b

Suppose you wanted to use this to calculate an approximation to the golden ratio. You’d need to write it again:

def golden():
    a, b = 1,1
    while True:
        a,b = b, a+b
        print("Golden Ratio approximation", a/b)
        time.sleep(0.1)

This is a little contrived, but the point here is about repetition. Suppose however, we change our original example like this:

def fibonnaci():
    a, b = 1,1
    while True:
        yield a     # NOTE: instead of print("Fib:", a)
        a,b = b, a+b

Now, instead of printing that to the console, it creates a generator which yields values. This can be used like this:

fibs = fibonnaci()
while True:
    print("Fib:", next(fibs))

And can also be used like this:

fibs = fibonnaci()
last = next(fibs)
curr = next(fibs)
while True:
    print("Golden Ratio approximation", curr/last)
    last, curr = curr, next(fibs)

NB. if our generator looked like this:

def fibonnaci(count):
    a, b, n = 1, 1, 0
    while n < count:  # NOTE: Terminates
        n += 1
        yield a
        a,b = b, a+b

Then we could use it like this:

for i in fibonnaci(22):
    print("FIB: ", i)

… which makes the idea of these being used as iterators far more obvious.

You can also send values into a generator. This is useful in many circumstances since it allows you to implement things like producer consumer models relatively easily.

You can also obviously do things like run lots of these concurrently, which is useful if you use them to manage things like network sockets and have many more clients than you support using threads efficiently.

An example fragment of what that could look like:

# NB, pseudo-code
def socketHandler(sock):
    while True:
        msg, data = yield Ready()
        if msg == "send":
            bytes.append(data)
            nsent = sock.send(bytes)
            bytes = bytes[nsent:]
            yield BytesSent(nsent)

        if msg == "sockready":
            recv = sock.recv()
            yield BytesRecieved(bytes)

Where’s the code?

Update: 11/4/2023

Want to read and runcode, not the blog? I know the feeling. I recently created a github repo specifically for putting code from blog posts in. Though recognising that it might also have non-code stuff added at some point, I’ve called it blog-extras.

The blog-extras repo can be found here:

• https://github.com/sparkslabs/blog-extras

The code specific to this series of short posts can be found here:

• https://github.com/sparkslabs/blog-extras/tree/main/by-date/2023/Coroutines/0

That repo will get fleshed out with code/examples from other posts in this blog too.

Discussion

Of note - this means the thing that looks like a single function is the generator, not things it calls. There is a special syntax (yield from) available in python to allow yielding from a function call, but it’s explicit not implicit. This means for example you can’t “wrap” the yield. Because no stack is maintained between yield calls - just the single frame for the generator which is stored on the heap - these get called stackless coroutines.

By contrast, some languages do allow you to yield from something you’ve called - to “wrap” the yield so to speak. These maintain a callstack and get called “stackful coroutines”, but I think they’re better called lightweight threads in those languages.

On the surface of things, it can look difficult to see how this can scale up, but with thought it can scale up well. It should be clear from the socketHandler example what sort of form that takes. Also, while I do think the “only communicate through the yield command” is somewhat limiting, it does work, and is definitely better than manual state machines.

NEXT POST: This sort of simple generator / minimal coroutine has previously proven useful in a number of real world scenarios, so the question for this series is “how do we implement this in C++?”. The next post ponders “what can coroutines look like in C++?”