Error when using twisted and greenlets

I'm trying to use twisted with greenlets, so I can write synchronous looking code in twisted without using inlineCallbacks.

Here is my code:

import time, functools
from twisted.internet import reactor, threads
from twisted.internet.defer import Deferred
from functools import wraps
import greenlet

def make_async(func):
    def wrapper(*pos, **kwds):
        d = Deferred()

        def greenlet_func():
                rc = func(*pos, **kwds)
            except Exception, ex:
                print ex

        g = greenlet.greenlet(greenlet_func)

        return d
    return wrapper

def sleep(t):
    print "sleep(): greenelet:", greenlet.getcurrent()
    g = greenlet.getcurrent()
    reactor.callLater(t, g.switch)

def wait_one(d):
    print "wait_one(): greenelet:", greenlet.getcurrent()
    g = greenlet.getcurrent()
    active = True

    def callback(result):
        if not active:
            reactor.callLater(0, g.switch, result)

    def errback(failure):
        if not active:
            reactor.callLater(0, g.throw, failure)


    active = False
    rc = g.parent.switch()
    return rc

def inner():
    print "inner(): greenelet:", greenlet.getcurrent()

    import random, time
    interval = random.random()

    print "Sleeping for %s seconds..." % interval
    print "done"

    return interval

def outer():
    print "outer(): greenelet:", greenlet.getcurrent()
    print wait_one(inner())
    print "Here"

reactor.callLater(0, outer)

There are 5 main parts:

  • A sleep function, that starts a timer, then switches back to the parent greenlet. When the timer goes off, it switches back to the greenlet that is sleeping.
  • A make_async decorator. This takes some synchronous looking code and runs it in a greenlet. IT also returns a deferred so the caller can register callbacks when the code completes.
  • A wait_one function, which blocks the greenlet until the deferred being waited on resolves.
  • The inner function, which (when wrapped) returns a deferred, sleeps for a random time, and then passes the time it slept for to the deferred.
  • The outer function, which calls inner(), waits for it to return, then prints the return value.

When I run this code I get this output (Note the error on the last two lines):

outer(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
inner(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
Sleeping for 0.545666723422 seconds...
sleep(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
wait_one(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
Exception twisted.python.failure.Failure: <twisted.python.failure.Failure <class 'greenlet.GreenletExit'>> in <greenlet.greenlet object at 0xb729ce3c> ignored
GreenletExit did not kill <greenlet.greenlet object at 0xb729ce3c>

Doing a bit of research I've found that:

  • The last line is logged by greenlet.c
  • The previous line is logged by python itself, as it's ignoring an exception raised in a del method.

I'm having real trouble debugging this as I can't access the GreenletExit or twisted.python.failure.Failure exceptions to get their stack traces.

Does anyone have any ideas what I'm doing wrong, or how I get debug the exceptions that are being thrown?

One other data point: If I hack wait_one() to just return immediately (and not to register anything on the deferred it is passed), the errors go away. :-/


Rewrite your error callback in wait_one like this:

  def errback(failure):
    ## new code
    if g.dead:
    if not active:
        reactor.callLater(0, g.throw, failure)

If greenlet is dead (finished running), there is no point throwing exceptions in it.

mguijarr's answer fixed the problem, but I wanted to write up how I got into this situation.

I have three greenlets:

  • {main} that's runing the reactor.
  • {outer} that's running outer().
  • {inner} that's rrunning inner().

When the sleep finishes the {main} switches to {inner} which switches to {outer}. Outer then returns and raises GreenletExit in {inner}. This propogates back to twisted. It sees an exception being raised from callback(), and so invokes errback(). This tries to throw the exception into {outer} (which has already exited), and I hit the error.


 ? Can I set a custom name displayed for a gevent greenlet while logging?
 ? Greenlet version is too old error even the 0.4.5 is installed. Why?
 ? Does thread-safe generally imply greenlet-safe in Python?
 ? Default values for boto3 method parameters
 ? Default values for boto3 method parameters
 ? Default values for boto3 method parameters
 ? Boto3: 's3.Object' object has no attribute 'md5'
 ? Boto3: 's3.Object' object has no attribute 'md5'
 ? Boto3: 's3.Object' object has no attribute 'md5'
 ? boto3 cannot create client on pyspark worker?