Lately I've been doing some Python multi-threading to make the best use of some of our amazing server resources. As I was pondering the reasons why one of our 8-core servers reported 83% idle despite 8 threads banging away, I re-discovered the Global Interpreter Lock.
BLECH!
The GIL enforces Python's requirement that only a single bytecode operation is executed at a time. My nicely coded multi-threaded app was only being executed serially!! Sadly, this seems unlikely to change, even in Python 3000. Last year Guido said:
"Just Say No to the combined evils of locking, deadlocks, lock granularity, livelocks, nondeterminism and race conditions."
I was brought up to believe that threading was dirty and independent communicating processes were the way to go. But even I realize that this just isn't practical in these days of GUIs, multi-core processors, and application servers.
Why does the Python community accept the GIL? Is it because most people only use Python as a scripting language? Are there simple workarounds (e.g. not forking, shared memory, or the like) that I'm missing?
It's definitely not because
It's definitely not because everyone's using python just for quick scripts, or whatever.
I spent a bit of time poking around the GIL issue a few jobs ago, on a Zope system, and came to the conclusion that there are enough alternate approaches for most problems, that the advantages of Python generally make it worth while. Having said that, we were dependent on the ZODB, Zope's object database, to get multiple things done at once, and it really couldn't manage.
a pity
I think we accept the GIL because we love python so much that by the time we encounter it, we can't give it up. I discovered the dreaded GIL during a class project with genetic algorithms. I had four processors to work with and was hoping for a 4x speed-up, by evaluating four "genes" at a time. It was depressing.
There is Stackless...
There is/was something called Stackless Python which did, I believe, do away with the GIL (it also introduced green threads and continuations, though not necessarily all at the same time).
I'm not sure why they accept the GIL. It could be that they've found that any cases they have where it's really a problem can be dealt with using multiple processes, and ditching it would add significant complexity to the runtime (concurrent garbage collection and having a lock on every object to protect its reference count or a global refcount lock to use when adjusting reference counts are two of the things that come to mind).
But yeah, the present state... I don't know. It does seem that they're somewhat crippling themselves given that increasing parallelism is the direction of increased performance for the forseeable future.
I don't believe that
I don't believe that Stackless did away with the GIL. If it had, we would all be using Stackless.
As for easy concurrency, look at the processing package. It offers an interface similar to the Threading module in Python, only it uses processes. There are queues and other shared data structures that allow the each process to easily pass data around (including sockets on Windows and Linux).
Regarding why Python programmers "accept" the GIL, it's not about "acceptance", it's about "it's really hard to get rid of a single coarse-grained lock and replace it with a fine-grained lock". And doing so doesn't necessarily guarantee an increase in performance.
Although not a definitive
Although not a definitive answer by any means, Stackless certainly seems to keep all my cores working hard when it needs to.