Wednesday, 10 June 2009

Parallel computing with Python

I've been writing a library that streamlines the process of parallelization. It is inspired by Cilk and thus has a really simple design. The name isn't yet decided, but I think I will call it pylk. I'm also trying to write a cluster-library, and once that is finished I will try to add a backend that executes the functions that were spawned on the cluster.

Below is some (commented) example code.
# Use multiprocessing backend.
backend = MPBackend()
# Start the thread that listens for I/O from the processes.
backend.iohandler.start()

# Do some calculations.
a = backend.spawn(reduce, operator.mul, xrange(1, 50000))
b = backend.spawn(reduce, operator.mul, xrange(1, 50000))

# When you need the results, sync them.
a, b = sync(a, b)
# When done, shut down the I/O thread so the application can exit.
backend.iohandler.stop()

No comments: