Package park :: Package optim :: Module pmap

Module pmap

Parallel map-reduce implementation using threads.

Classes
	Collector Abstract interface to map-reduce accumulator function.
	Mapper Abstract interface to map-reduce mapper function.

Functions

pmap(mapper, inputs)
Apply function mapper to all inputs.

source code

preduce(collector, outputs)
Collect all outputs, calling collector(item) for each item in the sequence.

source code

pmapreduce(mapper, collector, inputs)
Apply function mapper to inputs, accumulating the results in collector.

source code

main()

source code

Variables
	profile_mapper = `False` True if the mapper cost should be profiled.
	_ = `'\n# The choice of job to do next is complicated.\n# 1. St...`

Function Details

pmap(mapper, inputs)

source code

Apply function mapper to all inputs.

This is the serial version of a parallel iterator, yielding the next sequence value as soon as it is available. There is no guarantee that the order of the inputs will be preserved in the parallel version, so don't depend on it!

pmapreduce(mapper, collector, inputs)

source code

Apply function mapper to inputs, accumulating the results in collector.

Collector is a function which accepts the result of mapper(item) for each item of inputs. There is no guarantee that the outputs will be received in order.

The map is executed in a separate thread so the function returns to the caller immediately.

Variables Details

_

Value:

'''
# The choice of job to do next is complicated.
# 1. Strongly prefer a job of the same type as is already running
# on the node.  If this job requires significant resources (e.g.,
# a large file transfer) increase that preference.
# 2. Strongly prefer sending a user\'s own job to their own machine.
# That way at least they can make progress even if the cluster is busy
.
...