3.3. pyopus.parallel.pypvm — The pypvm PVM library wrapper

The pypvm PVM library wrapper

Written by W. Michael Petullo and Greg Baker.

http://pypvm.sourceforge.net

Functions pkbytestr() and upkbytestr() added by Arpad Buermen.

Currently supported under Linux only.

pyopus.parallel.pypvm.pk(obj)
Packs a Python object obj into the default pvm buffer by picking it and calling the pkbytestr() function.
pyopus.parallel.pypvm.upk()
Unpacks a Python object obj from the default pvm buffer by first calling the upkbytestr() function and then unpickles the Python object which is returned to the caller.
pyopus.parallel.pypvm.addhosts()

pypvm.addhosts(hostlist) adds the computers named in [hostlist] to the configuration of computers making up the virtual machine. The names should have the same syntax as lines of a pvmd hostfile (see man page for pvmd3): A hostname followed by options of the form xx=y.

pypvm.addhosts returns a list; this list will (hopefully) just contain zeros. However, if any of them are negative numbers, they are indicativethat the addhost operation failed on that corresponding machine. The return results are equal to members of of the pypvm.results dictionary.

  • pypvm.results['BadParam'] => bad hostname syntax.
  • pypvm.results['NoHost'] => no such host.
  • pypvm.results['CantStart'] => failed to start pvmd on host.
  • pypvm.results['DupHost'] => host already configured.
  • pypvm.results['BadVersion'] => pvmd protocol versions don’t match.
  • pypvm.results['OutOfRes'] => PVM has run out of system resources.
pyopus.parallel.pypvm.archcode()

The routine pypvm.archcode() returns an integer given an architecture name. The code returned identifies machines with compatible binary data formats. For example, SUN4 and RS6K have the same code, while ALPHA has a different one (because a few datatypes have different sizes). This lets you know when you can get away with using pypvm.data.raw instead of the default encoding to pass messages between tasks on two machines.

Naturally, you shouldn’t assume the values returned by pvm_archcode are etched in stone; the numbers have no intrinsic meaning except that if two different arch names map to the same value then they’re compatible.

pyopus.parallel.pypvm.barrier()

pypvm.barrier(group, count)

The routine pypvm.barrier blocks the calling process until count members of the group have called pypvm.barrier.

pyopus.parallel.pypvm.bcast()

pypvm.bcast(group, msgtag)

Broadcasts the data in the active message buffer to the processes in group

pyopus.parallel.pypvm.bufinfo()

pypvm.bufinfo(bufid) returns a tuple of information about a message buffer - (bytes, msgtag, tid) where

  • bytes is an integer returning the length in bytes of the entire message.
  • msgtag is an integer returning the message label. Useful when the message was received with a wildcard msgtag
  • tid is an integer returning the source of the message. Useful when the message was received with a wildcard tid.
pyopus.parallel.pypvm.catchout()

pypvm.catchout(file) causes the calling task (the parent) to catch output from tasks spawned after the call to pypvm.catchout(). Characters printed on stdout or stderr in children tasks are collected by the pvmds and sent in control messages to the parent task, which tags each line and appends it to the specified file. Output from grandchildren (spawned by children) tasks is also collected, provided the children don’t reset PvmOutputTid.

If file is not specified, or is None, output collection is turned off.

If pypvm.exit() is called while output collection is in effect, it will block until all tasks sending it output have exited, in order to print all their output. To avoid this, turn collection off first.

pyopus.parallel.pypvm.config()

pypvm.config() returns a three element tuple

  • the number of hosts in the PVM cluster
  • the number of architectures in the cluster
  • a list of dictionaries giving information about each machine in the cluster.

Each dictionary has keys

  • dtid
  • hostname
  • arch
  • speed

Note that the speed field may have no particularly relationship with reality, as it just reporting the number stored in the startup hostsfile for the master pvmd. Read the pvmd3 man pages for more information.

The information provided by pypvm.config is also available through pypvm.hostinfo and pypvm.narch.

pyopus.parallel.pypvm.delete()
pypvm.delete(name, index) deletes entry <name, index> from the database ; see pvm_insert(3PVM) for a description of this database.
pyopus.parallel.pypvm.delhosts()

pypvm.delhosts(hostlist) deletes the computers pointed to in hostlist from the existing configuration of computers making up the virtual machine. All PVM processes and the pvmd running on these computers are killed as the computer is deleted.

The returned array can checked to determine which host caused the error, and compared against the elements in the pypvm.results dictionary.

pyopus.parallel.pypvm.exit()
The routine pypvm.exit() tells the local pvmd that this process is leaving PVM. This routine does not kill the process, which can continue to perform tasks just like any other serial process.
pyopus.parallel.pypvm.export()
pypvm.export(name) is provided for convenience in editing the environment variable PVM_EXPORT, while maintaining the colon-separated list syntax it requires.
pyopus.parallel.pypvm.freebuf()
The routine pypvm.freebuf(bufid) frees the memory associated with the message buffer identified by bufid. Message buffers are created by pypvm.mkbuf, pypvm.initsend, and pypvm.recv.
pyopus.parallel.pypvm.freecontext()
pypvm.freecontext(ctx) frees ctx so that it may be reused. Contexts are a system resource that will be exhausted if not recycled.
pyopus.parallel.pypvm.getcontext()
pypvm.getcontext() returns the current context of the requesting task.
pyopus.parallel.pypvm.getinst()
pypvm.getinst(group, tid) returns the instance number in group of a PVM process identified by tid
pyopus.parallel.pypvm.getopt()

pypvm.getopt(what) returns the value of the specified option in PVM. For a discussion of options and values, see the pvm_setopt man page.

The permissible values of what are in the pypvm.opt dictionary:

  • pypvm.opt['Route'] ... 1 ... Message routing policy
  • pypvm.opt['DebugMask'] ... 2 ... Libpvm debug mask
  • pypvm.opt['AutoErr'] ... 3 ... Auto error reporting
  • pypvm.opt['OutputTid'] ... 4 ... Stdout destination for children
  • pypvm.opt['OutputCode'] ... 5 ... Output message tag
  • pypvm.opt['TraceTid'] ... 6 ... Trace data destination for children
  • pypvm.opt['TraceCode'] ... 7 ... Trace message tag
  • pypvm.opt['FragSize'] ... 8 ... Message fragment size
  • pypvm.opt['ResvTids'] ... 9 ... Allow messages to reserved tags and TIDs
  • pypvm.opt['SelfOutputTid'] ... 10 ... Stdout destination
  • pypvm.opt['SelfOutputCode'] ... 11 ... Output message tag
  • pypvm.opt['SelfTraceTid'] ... 12 ... Trace data destination
  • pypvm.opt['SelfTraceCode'] ... 13 ... Trace message tag
  • pypvm.opt['ShowTids'] ... 14 ... pvm_catchout prints task ids with output
  • pypvm.opt['PollType'] ... 15 ... Message wait policy (shared memory)
  • pypvm.opt['PollTime'] ... 16 ... Message spinwait duration
Example:
pypvm.getopt(pypvm.opt[‘FragSize’])
pyopus.parallel.pypvm.getrbuf()
pypvm.getrbuf() returns the message buffer identifier for the active receive buffer.
pyopus.parallel.pypvm.getsbuf()
pypvm.getsbuf() returns the message buffer identifier for the active send buffer.
pyopus.parallel.pypvm.gettid()
pypvm.gettid(group, instance) returns the tid of the process identified by group and instance.
pyopus.parallel.pypvm.gsize()
pypvm.gsize(group) returns the number of members presently in group.
pyopus.parallel.pypvm.halt()
pypvm.halt() shuts down the entire PVM system.
pyopus.parallel.pypvm.hostinfo()

Returns a list (one tuple for each host in the virtual machine) - each tuple is (task id of pvm daemon, hostname, architecture, speed).

This is a subset of the information that can be retrieved with pypvm.config()

pyopus.parallel.pypvm.initsend()
pypvm.initsend(encoding) / pypvm.initsend() clears the default send buffer and specify message encoding to be encoding/default. Valid encoding values are in pypvm.data and can be either pypvm.data['default'], pypvm.data['raw'] or pypvm.data['inplace']. I don’t recommend using inplace with this version of PyPVM.
pyopus.parallel.pypvm.insert()
pypvm.insert(name, index, data) stores data in the simple database maintained by the master pvmd , which can be used to store values such as tids and make them accessible anywhere within a virtual machine. This is useful when building an application such as the group server, which must advertise its task id so clients can register send messages to register. data (an integer) gets stoed as the entry <name, index>.
pyopus.parallel.pypvm.joingroup()
pypvm.joingroup(group) enrolls the calling process in group
pyopus.parallel.pypvm.kill()
pypvm.kill(tid) sends a terminate (SIGTERM) signal to the PVM process tid.
pyopus.parallel.pypvm.lookup()

pypvm.lookup(name, index) retrieves data stored in the location given by <name,*index*>. If index is -1, the data stored at the first existing index in the named class is returned.

See pvm_insert(3PVM) (or pypvm.insert.__doc__) for a description of this database.

pyopus.parallel.pypvm.lvgroup()
pypvm.lvgroup(group) unenrolls the calling process from a named group.
pyopus.parallel.pypvm.mcast()
pypvm.mcast(tids, msgtag) multicasts a message stored in the active send buffer to the tasks specified in the tids list. The message is not sent to the caller even if listed in tids. The content of the message can be distinguished by msgtag.
pyopus.parallel.pypvm.mkbuf()
pypvm.mkbuf(encoding) creates a new message buffer. Valid encoding values are in pypvm.data and can be either default, raw or inplace. I don’t recommend using inplace with this version of PyPVM. The encoding argument is optional.
pyopus.parallel.pypvm.mstat()

pypvm.mstat(host) returns the status of a host in the virtual machine.

value MEANING
PvmOk host is OK
PvmNoHost host is not in virtual machine
PvmHostFail host is unreachable (and thus possibly failed)
pyopus.parallel.pypvm.mytid()
pypvm.mytid() returns the tid of the calling process.
pyopus.parallel.pypvm.narch()
pypvm.narch() returns a count of the number of different data formats being used (narch = number of architectures). This is a subset of the information returned by pypvm.config()
pyopus.parallel.pypvm.newcontext()
pypvm.newcontext() returns a newly allocated context. However, this new context is not yet active. See pypvm.setcontext.
pyopus.parallel.pypvm.notify()

pypvm.notify(what, msgtag, tids, count) requests PVM to notify the caller on detecting certain events. One or more notify messages (see below) are sent by PVM back to the calling task. The messages have tag msgtag supplied to notify.

what is the type of event to trigger the notification. Presently one of:

  • pypvm.notifyDict[‘TaskExit’] => Task exits or is killed
  • pypvm.notifyDict[‘HostDelete’] => Host is deleted or crashes
  • pypvm.notifyDict[‘HostAdd’] => New host is added

msgtag Message tag to be used in notification.

tids For TaskExit and HostDelete, list of task of pvmd TIDs to be notified about. It is not used when what is HostAdd.

count for HostAdd, it determines how many messages will be sent. It is not used for TaskExit or HostDelete.

The notification messages have the following format:

  • pypvm.notifyDict[‘TaskExit’] One notify message for each TID requested. The message body contains a single TID of exited task.
  • pypvm.notifyDict[‘HostDelete’] One notify message for each TID requested. The message body contains a single pvmd-TID of exited pvmd.
  • pypvm.notifyDict[‘HostAdd’] count notify messages are sent, one each time the local pvmd’s host table is updated. The message body contains an integer length followed by a list of pvmd-TIDs of new pvmds. The counter of HostAdd messages yet to be sent is replaced by successive calls to pvm_notify. Specifying a count of -1 turns on PvmHostAdd messages until a future notify; a count of zero disables them. count defaults to 0.

TIDs in the notify messages are packed as integers. The calling task is responsible for receiving messages with the specified tag and taking appropriate action. Future versions of PVM may expand the list of available notification events.

pyopus.parallel.pypvm.nrecv()

pypvm.nrecv(tid, msgtag) is a non-blocking receive.

The routine pypvm.nrecv checks to see if a message with label msgtag has arrived from tid and also clears the current receive buffer if any, If a matching message has arrived pypvm.nrecv immediately places the message in a new active receive buffer, and returns the buffer identifier in bufid.

If the requested message has not arrived, then pypvm.nrecv immediately returns with a 0 in bufid.

A -1 or a missing in msgtag or tid matches anything.

pyopus.parallel.pypvm.pack_by_type()
pypvm.pack_by_type(arg) packs the active message buffer with the contents of arg. arg will probably be a list, although it can be a single value. At the moment, only ints, floats and strings can be packed (they get packed with pvm_pkint, pvm_pkdbl and pvm_pkstr respectively - the programmer has no control over this... future versions will see improvements)
pyopus.parallel.pypvm.parent()
pypvm.parent() returns the tid of the process that spawned the calling process.
pyopus.parallel.pypvm.perror()
pvm_perror(msg) returns the error message of the last PVM call. The user can (optionally) use msg to add additional information to the error message, for example, its location. I can’t imagine this function being used all that often.
pyopus.parallel.pypvm.pkbyte()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pkbytestr()
pypvm.pkbytestr(str) packs the string str. The string can contain null characters, because it gets packed as a sequence of bytes.
pyopus.parallel.pypvm.pkdouble()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pkfloat()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pkint()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pklong()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pkshort()

The family of functions pypvm.pk*(list, stride) take a list and sends them as an array of whatever type is given in the argument (byte, short, int, long, float, or double). stride defaults to 1, and is the separation to use.

One day in the future (not now) these functions will have one-argument forms so that if the argument given is not a list, that it will pack it as if it were a one-element list.

pyopus.parallel.pypvm.pkstr()
pypvm.pkstr(str) packs the string [str]
pyopus.parallel.pypvm.probe()
pypvm.probe(tid, msgtag) checks to see if a message with label msgtag has arrived from tid. If a matching message has arrived pypvm.probe returns a buffer identifier. This bufid can be used in a pypvm.bufinfo call to determine information about the message such as its source and length.
pyopus.parallel.pypvm.pstat()
pypvm.pstat(tid) returns the status of the process identified by tid. Also note that pypvm.notify() can be used to notify the caller that a task has failed.
pyopus.parallel.pypvm.recv()
pypvm.recv(tid, msgtag) blocks the process until a message with label msgtag has arrived from tid. pypvm.recv() then places the message in a new active receive buffer, which also clears the current receive buffer. tid and msgtag can be -1, meaning “match anything”, which is the default if either is not mentioned.
pyopus.parallel.pypvm.send()
pypvm.send(tid, msgtag) sends a message stored in the active send buffer to the PVM process identified by tid. msgtag is used to label the content of the message.
pyopus.parallel.pypvm.sendsig()
pypvm.sendsig(tid, signum) sends the signal number signum to the PVM process identified by tid.
pyopus.parallel.pypvm.setcontext()
pypvm.setcontext(newctx) changes the current context from old_ctx (which is returned to newctx.
pyopus.parallel.pypvm.setopt()

pypvm.setopt(what, value) is a general purpose function used to set miscellaneous options in the PVM library. The previous value is returned.

what is an integer defining what to set. One of:

  • pypvm.opt['Route'] ... 1 ... Message routing policy
  • pypvm.opt['DebugMask'] ... 2 ... Libpvm debug mask
  • pypvm.opt['AutoErr'] ... 3 ... Auto error reporting
  • pypvm.opt['OutputTid'] ... 4 ... Stdout destination for children
  • pypvm.opt['OutputCode'] ... 5 ... Output message tag
  • pypvm.opt['TraceTid'] ... 6 ... Trace data destination for children
  • pypvm.opt['TraceCode'] ... 7 ... Trace message tag
  • pypvm.opt['FragSize'] ... 8 ... Message fragment size
  • pypvm.opt['ResvTids'] ... 9 ... Allow messages to reserved tags and TIDs
  • pypvm.opt['SelfOutputTid'] ... 10 ... Stdout destination
  • pypvm.opt['SelfOutputCode'] ... 11 ... Output message tag
  • pypvm.opt['SelfTraceTid'] ... 12 ... Trace data destination
  • pypvm.opt['SelfTraceCode'] ... 13 ... Trace message tag
  • pypvm.opt['ShowTids'] ... 14 ... pvm_catchout prints task ids with output
  • pypvm.opt['PollType'] ... 15 ... Message wait policy (shared memory)
  • pypvm.opt['PollTime'] ... 16 ... Message spinwait duration

val Integer specifying new setting of option. Usually this is just any old number, but there are some predefined values for pypvm.opt['Route']:

  • pypvm.opt['DontRoute'] ... 1 ... Don’t request or grant connections
  • pypvm.opt['AllowDirect'] ... 2 ... (Default) Don’t request but allow
  • pypvm.opt['RouteDirect'] ... 3 ... Request and allow connections
pyopus.parallel.pypvm.setrbuf()
pypvm.setrbuf(bufid) switches the active receive buffer to bufid and returns the previous buffer id.
pyopus.parallel.pypvm.setsbuf()
pypvm.setsbuf(bufid) switches the active send buffer to bufid and returns the previous buffer id.
pyopus.parallel.pypvm.spawn()
pypvm.spawn(task, argv, flag, where, ntasks) starts ntask copies of the executable named task. The tids are returned in a list.
pyopus.parallel.pypvm.tasks()

Returns a list (one tuple for each task) of all tasks running in the virtual machine. Each tuple is (task id, parent task id, task id of the pvm daemon on the tasks’ machine, flags it was called with, the name of the task, and the process id of the task).

If called with a tid as an argument, it will just return that tasks info; if called with the tid of a pvm daemon, it will return all the tasks on that host.

pyopus.parallel.pypvm.tidtohost()
pypvm.tidtohost(tid) returns the tid of the daemon process on the host running the process identified by tid is located.
pyopus.parallel.pypvm.trecv()

pypvm.trecv(timeout, msgtag, tid) blocks the process until a message with label msgtag has arrived from tid. pypvm.trecv then places the message in a new active receive buffer, also clearing the current receive buffer. If no matching message arrives within the specified waiting time timeout (in seconds), pypvm.trecv returns without a message.

As usual tid and msgtag can be -1, which is the default if either are missing.

pypvm.trecv returns the bufid of the new active receive buffer. Note that the order of arguments here is different to the pvm_trecv call, which puts the arguments in the opposite order.

pyopus.parallel.pypvm.unexport()
pypvm.unexport(name) is provided for convenience in editing the environment variable PVM_EXPORT, while maintaining the colon-separated list syntax it requires.
pyopus.parallel.pypvm.upkbyte()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upkbytestr()
pypvm.upkbytestr() unpacks a byte string from the active receive buffer NULL characters are allowed in the string.
pyopus.parallel.pypvm.upkdouble()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upkfloat()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upkint()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upklong()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upkshort()
Each of the pypvm.upk*(nitem, stride) routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the routines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and the (optional) stride which is the stride to use when unpacking.
pyopus.parallel.pypvm.upkstr()
pypvm.upkstr() unpacks a string from the active receive buffer
pyopus.parallel.pypvm.version()
Returns the version string of the PVM this was built against.

Previous topic

3.2. pyopus.parallel.pvm — A virtual machine based on the PVM library

Next topic

3.4. pyopus.parallel.evtdrvms — Master-slave event-driven algorithm model

This Page