Module server

source code

Define the service architecture.

The PARK fitting service manages groups of jobs from a number of users at the same time. The jobs can be long running, with clients able to connect and disconnect from multiple locations.

The goal is not a simple batch queueing system. Traditional queuing systems allocate a group of processors to a particular job for a fixed period of time. Instead we want dynamic resource allocation, where individual work units are meted out according to the criteria such as the relative priority of the jobs and the length of time they are in the queue. Long running low priority jobs such as those which are performing an exhaustive search on the parameter space should get some resources even when higher priority jobs are still running. If nothing else, individual users may contribute their desktop resources to the pool at least for the purposes of completing their own jobs quicker.

We still need to be able to tie into legacy systems with batch queues such as torque. This is supported by starting a private parallel kernel system on the allocated nodes which runs for the duration of the job.

The top level of the queuing system is the job manager (park.job.JobManager). This service keeps a list jobs in a persistent store, and manages the interaction with the remote client. Individual jobs (park.job.Job) have properties such as owner, priority and completion status.

In general jobs do not have a user id. Instead they are tagged with information such as the users name, the service, the time the job was started, service specific tags, and arbitrary strings assigned by the user such as job name. Some servers may require user authentication for example to track the user through the teragrid requests, but applications should not rely on this.

Notification is handled through a message service. Users can also request notification of job completion by email (daily digest by default) by adding email details to the job submission request.

All interaction between the client, the job manager and the individual jobs is handled through a service manager (park.service.ServiceManager). This can be running locally for single system fits or remotely.

Resource allocation is based on credits toward job priority. Each unit of work for higher priority jobs counts more toward usage, so an individual user can choose between getting a small amount of work done quickly or a larger amount of work done when the system has more slack. The default priority is free.

Need a message stream for the job queue. That way as jobs are created, destroyed, or otherwise change status the application can reflect these changes in the GUI. The stream header will be the entire list of active jobs. We will provide the client side data structure to keep a consistent view of the list, but the GUI client will still need notification that changes are occurring. The same technique can be used to maintain an AJAX view of the job table.

PARKTAG - the normal user tag
PARKMAIL - the user portion of the notification address
LOGNAME - the Unix user name, unless it is 'root'
USERNAME - the Windows user name unless it is 'Administrator'