Developing with vesper¶

Vepser introduction¶

1. What is the vesper

Vepser is a python/c++ based platform for providing parallel computing service,
which include a distributed file system(DFS),
a dyanmic job scheduler and a dynamic resource scheduler.

2. When to use vesper
   vesper is recommand to used if there are some tasks could be split to run in parallel, or processing mass data computing.

3. How to use vesper
   Any python script could be run with vesper after configure the python site-packages General way: Vesper xxx.py
   Interactive mode: vesper -i xxx.py

4. Scheduler/Worker Introduction
   Vesper will launch one scheduler and several workers
   Scheduler: used to parse input script, schedule job and worker, so DONOT do heavy task on the scheduler!!!! worker: used to execute jobs only, could be launch in local machine and remote machines

5. How to launch workers
   

wait—for—workers is not required, but it is recommand if you want to all liobs start at the same ti me.

6. Job introduction
   Job system is a DAG seems like
   
   a job could be no any dependency or depend on some other jobs, it will be dispatched to workers immidiately if there is no dependecy else it will be dispatched after all dependency were ready.
   the dependency is always a file list, a name node service used to guarantee the dependency system works well a job is a python function could be a job if which was decorated by as_job.
   Any time when you want to run a task in a worker ,you should defined it as a job
   Anywhere when you want to run a task which depend on another job, you should defined it as a job job has no any return values.
   
7. MapReduce Introduction
   Mapreduce is a easy way to do some work in parallel(based on job system), it is recommend for simple work such as numerical computation.
   Mapreduce always work on a collection object such as listidictituple.
   

8.UserView introduction
UserView is a dict like object(key should be string) could be shared bit different workers, which seems like on scheduler:
uv = UserView("tmp") uv["key1"] = 999
on worker1: uv["key2"] = 9999 on worker2 visit: key1 = uv["key1"] key2 = uv["key2"]

High level usage¶

Low level usage¶