Developing with vesper » History » Revision 2
Revision 1 (jun chen, 03/09/2025 10:59 PM) → Revision 2/7 (jun chen, 03/10/2025 01:01 AM)
# Developing with vesper ## Vepser introduction 1. What is the Developing with vesper Vepser is a python/c++ based platform for providing parallel computing service, which include a distributed file system(DFS), a dyanmic job scheduler and a dynamic resource scheduler. 2. When to use vesper vesper is recommand to used if there are some tasks could be split to run in parallel, or processing mass data computing. 3. How to use vesper Any python script could be run with vesper after configure the python site-packages General way: Vesper xxx.py Interactive mode: vesper -i xxx.py 4. Scheduler/Worker Introduction Vesper will launch one scheduler and several workers Scheduler: used to parse input script, schedule job and worker, so DONOT do heavy task on the scheduler!!!! worker: used to execute jobs only, could be launch in local machine and remote machines 5. How to launch workers  wait—for—workers is not required, but it is recommand if you want to all liobs start at the same ti me. 6. Job introduction Job system is a DAG seems like  a job could be no any dependency or depend on some other jobs, it will be dispatched to workers immidiately if there is no dependecy else it will be dispatched after all dependency were ready. the dependency is always a file list, a name node service used to guarantee the dependency system works well a job is a python function could be a job if which was decorated by as_job. Any time when you want to run a task in a worker ,you should defined it as a job Anywhere when you want to run a task which depend on another job, you should defined it as a job job has no any return values.  7. MapReduce Introduction Mapreduce is a easy way to do some work in parallel(based on job system), it is recommend for simple work such as numerical computation. Mapreduce always work on a collection object such as listidictituple.   8.UserView introduction UserView is a dict like object(key should be string) could be shared bit different workers, which seems like on scheduler: uv = UserView("tmp") uv["key1"] = 999 on worker1: uv["key2"] = 9999 on worker2 visit: key1 = uv["key1"] key2 = uv["key2"] ## High level usage ## Low level usage