Core_EDA

# Developing with vesper

{{toc}}

## Vepser introduction 

### 1. What is the vesper

Vepser is a python/c++ based platform for providing parallel computing service, 

which include a distributed file system(DFS), 

a dyanmic job scheduler and a dynamic resource scheduler.

### 2. When to use vesper

vesper is recommand to used if there are some tasks could be split to run in parallel, or processing mass data computing.

### 3. How to use vesper

Any python script could be run with vesper after configure the python site-packages General way: Vesper xxx.py

Interactive mode: vesper -i xxx.py

### 4. Scheduler/Worker Introduction

Vesper will launch one scheduler and several workers

Scheduler: used to parse input script, schedule job and worker, so DONOT do heavy task on the scheduler!!!! worker: used to execute jobs only, could be launch in local machine and remote machines

### 5. How to launch workers

![](clipboard-202503100058-muhgk.png)

wait—for—workers is not required, but it is recommand if you want to all liobs start at the same ti me.

### 6. Job introduction

Job system is a DAG seems like

![](clipboard-202503100058-ggac0.png)

a job could be no any dependency or depend on some other jobs, it will be dispatched to workers immidiately if there is no dependecy else it will be dispatched after all dependency were ready.

the dependency is always a file list, a name node service used to guarantee the dependency system works well a job is a python function could be a job if which was decorated by as_job.

Any time when you want to run a task in a worker ,you should defined it as a job

Anywhere when you want to run a task which depend on another job, you should defined it as a job job has no any return values.

![](clipboard-202503100058-g5jyn.png)

### 7. MapReduce Introduction

Mapreduce is a easy way to do some work in parallel(based on job system), it is recommend for simple work such as numerical computation.

Mapreduce always work on a collection object such as listidictituple.

![](clipboard-202503100058-idmq4.png)

![](clipboard-202503100058-qpcss.png)

### 8.UserView introduction

UserView is a dict like object(key should be string) could be shared bit different workers, which seems like 

on scheduler:

```

uv = UserView("tmp") 

uv["key1"] = 999

```

on worker1: 

```

uv["key2"] = 9999 

```

on worker2 visit: 

```

key1 = uv["key1"] 

key2 = uv["key2"]

```

## High level usage

### Python level developing

In high level, only use python code to do parallel computation. 

use MapReduce:

![](clipboard-202503100107-v8oow.png)

use Job:

![](clipboard-202503100108-9ovps.png)

## Low level usage