Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System shut down with HawkesEM and multiprocessing #486

Open
rambor12 opened this issue Apr 1, 2022 · 0 comments
Open

System shut down with HawkesEM and multiprocessing #486

rambor12 opened this issue Apr 1, 2022 · 0 comments

Comments

@rambor12
Copy link

rambor12 commented Apr 1, 2022

Hello,

During the Monte Carlo simulation, I found that the HawkesEM with multiprocessing using apply_async leads to a sudden system shut down.
This is still happening with another parallel processing package ray.
Since I am not an expert in software engineering, I have no idea how to handle this problem.
So I just ask here to get any hint to solve this issue.
Here is the example:

import gc
import numpy as np
from tick.hawkes import SimuHawkesExpKernels, HawkesExpKern, HawkesEM

def run(seed):
    
    # Point generation
    hawkes = SimuHawkesExpKernels([[alpha/beta]], [[beta]], baseline, 
                                  end_time=T, verbose=False, seed=seed, 
                                  period_length=T)
        
    hawkes.simulate()
    points = hawkes.timestamps[0]
    del hawkes

    # Estimation    
    model = HawkesEM(kernel_support=10)
    model.fit([[points]])
    res = model.kernel.sum()
            
    del points, model
    gc.collect()
    print(end='.')
        
    return np.array(res)

N_sim = 6000
alpha = 1.6
beta = 2.0
T = 40000.
baseline = np.repeat(1,T).reshape((1,-1))

from functools import partial
from multiprocessing import Pool, cpu_count

def log_out(i, out):
    pool_out[i] = out

pool_out = {}
pool = Pool(cpu_count()-2)
try:
    for seed in range(N_sim):
        if seed not in pool_out:
            pool.apply_async(run, args=(seed,), 
                             callback=partial(log_out, seed))
    pool.close()
    pool.join()
except KeyboardInterrupt:
    pool.terminate()
pool_out=dict(sorted(pool_out.items()))

The run tine would be several minutes depends on your system.
Here is my system info:

Linux-5.4.0-107-generic-x86_64-with-debian-buster-sid
Python: 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]
Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
tick Version: 0.6.0.0

Thank you for the reading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant