Parallel estimation MCMC diagnostics issue

Estimation ML/Bayesian estimation
1

Dear all,

I am trying to estimate a Smets-Wouters model using the parallel capabilities with Dynare 4.6.1 and Matlab 2019a in a Max.

Half of the times I estimate the model there is no issue at all and the estimation runs successfully.
In the other half, estimating the model with the exact same specifications and options as in the successful cases, when MCMC finishes and Dynare is about to calculate the convergence diagnostics, it stops working. More specifically when I get the following message

Estimation::mcmc::diagnostics: Univariate convergence diagnostic, Brooks and Gelman (1998):
ssh -p  ‘/bin/bash --norc -c “rm -f McMCDiagnostics_core_output_.mat"': Signal 127
ssh -p  '/bin/bash --norc -c "rm -f comp_status_McMCDiagnostics_core.mat”’: Signal 127
ssh -p  ‘/bin/bash --norc -c “rm -f slaveParallel_break.mat”’: Signal 127

and the bars for the diagnostics calculation, the programme stops and produces no results remaining in the “initialising” phase.
As I said this happens almost half of the times, the rest the estimation finishes successfully.
Usually when the estimation doesn’t go through Matlab gives the following message:

Warning: Unexpected end-of-file while reading compressed data.
In masterParallel (line 680)
In posterior_sampler (line 150)
In dynare_estimation_1 (line 465)
In dynare_estimation (line 105)
In NK_SW.driver (line 649)
In dynare (line 293)

In case it helps the parallel options are

options_.parallel = struct(‘Local’, 1, ‘ComputerName’,‘’, ‘CPUnbr’, [1:8], ‘UserName’,‘’,‘Password’,‘’, ‘RemoteDrive’, ‘’, ‘RemoteDirectory’,‘’, ‘MatlabOctavePath’, ‘/Applications/MATLAB_R2019a.app/bin/matlab’, ‘DynarePath’,…
‘/Applications/Dynare/4.6.1/matlab’, ‘NumberOfThreadsPerJob’,1,‘Port’,22, ‘NodeWeight’,‘1’,‘SingleCompThread’,‘false’,‘OperatingSystem’,‘unix’);

Could you please help with that?

Best,
Stelios

2

The warning

indicates that the file you are trying to load is corrupted. Because it only triggers a warning, not an error, the estimation freezes. Maybe @rattoma has more insights.

3

Thank you Johannes. The estimation freezes even without this warning some other times. When the panel with Parallel MCMCdiagnostics_core comes up, it freezes.

4

An update on this. I have found out that the only way this goes through the MCMC diagnostics is with one node and CPUnbr = 2, although my mac has 4 cores (CPUnbr = 8).

5

Do you have the same issue with other models? If not, could you provide us with the mod file and configuration file you use? Also, what is your setup regarding ssh keys, i.e. do you have any files in

/Users/you/.ssh/

On our university cluster I had a weird issue where I had to put my public key (which is in a file called id_rsa.pub) into authorized_keys… However, that was on a Linux machine…

6

Hi,

Yes I do. It’s the first time I use the parallel estimation but checked and indeed it happens to my other models as well. I am starting to believe that it’s a pure ssh problem and I would appreciate some help.

In Users/you/.ssh/
there are these files:
authorized_keys config id_rsa id_rsa.pub known_hosts

7

Try to add the content of your id_rsa.pub file to authorized_keys as a new line.

8

Thanks. I did that and unfortunately nothing changes. The machine stops doing anything after the Parallel posterior_sample_core window appears and sticks to ‘‘initialising’’. Tried with previous Dynare versions, still nothing. Just to remind you that it’s a local parallelisation.

In the command window I also see this:

ssh  @localhost rm -f //posterior_sampler_core_output_*.mat: Signal 127
ssh  @localhost rm -f //comp_status_posterior_sampler_core*.mat: Signal 127
ssh  @localhost rm -f //slaveParallel_break.mat: Signal 127
9

Very sorry, but this is hard to debug without being on the machine. Have you another machine available to you, to see whether you can get it up and running there?

10

Thanks. Unfortunately not at the moment. But I will and will check this.