I’m trying to run dynare 4.4.3 in parallel with the MCMC routine on Mac OS X 10.9.5 Mavericks with Matlab R2014a Student Version. The parallel_test call completes fine, but when trying to run the code with parallel call locally it gets stuck at
ssh @localhost rm -f //random_walk_metropolis_hastings_core_output_*.mat: Signal 127
ssh @localhost rm -f //comp_status_random_walk_metropolis_hastings_core*.mat: Signal 127
ssh @localhost rm -f //slaveParallel_break.mat: Signal 127
The MCMC window with status bars is stuck saying initialize with 4 bars corresponding to 4 cores on my machine.
I also had to make a change to the AnalyseComputationalEnvironment.m file at line 548:
It used to say
[si0 de0]=system('grep processor /proc/cpuinfo');
I had to change it to
[si0 de0]=system('sysctl -a | grep machdep.cpu | grep core_count');
To get it to work
Any idea why it gets stuck at this spot? Help would be appreciated.
I attached the .mod file I’m working with and the config file. I tried attaching the data but I see I can’t seem to upload it. Hopefully you can find some data to make it work for you if you need to run the code
test_small_base2.mod (11.9 KB)
Model.txt (7.5 KB)
AnalyseComputationalEnvironment.m (22.8 KB)
dynare_parallel.txt (89 Bytes)
Do you really mean Dynare 4.3.3? The current version is 4.4.3…
You can post the data here as text: gist.github.com/
Also, what version of Matlab and OS X do you have?
I’m using OS X 10.9.5 (Mavericks), Dynare 4.4.3, Matlab R2014a Student Version.
I added the data as tab delimited .txt file to the main post. It will need to be converted to a .mat file to be used in the .mod file.
I’m wondering if I don’t have ssh keys installed properly? What is the correct procedure to do this for dynare?
Please simply just rename the .mat file as a .txt file and upload it again.
Please also upload DYNARE_estimdata
As long as you can type “ssh localhost” into Terminal (you’re running this in parallel on your computer, right?) and it works, everything should be fine.
I uploaded an updated version of the file. It will read in ‘Model.txt’ and use the data in it directly. No need to change anything to get it to read the file anymore. I also removed the step that used Dynare_estimdata. Hope this helps.
Yes, thanks. Now it fails because of a ref to non-existent field ‘data_I’.
Please fix and run on your end to make sure you run into the parallel problem then post the updated file.
Sorry about that, must have ran it before I saved it. It runs on my end now.
I confirm the problems above.
We will work on this and get back to you.
Has there been any updates on this issue?
Is there some solution for this problem? I am experiencing the same issue:
ssh @localhost rm -f //random_walk_metropolis_hastings_core_output_.mat: Signal 127
ssh @localhost rm -f //comp_status_random_walk_metropolis_hastings_core.mat: Signal 127
ssh @localhost rm -f //slaveParallel_break.mat: Signal 127
Any help would be appreciated!
Unfortunately, not yet. You can keep track at github.com/DynareTeam/dynare/issues/838. If you have any insights into what is going wrong, they are highly appreciated. As I am not a Mac user, it is hard for me to help.
Hi, are you using a snapshot? Any of the snapshots here should contain the update: dynare.org/snapshot/macosx/
Please report back, whether or not this works.
I am not using a snapshot. I wish I could, but under Mac OS Sierra there are some MEX incompatibility issues between Matlab R2017a and the pre-compiled versions of Dynare (including the snapshots). So, based on another thread (i.e. Saving the estimated shocks, where you have also posted) I had to use Homebrew to compile Dynare, which works nicely following the hints in the aforementioned thread.
However, I only managed to compile the stable version, but not the HEAD version. Below I give the output of the installation command
"brew install dynare --HEAD --with-matlab=/Applications/MATLAB_R2017a.app --with-matlab-version=9.2 --without-octave --with-tex"
Compilation of the stable version (so without the --HEAD option) works fine, but unfortunately not the HEAD version. I guess the fix for parallel Dynare under Mac OS is only implemented in the HEAD version and not in the stable one (at least parallel Dynare doesn’t work under the stable version on my laptop with Mac OS Sierra, Matlab R2017a).
Is there some other solution? For example, is it possible to replace a couple of Dynare files by newer versions from the GitHub repo?
Any advice would be greatly appreciated!!
Output from HEAD install:
==> Installing dynare from homebrew/science
==> Installing dependencies for homebrew/science/dynare: boost
==> Installing homebrew/science/dynare dependency: boost
==> Downloading homebrew.bintray.com/bottles/bo … tle.tar.gz
==> Pouring boost-1.64.0.sierra.bottle.tar.gz
/usr/local/Cellar/boost/1.64.0: 12,628 files, 395.7MB
==> Installing homebrew/science/dynare --HEAD --with-matlab= --with-matlab-version= --with-tex --without-octave
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Cloning github.com/DynareTeam/dynare.git
==> Checking out branch master
==> autoreconf -fvi
Last 15 lines from /Users/Joris/Library/Logs/Homebrew/dynare/01.autoreconf:
2017-04-21 09:28:45 +0200
READ THIS: docs.brew.sh/Troubleshooting.html
Besides, the full 01.autoreconf file is included below. Still don’t know why the HEAD version does not compile (while the stable version does compile fine).
Joriss-MacBook-Pro:~ Joris$ cat /Users/Joris/Library/Logs/Homebrew/dynare/01.autoreconf
2017-04-21 10:31:57 +0200
Last commit: 5 days ago
Core tap ORIGIN: github.com/Homebrew/homebrew-core
Core tap HEAD: cae5a2ba25264dbac1b73b8e4341178809dc42c5
Core tap last commit: 2 hours ago
CPU: octa-core 64-bit ivybridge
Homebrew Ruby: 2.0.0-p648
Clang: 8.1 build 802
Git: 2.11.0 => /Library/Developer/CommandLineTools/usr/bin/git
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
X11: 2.7.11 => /opt/X11
Thanks for pointing this out. The people at Homebrew update the formulas as other things change in their project. It’s a huge undertaking and I think that’s how this bug appeared. I fixed it here and it has been merged into homebrew-science. So, you should now be able to install the latest version of Dynare by running:
brew update && brew upgrade;
brew install dynare --HEAD --with-matlab=/Applications/MATLAB_R2017a.app --with-matlab-version=9.2 --without-octave --with-tex
Thanks for the quick response! With your instructions, I managed to install the HEAD version of Dynare.
Initially, even with the HEAD version, I could not run Dynare in parallel. However, after experimenting with some settings, I finally managed to run Dynare in parallel. I was not aware that I had to set the MatlabOctavePath option in the .dynare config file (the various PDFs with documentation seem to be a little bit confusing about this), but this option did the trick for me. See below how my .dynare config file looks like.
So far so good. But then I realised that the HEAD version (unstable 4.5) runs considerably slower than the current stable version. Given that it was my own mistake that the stable version did not run in parallel, I thought maybe I could go back to the stable version and use the appropriate .dynare config file, and run the stable version in parallel after all. This works nicely. So, I recorded the timings of some example estimation with the following results (I did run every version twice to be sure):
Parallel 4.5: 3m38s
Parallel 4.4.3: 2m16s
Normal 4.5: 6m11s
Normal 4.4.3: 4m49s
So, at least on my laptop and for my example estimation, parallel Dynare is considerably faster than normal Dynare (as is the purpose of using parallel Dynare in the first place). But the bad news seems to be that the newer version of Dynare is considerably slower than the current stable version. I thought this is useful information to share …
What is your experience with the relative speed of Dynare 4.5 (unstable) vs Dynare 4.4.3 (stable)? Do you have an explanation for my finding?
.dynare config file that works:
Name = local
Members = n1
Name = n1
ComputerName = localhost
MatlabOctavePath = /Applications/MATLAB_R2017a.app/bin/matlab
CPUnbr = 4
Glad it finally works.
- Yes, you have to set that path unless the default
works. That should be documented in the manual of the unstable version.
- I did a quick profiling and the reason seems to be a commit making the Kalman filter more robust github.com/DynareTeam/dynare/pull/1088 by properly conditioning a matrix. The increases the runtime of the Kalman filter by 30 to 40 percent in my tests. Thanks for reporting this. I opened a ticket: github.com/DynareTeam/dynare/issues/1437