I am using ubuntu 20.04 and Matlab 2021a. After I used Dynare in Matlab my computations using c++ in Rstudio with OpenMP become very slow. The multithreading does not work in the same way as before using Dynare. I had this problem before but I solved it by reinstalling Ubuntu. However, I would like to avoid that this time. 1) Does Dynare or Matlab change the default settings of Ubuntu for multithreading? 2) If so, how can I revert back to the default settings?
Dynare does not modify any system-wide setting. And as far as I know, Matlab doesn’t either.
It’s impossible to give you a more detailed help, without knowing exactly which computations you are doing and what are the problems that you experience (“slowness” is too vague).
I don’t see how running Dynare could affect further computations in another process. A more plausible explanation is that when installing Dynare you pulled in other dependencies (e.g. different flavours of computational libraries, pthread vs OpenMP) that affect your other computations?
Thank you for your answer.
In Dynare/Matlab I use only stoch_simul and also call resol(0,M_,options_,oo_) repeatedly for estimation. I use order of approximation greater than one (2 or 3 or more). Within Matlab I don’t use any parallel computing, but I notice that for order 3 Dynare uses many cpus for just one job. When moving from order 2 to 3 the computer uses many more threads, but the computing time (to solve the model with perturbation methods) becomes 7 times larger. Perhaps the multithreading setup is not efficient? My processor is AMD® Ryzen threadripper 3970x 32-core processor × 64.
But my main concern is the system-wide setting. In Rstudio I use #pragma omp parallel for. All threads work independently, only at the end results are put together. It is now slow becomes it takes much longer than before to do the work, for example if I use 15 or 20 threads. Something that should take one hour, now it seems to never finish. Also I notice that if I use 64 threads for the loop, the command top tells me that it uses only about 23% of cpu. However, it should be close to 100% if the settings were the default.
Some months ago I had the same problem. I solved it by reinstalling Ubuntu. I thought it was because I used parallel for. So this time I didn’t install the toolbox for parallel computations, and I didn’t try to do anything parallel in Matlab. I created a thread about this:
https://www.mathworks.com/matlabcentral/answers/887359-openmp-options-changed-by-matlab?s_tid=srchtitle
I only use my computer for Rstudio (c++), Matlab and Dynare (and some libreoffice, internet).
In another computer I have used the same Rstudio c++ code for several years and I have never had this problem. This other computer has windows, and does not have matlab/dynare installed.
If the problem was at the installation of Dynare, how can I solve it?
Regarding Dynare performance while moving from order 2 to 3, it’s not necessarily surprising that it takes both more time and more threads. For medium-sized models, solving at order 3 can be very costly.
Regarding your problem, I did not say that it is related to the installation of Dynare. It’s just that under Ubuntu, this will typically pull in other packages, so I mentioned that as a hypothesis, but I have no certainty. The first step is to diagnose your problem by understanding why your parallel jobs don’t take 100% of the CPU. Possibly it is because some threads are waiting for completion of the others, but I don’t know for sure without having a look at your algorithm. In any case, I’m afraid I don’t have the time to help you further, since I don’t think this has anything to do with Dynare.
Am I right that the solution is at
https://www.mathworks.com/matlabcentral/answers/887359-openmp-options-changed-by-matlab?s_tid=srchtitle
?
The solution there is to reinstall Ubuntu. I know this solves the problem but I was hoping there would be a more simple solution. If I reinstall Ubuntu, that solves the problem. But if I install Dynare later, I am afraid the problem will appear again.
I’m still not convinced that the problem comes from Dynare itself, since it does not modify any system-wide setting.
The problem is that Matlab or/and Matlab/Dynare installs OpenBlas, and in Ubuntu 20.04 it has a bug, which has been reported elsewhere: OpenBLAS broken on Ubuntu 20.04 · Issue #2642 · xianyi/OpenBLAS · GitHub
The solution is to uninstall OpenBlas. The following worked for me:
sudo apt-get remove libopenblas0
sudo apt-get remove libopenblas0-pthread
The code slowed when operating with matrices. OpenBlas was causing the threads to conflict with each other. Before removing OpenBlas I tried OPENBLAS_NUM_THREADS=1, but it didn’t solve the performance issue. After removing OpenBlas my computations in Rstudio using c++ are as fast as they used to be (much much faster than using OpenBlas).
1 Like
I would not remove OpenBLAS since it means your linear algebra computations will be much slower (those will use the reference Netlib implementation, which is not as much optimized).
In the Ubuntu bug report that you linked to, it is said that the bug has been fixed in a more recent version of the OpenBLAS packages (versioned 0.3.8+ds-1ubuntu0.20.04.1). So if this bug is indeed the source of your problem, the right solution seems to upgrade OpenBLAS, not to remove it.
1 Like