Calculating RMSEs


Thank you for a great and helpfull forum.

I would like to calculate the RMSEs in order to compare two models, however I am a bit unsure whether I have to do it manually or I can use the dynare_sensitivity function.

I do a recursive forecasting procedure in the estimation with nobs and forecast=, and I find the forecast mean for each variable in the folder oo_RecursiveForecast.Mean.

If I calculate the RMSEs manually, do I then have to calculate the RMSE for each variable and horizon (1 step ahead, 2 step ahead, etc) according to the following formula:

RMSE_x^h = (1/T*sum((x_actual_t - x_forecast^h_t)^2))^(1/2) ?

where h is the horizon for the forecast, x_actual is the actual observed data, x_forecast^h_t is the forecasted value, T is the recursive forecast period.

I have tried to use the function dynare_sensitivity to calculate the RMSEs with the following code:
dynare_sensitivity(datafile=data_mean_div100, ppost=1, first_obs=1, nobs=[85:87], rmse=1);

However, it gives me the following error:
"Error using : (line 38)
dates::colon: In an expression like A:B, A and B must be dates objects!
Error in makedataset (line 252)
DynareDataset = DynareDataset(FIRSTOBS:lastobs);
Error in dynare_estimation_init (line 538)
[dataset_, dataset_info, newdatainterfaceflag] = makedataset(options_,
options_.dsge_var*options_.dsge_varlag, gsa_flag);
Error in dynare_sensitivity (line 124)
[dataset_,dataset_info,xparam1,hh, M_, options_, oo_,
estim_params_,bayestopt_]=dynare_estimation_init(M_.endo_names,fname_,1, M_,
options_, oo_, estim_params_, bayestopt_);
Error in Gali_estimation_2_baseline2 (line 403)
Error in dynare (line 223)
evalin(‘base’,fname) ;

I have searched the forum without finding concrete help, so I hope someone have the time to answer my question.

I have attached my code and data (I have set a small number of MH-draws and nobs so it won’t take too long to calculate, while I am still unsure of the procedure).
Code (6.9 KB)

// Anna

@rattoma I guess one can compute the RMSE based on ppost for a Monte Carlo sample. The problem here is that there seems to be a bug that prevents running the dynare_sensitivity-command with recursive estimation.

I would recommend computing the RMSE manually for now. That should be straightfoward based on the results in oo_recursive

Hi Johannes
Many thanks. Yes I think this would be the only way for now