How to see whether the model fit the real data well?

#1

Dear all

After estimate the model using bayesian method, how should I judge whether the estimation result fit the real data well?
I know in the estimation command, we can calculate the 1-step ahead forecast filtered variable, so is it ok to compare the forecast filtered variable with the real data and see whether the real data locate in the confidence intervals?

Besides, is there any method to compare the standard deviation between the model simulation data and the real data? Can we calculate the hpd intervals for the standard deviation of simulation data and see whether the standard deviation of real data locate in the hpd intervals?

Really looking for your kind help! Thank you so much.

#2

Full information estimation works essentially via minimizing the one-step ahead forecast errors. So checking the forecasting performance within the estimation sample does not make sense.

Comparing the standard deviations of simulated and actual data in contrast is a valid and worthwhile exercise, because the ML estimator will weight all moments according to their precision (and there are covariances at all leads and lags) while economists care particularly about a few select second moments.

#3

Thanks for your reply. I tried to do the simulation and calculate the standard deviation of the simulated endogenous variables. However, I don’t know whether my method is right. Can you help me to have a look at the code below?

shocks;
var e_ph_os; stderr 0.3821;
var e_h_os; stderr 0.0181;
var e_r_os; stderr 0.0115;
var e_c_os; stderr 0.0084;
var e_iv_os; stderr 0.0078;

end;
stoch_simul(periods=1000000,nograph) y;

Using this command, I get 1000000 periods simulation values of y. Then I calculated the standard deviation for every 100 period. Finally I get 10000 standard deviation values. So I can get the 5% quantile and 95% quantile.

#4

As a statistical test, that might work. However, most of the literature does more of an eyeballing comparison.