State Space Model with missing observations

problem.zip (61.0 KB)
A mixed quarterly, monthly SSM. Bayesian estimation. After sampling, marginal posteriors successfully obtained. Smoothed estimation of the unobserved state variables in model results in error message:

Subscripted assignment dimension mismatch.

Error in missing_DiffuseKalmanSmootherH1_Z (line 137)
Fstar(:,:,t) = ZZ*Pstar(:,:,t)*ZZ’ + H(di,di); % (5.7) in DK
(2012)

Error in DsgeSmoother (line 231)
[alphahat,epsilonhat,etahat,ahat,P,aK,PK,decomp,state_uncertainty] =
missing_DiffuseKalmanSmootherH1_Z(ST, …

Error in prior_posterior_statistics_core (line 208)
DsgeSmoother(deep,gend,Y,data_index,missing_value,M_,oo_,options_,bayestopt_,estim_params_);

Error in prior_posterior_statistics (line 241)
[fout] = prior_posterior_statistics_core(localVars,1,B,0);

Error in dynare_estimation_1 (line 520)
prior_posterior_statistics(‘posterior’,dataset_,dataset_info);

Error in dynare_estimation (line 105)
dynare_estimation_1(var_list,dname);

Error in example_problem (line 773)
oo_recursive_=dynare_estimation(var_list_);

Error in dynare (line 223)
evalin(‘base’,fname) ;

Any help would be greatly appreciated! I have included the code and data in the zip file. (Other versions of this SSM model work, versions which only slightly differ from this current version.)

Thank you!

Sincerely,
Rob Luginbuhl

Are you using Dynare 4.5.x? In this version several bug related to the missing diffuse Kalman filter have been fixed. See e.g. https://github.com/DynareTeam/dynare/pull/1235

I am using 4.5.3.
Thank you for the prompt reply!
Sincerely,
Rob Luginbuhl

That is strange. Could you please post the full log-file. I cannot replicate any problem on my machine. Or what do I need to do with your attached files?

I think if you just unpack the files in my original zip file and run the mod file with Dynare that the problem should occur. At least that is what happens in my work environment.
NLcycle_ABcycles5_res34.zip (6.6 KB)

I will include the log file with this reply (in zip).

Thank you.

Sincerely,
Rob Luginbuhl

UPDATE: we updated Matlab to R2017b over the Christmas vacation and now the error message seems to have disappeared (at least for a short run of 2 chains and 1000 replications). I will try a longer run of 4 chains and 200000 replications to see if this also works.

I am in any case puzzled by the error/warning message:

“univariate_diffuse_kalman_filter:: T does influence the rank of Pinf!”

Sincerely,
Rob Luginbuhl

(NB: mod file in Original zip contains faulty Matlab code at the end of the file. I don’t care about this error, as I am not yet using this code. In any case, the actual error message of interest occurs before dynare ever gets to this matlab code.)

That is strange. Which Matlab version were you using?
Regarding the

univariate_diffuse_kalman_filter:: T does influence the rank of Pinf!
you need to find out why the univariate filter is used in the first place

NLcycle_ABcycles5_NOres.zip (93.2 KB)

Sorry, the error message

{Subscripted assignment dimension mismatch.

Error in missing_DiffuseKalmanSmootherH1_Z (line 137)

has returned! Apparently this was not due to the Matlab version. The version I am now using is R2017b, by the way.

Above I have included a zip to yet another version of the model I am estimating (now an unrestricted version). The mod file makes use of the existing mode file found using mod=6, and it also reads in the existing replications from 2 chains (which I did not include in the zip file, but there were only 2000 replications per chain).

I am not sure, but perhaps the problem occurs when I read in the old mod file and exising replications? (Although I am also fairly sure that I’ve had this problem when starting without a mod file or existing replications, and use mod=6).

Sincerely,
Rob Luginbuhl

Thanks for reporting this. This is indeed a bug. You can find a fix at https://github.com/DynareTeam/dynare/pull/1581

Glad to help!

Thank you again for the prompt and helpful replies!

-Rob

Hello again. I have started using Dynare 4.5.4 with US data. The data structure is similar. There is a mix of quarterly and monthly data in the model with 5 observable series: GDP, Credit, House prices, Indust. Prod., and cummulated Schillers PE. (All series real). There is something wrong with the priors being used in Dynare. The trends are local linear trends (I(2), so non-stationary) and must have diffuse priors, but some of the estimated trends are starting at 0 which indicates that the prior is proper around 0.

Here are the files to replicate the problem:
DynareSSMproblemMissing.zip (80.6 KB)

Any help would be greatly appreciated!

Thank you.

Sincerely,
Rob Luginbuhl

Dear Rob,
there are many moving parts here that make the problem hard to debug. For one, your model reverts to using the univariate diffuse filter.
Does the problem also appear in a univariate context, i.e. if you only estimate the unobserved components model for lc? If yes, could you please provide me with that file.

Dear Johannes,

Thank you again for taking the time to study this problem! Here is my update, while I continue to work on a univariate version to send to you.

I have misspecified the seasonal component for the higher frequency for all three series for which there is only quarterly data available. For example:

seas52 = -seas52(-1) + seassig51*e_seas52;

This incorrectly says that the period is two months, which for quarterly data is not identifiable, and so this is wrong. This could potentially be specified as

seas52 = -seas52(-3) + seassig51*e_seas52;

but this does not work.

Clearly the combination of a diffuse KF and trigonometric seasonal components (which are stationary) in the presence of missing values (quarterly series in a monthly multivariate model) is proving to be too much for Dynare. In theory the KF should be able to handle this, but something is going wrong.

I am now running a univariate model for US credit with the seasonal components removed from the model. I want to see if this works before I submit any more mod files to you.

Once again thank you for your help. I will send a new mod file with a univariate version as soon as I can figure out what would be most useful to send to you.

If in the meantime you have any more thoughts or suggestions I would be glad to hear them!

Sincerely,
Rob

Hi,
thanks for the reply. What is the problem you are experiencing when doing

?

Sorry, actually I guess that the specification of the seasonal with (-1)

seas52 = -seas52(-1) + seassig51*e_seas52;

is actually okay, because (-1)^3 = -1 so after three months this still gives a value of -1…

I have experimented with univariate estimation. I’m sending you in the zip a univariate version with US credit. As specified it now will run.

But it should also run with first_obs=1 so that the first period of yearly data is also used in the estimation. The seasonal components in the Kalman Filter will not be determined until the quarterly data starts, but the difuse KF should be able to handle that it seems to me. And yet Dynare crashes if first_obs=1 is used.

Thanks again for the help!

Sincerely,
Rob US_cred.zip (55.4 KB)

Dear Johannes,

I hope you don’t mind if I check with you about my last reply. I am not sure if I was very clear in my response. I am still unable to estimate my model when the data is initially yearly data and then at some point becomes quarterly or monthly. This seems to be the case when I include seasonal components in the model.

I am also puzzled by the warning message that repeatedly appears on the screen during estimation about the univariate diffuse kalman filter: T does influence the rank of Pinf. I would expect that for a multivariate model with covariances across series that the univariate kalman filter would be inappropriate.

Thanks again for the help!

Sincerely,
Rob

Hi,
I am not entirely sure what is happening and did not have the time to look at this in detail. But Durbin/Koopman (2012) in their book discuss on page 125 that numerical problems may occur if you have missing values at the beginning of your time series. This seems to be what is happening here. It may explain why the univariate must be used. Note that the univariate Kalman filter still correctly deals with covariances across series. See again the book of Durbin/Koopman. I hope to have a deeper look at this soon.