Comparing AR and ARMA model – minimization of squared error

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)

As discussed in the previous post, the ARMA model is a generalized model that is a mix of both AR and MA model. Given a signal \(x[n]\), AR model is easiest to find when compared to finding a suitable ARMA process model. Let’s see why this is so.

AR model error and minimization

In the AR model, the present output sample \(x[n]\) and the past \(N-1\) output samples determine the source input \(w[n]\).The difference equation that characterizes this model is given by
$$x[n] + a_1 x[n-1] + a_2 x[n-2] + … + a_N x[n-N] = w[n]$$

The model can be viewed from another perspective, where the input noise \(w[n]\) is viewed as an error – the difference between present output sample \(x[n]\) and the predicted sample of \(x[n]\) from the previous \(N-1\) output samples. Let’s term this “AR model error”. Rearranging the difference equation,
$$w[n]= x[n]-\left(-\sum^{N}_{k=1}a_k x[n-k] \right)$$

The summation term inside the brackets are viewed as output sample predicted from past \(N-1\) output samples and their difference being the error \(w[n]\).
AR model error

Least Squared Estimated of the co-efficients – \(a_k\) are found by evaluating the first derivative of the squared error with respect to \(a_k\) and equating it to zero – finding the minima.From the equation above, \(w^2[n]\) is the squared error that we wish to minimize. Here, \(w^2[n]\) is a quadratic equation of unknown model parameters \(a_k\). Quadratic functions have unique minima, therefore it is easier to find the Least Squared Estimates of \(a_k\) by minimizing \(w^2[n]\).

ARMA model error and minimization

The difference equation that characterizes this model is given by
$$x[n] + a_1 x[n-1] + … + a_N x[n-N] = b_0 w[n] + b_1 w[n-1] + … + b_M w[n-M]$$

Re-arranging, the ARMA model error \(w[n]\) is given by
$$w[n]= x[n]-\left(-\sum^{N}_{k=1}a_k x[n-k] + \sum^{M}_{k=1}b_k w[n-k] \right)$$

Now, the predictor (terms inside the brackets) considers weighted combinations of past values of both input and output samples.
ARMA model error

The squared error, \(w^2[n]\) is NOT a quadratic function and we have two sets of unknowns – \(a_k\) and \(b_k\). Therefore, no unique solution may be available to minimize this squared error-since multiple minimas pose a difficult numerical optimization problem.

Recommended Books

See also:

[1]An Introduction to Estimation Theory
[2]Bias of an Estimator
[3]Minimum Variance Unbiased Estimators (MVUE)
[4]Maximum Likelihood Estimation
[5]Maximum Likelihood Decoding
[6]Probability and Random Process
[7]Likelihood Function and Maximum Likelihood Estimation (MLE)
[8]Score, Fisher Information and Estimator Sensitivity
[9]Introduction to Cramer Rao Lower Bound (CRLB)
[10]Cramer Rao Lower Bound for Scalar Parameter Estimation
[11]Applying Cramer Rao Lower Bound (CRLB) to find a Minimum Variance Unbiased Estimator (MVUE)
[12]Efficient Estimators and CRLB
[13]Cramer Rao Lower Bound for Phase Estimation
[14]Normalized CRLB - an alternate form of CRLB and its relation to estimator sensitivity
[15]Cramer Rao Lower Bound (CRLB) for Vector Parameter Estimation
[16]The Mean Square Error – Why do we use it for estimation problems
[17]How to estimate unknown parameters using Ordinary Least Squares (OLS)
[18]Essential Preliminary Matrix Algebra for Signal Processing
[19]Why Cholesky Decomposition ? A sample case:
[20]Tests for Positive Definiteness of a Matrix
[21]Solving a Triangular Matrix using Forward & Backward Substitution
[22]Cholesky Factorization and Matlab code
[23]LTI system models for random signals – AR, MA and ARMA models
[24]Comparing AR and ARMA model - minimization of squared error
[25]Yule Walker Estimation
[26]AutoCorrelation (Correlogram) and persistence – Time series analysis
[27]Linear Models - Least Squares Estimator (LSE)
[28]Best Linear Unbiased Estimator (BLUE)