Shrink to Win: Regularization’s Role in Model Success

Overfitting? Shrink your model! Regularization prevents memorization, enabling true learning & robust success. Discover how. A central problem in machine learning is to develop algorithms that work well both on training data and on new inputs (test data). Most machine learning tasks can be generalized as the estimation of a function (hat{f}(X)) that maps the … Read more

Hidden Markov Models (HMM) – Simplified !!!

Markov chains are useful in computing the probability of events that are observable. However, in many real world applications, the events that we are interested in are usually hidden, that is we don’t observe them directly. These hidden events need to be inferred. For example, given a sentence in a natural language we only observe the … Read more

Markov Chains – Simplified !!

Key focus: Markov chains are a probabilistic models that describe a sequence of observations whose occurrence are statistically dependent only on the previous ones. ● Time-series data like speech, stock price movements.● Words in a sentence.● Base pairs on the rung of a DNA ladder. States and transitions: markov chains Assume that we want to … Read more

Generate color noise using Auto-Regressive (AR) model

Key focus: Learn how to generate color noise using auto regressive (AR) model. Apply Yule Walker equations for generating power law noises: pink noise, Brownian noise. Auto-Regressive (AR) model An uncorrelated Gaussian random sequence \(x[n]\) can be transformed into a correlated Gaussian random sequence \(y[n]\) using an AR time-series model. If a time series random … Read more

Generating colored noise with Jakes PSD: Spectral factorization

The aim of this article is to demonstrate the application of spectral factorization method in generating colored noise having Jakes power spectral density. Before continuing, I urge the reader to go through this post: Introduction to generating correlated Gaussian sequences. In spectral factorization method, a filter is designed using the desired frequency domain characteristics (like … Read more

Generate correlated Gaussian sequence (colored noise)

Key focus: Colored noise sequence (a.k.a correlated Gaussian sequence), is a non-white random sequence, with non-constant power spectral density across frequencies. Introduction Speaking of Gaussian random sequences such as Gaussian noise, we generally think that the power spectral density (PSD) of such Gaussian sequences is flat.We should understand that the PSD of a Gausssian sequence … Read more

Linear Models – Least Squares Estimator (LSE)

Key focus: Understand step by step, the least squares estimator for parameter estimation. Hands-on example to fit a curve using least squares estimation Background on least squares estimator The various estimation concepts/techniques like Maximum Likelihood Estimation (MLE), Minimum Variance Unbiased Estimation (MVUE), Best Linear Unbiased Estimator (BLUE) – all falling under the umbrella of classical … Read more

AutoCorrelation (Correlogram) and persistence – Time series analysis

The agenda for the subsequent series of articles is to introduce the idea of autocorrelation, AutoCorrelation Function (ACF), Partial AutoCorrelation Function (PACF) , using ACF and PACF in system identification. Introduction Given time series data (stock market data, sunspot numbers over a period of years, signal samples received over a communication channel etc.,), successive values … Read more

Yule Walker Estimation: AR Model in Matlab and Python

Yule Walker estimation fits an auto-regressive model to a time series. Suppose the series is modeled as an AR(N) process. The model parameters $a_1,\dots,a_N$ can be estimated by least squares, but the equations become cumbersome as the order $N$ grows. The Yule-Walker equations are a cleaner route. They tie the AR coefficients to the autocorrelation … Read more