Tag: linear regression

The Most Important Topics to Learn in Machine Learning

Keywords: machine learning, topics, probability, statistics, linear algebra, data preprocessing, supervised learning, unsupervised learning, deep learning, reinforcement learning, model evaluation, cross-validation, hyperparameter tuning.

Why the buzz ?

Machine learning has been generating a lot of buzz in recent years due to its ability to automate tasks that were previously thought to be impossible or required human-level intelligence. Here are some reasons why there is so much buzz in machine learning:

  1. Improved Data Processing: Machine learning algorithms can process vast amounts of data quickly and accurately. With the advent of big data, there is now more data available than ever before, and machine learning algorithms can analyze this data to extract meaningful insights.
  2. Automation: Machine learning can automate tasks that were previously done by humans, such as image recognition, natural language processing, and even decision making. This has the potential to increase efficiency and reduce costs in many industries.
  3. Personalization: Machine learning can be used to personalize experiences for users. For example, recommendation systems can use machine learning algorithms to suggest products or services that are relevant to a user’s interests.
  4. Predictive Analytics: Machine learning can be used to make predictions about future events based on historical data. This is particularly useful in industries like finance, healthcare, and marketing.
  5. Advancements in Technology: Advancements in technology have made it easier to collect and store data, which has made it possible to train more complex machine learning models. Additionally, the availability of cloud computing has made it easier for companies to implement machine learning solutions.

Overall, the buzz in machine learning is due to its ability to automate tasks, process vast amounts of data, and make predictions about future events. As machine learning continues to evolve, it has the potential to transform many industries and change the way we live and work.

The most important topics to learn in machine learning

There are several important topics to learn in machine learning that are crucial for building effective machine learning models. Here are some of the most important topics to learn:

  1. Probability and Statistics: Probability and statistics are the foundation of machine learning. It is important to have a solid understanding of concepts like probability distributions, statistical inference, hypothesis testing, and Bayesian methods.
  2. Linear Algebra: Linear algebra is used extensively in machine learning algorithms, especially in deep learning. Topics like matrices, vectors, eigenvectors, and eigenvalues are important to understand.
  3. Data Preprocessing: Data preprocessing is the process of cleaning and transforming raw data into a format that can be used by machine learning algorithms. It includes tasks like feature scaling, feature selection, data normalization, and data augmentation.
  4. Supervised Learning: Supervised learning is a type of machine learning where the model learns from labeled data to make predictions or classifications on new, unseen data. This includes topics like regression, classification, decision trees, and support vector machines.
  5. Unsupervised Learning: Unsupervised learning is a type of machine learning where the model learns from unlabeled data to discover patterns and relationships in the data. This includes topics like clustering, dimensionality reduction, and anomaly detection.
  6. Deep Learning: Deep learning is a subset of machine learning that involves training artificial neural networks with multiple layers. It is used for tasks like image recognition, natural language processing, and speech recognition.
  7. Reinforcement Learning: Reinforcement learning is a type of machine learning where an agent learns to take actions in an environment to maximize a reward signal. It is used for tasks like game playing, robotics, and autonomous driving.
  8. Model Evaluation and Selection: Model evaluation and selection is the process of selecting the best machine learning model for a given task. It includes topics like cross-validation, bias-variance tradeoff, and hyperparameter tuning.

Overall, these are some of the most important topics to learn in machine learning. However, it is important to note that the field of machine learning is constantly evolving, and there may be new topics and techniques to learn in the future.

Linear regression using python – demystified

Key focus: Let’s demonstrate basics of univariate linear regression using Python SciPy functions. Train the model and use it for predictions.

Linear regression model

Regression is a framework for fitting models to data. At a fundamental level, a linear regression model assumes linear relationship between input variables () and the output variable (). The input variables are often referred as independent variables, features or predictors. The output is often referred as dependent variable, target, observed variable or response variable.

If there are only one input variable and one output variable in the given dataset, this is the simplest configuration for coming up with a regression model and the regression is termed as univariate regression. Multivariate regression extends the concept to include more than one independent variables and/or dependent variables.

Univariate regression example

Let us start by considering the following example of a fictitious dataset. To begin we construct the fictitious dataset by our selves and use it to understand the problem of linear regression which is a supervised machine learning technique. Let’s consider linear looking randomly generated data samples.

import numpy as np
import matplotlib.pyplot as plt #for plotting

np.random.seed(0) #to generate predictable random numbers

m = 100 #number of samples
x = np.random.rand(m,1) #uniformly distributed random numbers
theta_0 = 50 #intercept
theta_1 = 35 #coefficient
noise_sigma = 3

noise = noise_sigma*np.random.randn(m,1) #gaussian random noise

y = theta_0 + theta_1*x + noise #noise added target
 
plt.ion() #interactive plot on
fig,ax = plt.subplots(nrows=1,ncols=1)
plt.plot(x,y,'.',label='training data')
plt.xlabel(r'Feature $x_1$');plt.ylabel(r'Target $y$')
plt.title('Feature vs. Target')
Simulated data for linear regression problem
Figure 1: Simulated data for linear regression problem

In this example, the data samples represent the feature and the corresponding targets . Given this dataset, how can we predict target as a function of ? This is a typical regression problem.

Linear regression

Let be the pair that forms one training example (one point on the plot above). Assuming there are such sample points as training examples, then the set contains all the pairs .

In the univariate linear regression problem, we seek to approximate the target as a linear function of the input , which implies the equation of a straight line (example in Figure 2) as given by

where, is the intercept, is the slope of the straight line that is sought and is always . The approximated target serves as a guideline for prediction. The approximated target is denoted by

Using all the samples from the training set , we wish to find the parameters that well approximates the relationship between the given target samples and the straight line function .

If we represent the variables s, the input samples for and the target samples as matrices, then, equation (1) can be expressed as a dot product between the two sequences

It may seem that the solution for finding is straight forward

However, matrix inversion is not defined for matrices that are not square. Moore-Penrose pseudo inverse generalizes the concept of matrix inversion to a matrix. Denoting the Moore-Penrose pseudo inverse for as , the solution for finding is

For coding in Python, we utilize the scipy.linalg.pinv function to compute Moore-Penrose pseudo inverse and estimate .

xMat = np.c_[ np.ones([len(x),1]), x ] #form x matrix
from scipy.linalg import pinv
theta_estimate = pinv(xMat).dot(y)
print(f'theta_0 estimate: {theta_estimate[0]}')
print(f'theta_1 estimate: {theta_estimate[1]}')

The code results in the following estimates for , which are very close to the values used to generate the random data points for this problem.

>> theta_0 estimate: [50.66645323]
>> theta_1 estimate: [34.81080506]

Now, we know the parameters of our example system, the target predictions for new values of feature can be done as follows

x_new = np.array([[-0.2],[0.5],[1.2] ]) #new unseen inputs
x_newmat = np.c_[ np.ones([len(x_new),1]), x_new ] #form xNew matrix
y_predict  = np.dot(x_newmat,theta_estimate)
>>> y_predict #predicted y values for new inputs for x_1
array([[43.70429222],
       [68.07185576],
       [92.43941931]])

The approximated target as a linear function of feature, is plotted as a straight line.

plt.plot(x_new,y_predict,'-',label='prediction')
plt.text(0.7, 55, r'Intercept $\theta_0$ = %0.2f'%theta_estimate[0])
plt.text(0.7, 50, r'Coefficient $\theta_1$ = %0.2f'%theta_estimate[1])
plt.text(0.5, 45, r'y= $\theta_0+ \theta_1 x_1$ = %0.2f + %0.2f $x_1$'%(theta_estimate[0],theta_estimate[1]))
plt.legend() #plot legend
Figure 2: Linear Regression – training samples and prediction

Rate this article: Note: There is a rating embedded within this post, please visit this post to rate it.

References

[1] Boyd and Vandenberghe , “Convex Optimization”, ISBN: 978-0521833783, Cambridge University Press, 1 edition, March 2004.↗

Related topics

[1] Introduction to Signal Processing for Machine Learning
[2] Generating simulated dataset for regression problems - sklearn make_regression
[3] Hands-on: Basics of linear regression

Books by the author


Wireless Communication Systems in Matlab
Second Edition(PDF)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart

Digital Modulations using Python
(PDF ebook)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart

Digital Modulations using Matlab
(PDF ebook)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart
Hand-picked Best books on Communication Engineering
Best books on Signal Processing

Generating simulated dataset for regression problems

Key focus: Generating simulated dataset for regression problems using sklearn make_regression function (Python 3) is discussed in this article.

Problem statement

Suppose, a survey is conducted among the employees of a company. In that survey, the salary and the years of experience of the employees are collected. The aim of this data collection is to build a regression model that could predict the salary from the given experience (especially for the values not seen by the model).

If you are developer, you often have no access to survey data. In this scenario, you wish you could simulate the data for building a regression model.

Generating the dataset

To construct a simulated dataset for this scenario, the sklearn.dataset.make_regression↗ function available in the scikit-learn library can be used. The function generates the samples for a random regression problem.

The make_regression↗ function generates samples for inputs (features) and output (target) by applying random linear regression model. The values for generated samples have to be scaled to appropriate range for the given problem.

import numpy as np
from sklearn import datasets
import matplotlib.pyplot as plt #for plotting

x, y, coef = datasets.make_regression(n_samples=100,#number of samples
                                      n_features=1,#number of features
                                      n_informative=1,#number of useful features 
                                      noise=10,#bias and standard deviation of the guassian noise
                                      coef=True,#true coefficient used to generated the data
                                      random_state=0) #set for same data points for each run

# Scale feature x (years of experience) to range 0..20
x = np.interp(x, (x.min(), x.max()), (0, 20))

# Scale target y (salary) to range 20000..150000 
y = np.interp(y, (y.min(), y.max()), (20000, 150000))

plt.ion() #interactive plot on
plt.plot(x,y,'.',label='training data')
plt.xlabel('Years of experience');plt.ylabel('Salary $')
plt.title('Experience Vs. Salary')
Figure 1: Simulated dataset for linear regression problem

If you want the data to be presented in pandas dataframe format:

import pandas as pd
df = pd.DataFrame(data={'experience':x.flatten(),'salary':y})
df.head(10)
Figure 2: Generated dataset presented as pandas dataframe

We have successfully completed generating simulated dataset for regression problems in Python3. Let’s move on to build and train a linear regression model using the generated dataset and use it for predictions.

Rate this article: Note: There is a rating embedded within this post, please visit this post to rate it.

Related topics

[1] Introduction to Signal Processing for Machine Learning
[2] Generating simulated dataset for regression problems - sklearn make_regression
[3] Hands-on: Basics of linear regression

Books by the author


Wireless Communication Systems in Matlab
Second Edition(PDF)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart

Digital Modulations using Python
(PDF ebook)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart

Digital Modulations using Matlab
(PDF ebook)
Note: There is a rating embedded within this post, please visit this post to rate it.
Checkout Added to cart
Hand-picked Best books on Communication Engineering
Best books on Signal Processing