12- Logistic regression

Introduction

This tutorial will guide you through the fundamentals of logistic regression, a statistical method used for binary classification. Logistic regression is essential in various fields such as medicine, finance, and social sciences for predicting the probability of an event occurring based on one or more predictor variables. By the end of this tutorial, you will understand how logistic regression works, its applications, and how to implement it.

Step 1: Understanding Logistic Regression

• Logistic regression is used when the dependent variable is categorical, typically binary (e.g., success/failure).
• It estimates the probability that a given instance belongs to a particular category.
• The logistic function (sigmoid function) is used to map predicted values to probabilities ranging from 0 to 1.

Key Concepts

• Odds: The ratio of the probability of an event occurring to the probability of it not occurring.
• Logit: The natural logarithm of odds. The logit function is the link function used in logistic regression.

Step 2: The Logistic Function

• The logistic function is defined as:

  P(Y=1|X) = 1 / (1 + e^(-z))
  

  Where:

  • P(Y=1|X) is the probability that the dependent variable equals 1 given the independent variables.
  • z is a linear combination of the independent variables (features).
  • e is the base of natural logarithm.

Practical Tip

• Visualize the logistic function graph to understand how it curves and approaches 0 and 1, illustrating the probability distribution.

Step 3: Fitting the Model

• The logistic regression model is fitted using maximum likelihood estimation (MLE) to find the best-fitting parameters.
• Steps to fit the model:
  1. Collect and prepare your data.
  2. Determine the independent and dependent variables.
  3. Use statistical software or libraries (like Python's scikit-learn) to fit the model.

Example Code

Here’s an example of fitting a logistic regression model using Python's scikit-learn:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
import pandas as pd

# Load your dataset
data = pd.read_csv('data.csv')

# Prepare your features (X) and target variable (y)
X = data[['feature1', 'feature2']]
y = data['target']

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and fit the model
model = LogisticRegression()
model.fit(X_train, y_train)

Step 4: Making Predictions

• After fitting the model, you can make predictions using the predict method.
• To get the predicted probabilities, use the predict_proba method.

Example Code

# Making predictions
predictions = model.predict(X_test)
probabilities = model.predict_proba(X_test)

Step 5: Evaluating the Model

• Evaluate your logistic regression model using metrics such as:
  • Accuracy
  • Precision
  • Recall
  • F1 Score
  • ROC Curve

Common Pitfall

• Be cautious of class imbalance in your dataset, as it may skew the accuracy. Consider techniques like oversampling, undersampling, or using metrics that account for imbalance.

Conclusion

Logistic regression is a powerful tool for binary classification problems. By following these steps, you can understand its foundational concepts, implement the model, and evaluate its performance effectively. As a next step, consider exploring more advanced topics such as regularization techniques, multinomial logistic regression, or applying logistic regression to real-world datasets to deepen your understanding.