Ensemble learning using the Voting Classifier
Learn how to leverage the strengths of multiple models using a variant of ensemble learning
In this article, I describe a simple ensemble algorithm. In general, ensemble models combine multiple base models to improve the predicting performance. The best-known example of an ensemble model is the Random Forest, which — greatly simplifying the algorithm’s logic — combines multiple Decision Trees and aggregates their predictions using majority vote in case of a classification problem or by taking the average for regression tasks.
Similarly to the Random Forest, the Voting Ensemble estimates multiple base models and uses voting to combine the individual predictions to arrive at the final ones. However, the key difference lies in the base estimators. Models such as Voting Ensemble (and Stacking Ensemble) do not require the base models to be homogenous. In other words, we can train different base learners, for example, a Decision Tree and a Logistic Regression, and then use the Voting Ensemble to combine the results.
The following diagram presents the workflow of the Voting Ensemble:
Voting Classifier supports two types of voting:
- hard: the final class prediction is made by a majority vote — the estimator chooses the class prediction that occurs most frequently among the base models.
- soft: the final class prediction is made based on the average probability calculated using all the base model predictions. For example, if model 1 predicts the positive class with 70% probability, model 2 with 90% probability, then the Voting Ensemble will calculate that there is an 80% chance of the observation belonging to the positive class and choose the positive class as the prediction. Additionally, we can use custom weights to calculate the weighted average. This is apt for cases in which we put more trust in some models, but still want to consider the ones we trust less.
One thing to bear in mind is that in order to use soft voting, all the base models must have the predict_proba method…
