Wrapper Feature Selection
Machine learning operates under the straightforward tenet that whatever you feed it, it will only produce more of the same. Here, I refer to data noise as garbage.
When there are a lot of features, this becomes even more crucial. It’s not necessary to use every feature available while building an algorithm. By supplying only the most crucial features, you may help your algorithm. I have personally seen feature subsets produce better results for the same algorithm as the entire set of features.
This can be highly helpful in industrial applications in addition to competitions. You not only cut down on training and evaluation time, but you also have fewer worries!
Top justifications for feature selection include:
- The machine learning algorithm can train more quickly as a result.
- A model becomes less complicated and is simpler to interpret as a result.
- If the proper subset is selected, a model’s accuracy will increase.
- Overfitting is decreased.
We attempt to use a subset of features and train a model utilising them in wrapper methods. We choose to add or subtract features from your subset based on the conclusions we derive from the prior model. Essentially, the issue is simplified to a search issue. These techniques are typically highly expensive to compute.
Forward feature selection, backward feature elimination, recursive feature elimination, etc. are some typical instances of wrapper methods.
Forward Selection: In the iterative process of forward selection, we begin by not including any features in the model. The feature that best enhances our model is added in each iteration until the performance of the model is not improved by the addition of a new variable.
Backward Elimination: This technique enhances the performance of the model by starting with all the features and removing the least important one at a time. We keep doing this until we see no improvement when we remove features.
Recursive feature elimination is a greedy optimization technique that looks for the feature subset with the best performance. The best or worst performing feature is set aside after each round of model creation. Up till all the features have been used, it builds the subsequent model using the left features. The features are then ranked according to the order in which they were removed.
