10 Python single lines can simplify functional engineering

10 Python single lines can simplify functional engineering
Edit pictures | Midjourney

Functional engineering is a critical process in most data analytics workflows, especially when building machine learning models. It involves creating new features based on existing raw data capabilities to extract deeper analytical insights and enhance model performance. To help turbocharge and optimize your functional engineering and data preparation workflow, this article introduces 10 single lines (single-line code that effectively and concisely completes meaningful tasks) specifically. 10 practical single lines to keep Perform feature engineering processes in various situations and types of data, all in a simplified way.

Before you start, you may need to import some of the key Python libraries and modules we will use. Additionally, we will be on Scikit-Learn datasets Modules: Wine dataset and Boston housing dataset.

Note that the two datasets are loaded into two PANDAS DataFrames whose variables are named df_wine and df_bostonrespectively.

1. Normalization of numerical features (z-score scaling)

Normalization is a common approach, that is, when its values ​​span different ranges or sizes, the numerical characteristics in the data set are a scale approach, and there may be some moderate outliers. This scaling method converts the numerical value of the attribute to follow a standard normal distribution with an average of 0 and a standard deviation of 1. Scikit-Learn’s StandardScaler The class provides a seamless implementation of this method: all you need to do is call it fit_transform Methods, passing the function of a data frame that requires standardization:

The resulting standardized properties will now have smaller values ​​of about 0, some are positive and some are negative. This is completely normal, even if your original eigenvalues ​​are all positive, because standardization not only scales the data, but also centers around the mean of the original attribute.

2. Minimum maximum zoom

When values ​​in a function vary greatly across individual instances (e.g., the number of students in each classroom in a high school), minimum maximum scaling is an appropriate way to scale data: it involves normalizing the function value to move in unit intervals [0,1]by applying this formula to each value x: x’=(x-min)/(maximum-min)based on the maximum value (minimum value) value in the function x belong. Scikit-Learn offers a similar category to standardization.

In the example above, we use the Boston shell dataset to extend all features except MEDV (Intermediate House Value), which is designed to be the target variable for machine learning tasks such as regression, and therefore, it has been removed before normalization.

3. Add polynomial features

Adding polynomial features is useful when the data is not strictly linear but exhibits nonlinear relationships. The process boils down to the addition of new features resulting from increasing the original functionality to power and the interaction between them. This example uses PolynomialFeatures To be created from two functions describing the alcohol and malic properties of wine, the new function is the square of the original two (degree = 2), and another function that shows the interaction between the two functions by applying the product operator:

The result is that three new features are created on top of the original two: “Alcohol^2”, “malic_acid^2” and “Alcohol malic_acid”.

4. One wall coded classification variable

The coding consists of taking a categorical variable that occupies a possible value or category of “m” and creating a numerical (or rather, binary) feature, each characterizing the occurrence or absence of a category in a data instance, using values ​​of 1 and 0, respectively. Thanks Pandas’ get_dummies Function, this process cannot be easier. In the following example, we assume that the CHAS attribute should be treated as a classification and apply the above function to the single hot encoding of the function.

Since this feature initially takes two possible values, two new binary functions are built on it. In many data analysis and machine learning processes, single-rotation coding is a very important process in which purely cannot be handled and requires encoding.

5. Discrete continuous variables

Discrete continuous numerical variables in several equal width sub-spaces or bins are regular processes in the analysis process, such as visualization processes, that help to obtain plots such as histograms or line charts that may not seem so overwhelming but still capture “large figures”. This example single line shows how to discrete the “alcohol” attributes in the wine dataset into four bins, marked 0 to 3:

6. Logarithmic conversion of skewed features

If one of your numerical features is correct rotation or frontal skew, that is, it visually exhibits a long tail to the right due to several values ​​larger than the rest, the logarithmic transformation helps to extend them into better forms for further analysis. Numpy’s log1p This transformation is performed only by specifying the functions in the data frame that need to be converted. The results are stored in the newly built dataset feature.

7. Create ratios between two functions

One of the most direct but common feature engineering steps in data analysis and preprocessing is to create new features as a ratio (division) between the two semantically related. For example, given the alcohol and malic acid levels of wine samples, we may be interested in having a new property to describe the ratio between these two chemical properties, as shown below:

Thanks to the power of giant pandas, departmental operations that lead to new functionality are performed at the instance level of each instance (row) in the dataset without any loops.

8. Remove features with lower differences

Often, some functions may show so little variability between their values ​​that it not only contributes little to analysis or machine learning models trained on data, but may even make the results worse. Therefore, identifying and removing these features with lower differences is not a bad idea. This single line shows how to use Scikit-Learn VarianceThreshold The class automatically deletes features whose variance is below the threshold. Try adjusting the threshold to see how it affects the resulting feature removal, more or less aggressive based on the specified variance threshold.

Note: The MEDV attribute is manually deleted because it is a target variable for the dataset, regardless of other functions being deleted due to a lower variance threshold.

9. Multiplication Interaction

Assume our client is a wine producer in Lanzarote (Spain) which is used for marketing purposes, and its mass score combines information about alcohol and the color intensity of the wine into a single score. This can be done through functional engineering, simply participating in the functionality that will be calculated for new scores for each wine and applying the mathematics that the customer wants us to reflect on. For example, the products of these two functions:

10. Track outliers

Although outliers are often removed from the data set in most data analysis scenarios, it can sometimes be interesting to stay on track after they are identified. Why not do this by creating a new feature that indicates whether the data instance is more exceptional?

Single-line manually uses the interquartile range (IQR) method to discover possible outliers for tax attributes, which is why it spans length compared to previous examples. Depending on the dataset and target features, you want to analyze to discover outliers, in which case the newly added feature has a value of 0 for all instances in the dataset.

in conclusion

This article has a glimpse of ten effective Python single lines that, once familiar, will effectively perform the process of various functional engineering steps, thus turning your data into a good shape for further analyzing or building trained machine learning models.