The k-nearest Neighbours (kNN) Algorithm in Python

Machine Learning Basics

Take a step back and quickly review machine learning in general to help you get on board. In this part, you will learn about the core concept of machine learning and how the kNN method connects to other machine learning technologies.

The main goal of machine learning is to train a model to recognize patterns in any historical data so that it can repeat those patterns on similar data in the future. Here is a flowchart showing the fundamental steps in machine learning:

This graph shows a machine learning model fitted to historical data. The initial observations, which included height, width, and form measurements, are shown on the left. Triangles, crosses, and stars make up the forms.

Different parts of the graph include the forms. You can see how those first observations were converted into a decision rule on the right. You need to know a new observation's width and height to identify which square it belongs in. What form it will take depends on which square it lands in.

For this objective, several alternative models might be applied. A mathematical formula that may be used to desrcibe data points is called a model. The linear model, which employs a linear function denoted by the equation y = ax + b, is one instance.

When you estimate or fit a model, an algorithm determines the best values for the fixed parameters. The parameters of the linear model are a and b. Fortunately, you won't need to create these estimating methods from srcatch to get started. Outstanding mathematicians have previously found them.

Once the model has been estimated, it may be used as a mathematical formula to forecast the values of your target variable by entering values for your independent variables. From a high level, that is everything that occurs!

Distinguishing Qualities of kNN

The next stage is to comprehend why so many models are accessible now that you know how machine learning works. Regression using a linear model is known as linear modeling.

Although it sometimes produces accurate predictions, linear regression is only sometimes effective. Mathematicians have developed a wide range of additional machine-learning models that you may utilize as a result. One of these is the k-Nearest Neighbors algorithm.

Each of these models has unique characteristics. If you work in machine learning, you should thoroughly grasp each one to apply the appropriate model to the problem. The next step is to examine how kNN stacks up against other machine learning models to comprehend why and when to utilize kNN.

This supervised machine learning algorithm is known as kNN. The distinction between supervised and unsupervised models is the first defining characteristic of machine learning algorithms. The problem statement distinguishes supervised and unsupervised models.

In supervised models, there are two different types of variables:

1. A target variable is sometimes called a dependent or y variable.
2. Independent variables, sometimes called "x variables" or "explanatory variables."

The variable you wish to forecast is known as the target variable. You cannot predict it in advance, and it depends on the independent variables. The independent variables are ones that you are aware of in advance. To forecast the desired variables, you can plug them into an equation. This makes it comparable to the y = ax + b situation.

The form of the data point is the goal variable in the previous graph and the subsequent graphs in this section, whereas height and breadth are independent variables. The following graph illustrates how supervised learning works:

Each data point in this graph has a height, breadth, and shape. There are triangles, crosses, and stars. A decision rule that a machine learning model may have learned is shown on the right.

In this instance, the observations highlighted with a cross are tall but not broad. Both towering and broad desrcibe stars. Triangles might be broad or narrow, but they are all short. The model has learned a decision rule that uses just an observation's height and breadth to determine whether it is more likely to be a cross, a star, or a triangle.

Target variables and independent variables are not divided into unsupervised models. Unsupervised learning aims to group data items by determining how similar they are.

Although you can never be sure that grouped data items genuinely belong together, as you can see in the example, grouping may be useful in practice if it makes sense. The following graph illustrates how unsupervised learning works:

Distinct forms no longer represent the observations in this graph. All of them are circles. However, depending on the separation of the points, they may still be divided into three categories. Three groups of points may be distinguished in this specific case based on the void space that separates them.

A supervised machine learning model is the kNN algorithm. In other words, it predicts a target variable based on one or more independent factors.

Read K-Means Clustering in Python: A Practical Guide to learn more about unsupervised machine learning methods.

A Nonlinear Learning Algorithm Is kNN

The ability of the models to estimate nonlinear connections is a second factor that significantly affects machine learning methods.

Models that forecast using lines or hyperplanes are known as linear models. The model is shown in the picture as a line connecting the locations. The standard example of a linear model is y = ax + b. In the following graphic, you can see how a linear model may match the test data:

The data points in this image are shown on the left as stars, triangles, and crosses. A linear model that can distinguish triangles from non-triangles is shown on the right. The choice is on a line. Triangles only exist below the line; all points above it are non-triangles.

The preceding graph would need to be drawn with the additional independent variable added as a dimension, resulting in a cube with the forms within. However, a line couldn't split a cube in half. The hyperplane is the line's equivalent in three dimensions. A hyperplane, which in the case of two-dimensional space happens to be a line, therefore represents a linear model.

Nonlinear models divide their instances into groups using a method other than a line. The decision tree, a lengthy series of if... else statements, is a well-known example. Suppose... else statements would let you design squares or any other shape you desired in the nonlinear graph. A nonlinear model used on the example data is shown in the graph below:

This graph demonstrates the nonlinearity of a choice. Three squares make up the decision rule. A new data point's anticipated form will be determined by the box into which it falls. Remember that using a line will only allow you to fit this some at a time: It takes two lines. The following if... else statements might be used to recreate this model:

The data point is a star if none of the above conditions are true. It is a triangle if the data point's height is low. It is a cross if the data point's width is low.

A nonlinear model is one like kNN. You'll return to the precise method for calculating the model later in this course.

In terms of both classification and regression, kNN is a supervised learner.

Based on the kind of target variable that they can forecast, supervised machine learning algorithms may be divided into two groups:

1. A prediction job with a categorical target variable is classification. Models for classification learn how to categorize every new observation. There is no middle ground; this class can be correct or wrong. The iris dataset, which uses physical measures of plants to infer their species, is a well-known example of categorization. Logistic regression is a well-known algorithm that may be applied to classification.
2. The prediction objective of regression uses a numeric target variable. The Kaggle challenge on housing prices is a well-known illustration of regression. Participants in this machine learning competition attempt to forecast homes' sales values using various independent factors.

You can see an example of a regression and a classification using the prior example in the following graph:

This image's left side is a categorization. The observation's shape, a categorical variable, serves as the target variable. Regression is in the right portion. The variable's goal is a number. Even if the two cases' interpretations of the decision rules may be identical, they differ.

Classifications provide a binary outcome for a single prediction, whereas regressions have an error on a continuous scale. Since having a numerical error measure is more useful, many classification models forecast both the class and the likelihood of belonging to either class.

Some models can perform classification, some can only perform regression, and others both. The kNN algorithm can handle both classification and regression with ease. In the following section of this course, you will discover exactly how this operates.

Fast and Interpretable kNN

Model complexity must be considered the last criterion for desrcibing machine learning models. Artificial intelligence, in particular machine learning, is now in a state of bloom and is being employed for various challenging tasks, including the processing of text, pictures, and voice, as well as for self-driving automobiles.

A k-Nearest Neighbors model can learn anything more sophisticated and complicated models, like neural networks, can. Those sophisticated models are, after all, quite powerful learners. Be warned, though, that this intricacy comes at a cost. You'll spend much more on developing the models to suit your forecast.

To fit a more complicated model, you'll also need a lot more data, and data isn't always accessible. More complicated models are more challenging for us to comprehend as humans, even though this interpretation can occasionally be beneficial.

Herein lies the power of the kNN model. It's relatively quick to construct and enables users to comprehend and analyze what's occurring inside the model. As a result, kNN is a fantastic model for many machine learning use cases that don't call for extremely complicated procedures.

Problems with kNN

Being open and honest about the kNN algorithm's shortcomings is only right. As was already said, its main weakness is kNN's inability to adjust to extremely complicated interactions between independent and dependent variables. Advanced tasks like computer vision and natural language processing are less likely for kNN to succeed.

You can maximize the performance of kNN by adding other machine learning methods. You'll examine a method called bagging, which is a strategy to enhance prediction results, in the final section of the class. However, regardless of how it was calibrated, kNN will likely be less efficient than alternative models at a certain level of complexity.

To estimate the age of sea slugs, use kNN.

For the remainder of this course, you'll look at the Abalone Dataset as an example dataset to follow along with the coding portion. Numerous abalone age metrics are included in this collection. For informational purposes only, here is what an abalone looks like:

Statement of the Abalone Problem

Cutting an abalone's shell open and counting the number of rings on it can reveal how old it is. The age measurements of many abalones and other physical measurements may be found in the Abalone Dataset.

The project aims to create a model that can only estimate an abalone's age from its physical characteristics. By doing so, scientists could calculate an abalone's age without splitting open its shell and counting its rings.

To obtain the most accurate prediction score, you will use a kNN.

The Abalone Dataset is being imported.

You will work with the Abalone Dataset in this session. Although you could download it and use pandas to import the data into Python, letting pandas do it automatically is faster.

Installing Python using Anaconda is advised to follow along with the tutorial's code. There are several crucial packages for data science included in the Anaconda distribution. Check read Setting Up Python for Machine Learning on Windows for extra assistance with environment setup.

The following is how to use pandas to import the data:

Explanation:

This code uses pandas to read the data after initially importing it. To have the material downloaded directly from the Internet, you must give the path as a URL.

You may do a simple check like the one below to ensure that you've imported the data correctly:

This should display the first five lines of the Python-imported Abalone Dataset's pandas DataFrame. The column names still need to be added, as you can see. The abalone. The names file on the UCI machine learning repository contains those names. The following is how to include them in your DataFrame:

The imported data makes more sense. There is one more thing you ought to do, though: delete the Sex column. The objective of the current experiment is to estimate the age of the abalone using physical measures. You should take sex out of the dataset because it is not a strictly physical factor. Using. Drop, you may remove the Sex column:

With this code, the Sex column is removed because it adds nothing to the modeling process.

Statistical Analysis of the Abalone Dataset

You must know the data you use while working with machine learning. Here is a quick look at some exploratory statistics and graphs without getting into great detail.

Rings are the exercise's target variable, so you should start there. A histogram will provide a quick and helpful overview of the expected age ranges:

Using the panda's graphing capabilities, this code creates a histogram with fifteen bins. A few tests led to the choice to employ fifteen bins. When determining the number of bins, you typically aim to have neither too many nor too few observations per bin. A histogram can lack smoothness if there are too many or too few bins, which can mask some patterns. The following graph shows the histogram:

The histogram reveals that while up to twenty-five rings are feasible, most of the dataset's abalones have between five and fifteen rings. In this dataset, elder abalones are underrepresented. This makes sense since age distributions are typically skewed in this way owing to natural processes.

Finding out which factors, if any, show a high link with age is a second pertinent area of investigation. A positive indication that physical measures and age are connected would be a high correlation between an independent variable and your aim variable.

The entire correlation matrix is shown in correlation_matrix. The correlations with the target variable Rings are the most significant. These correlations can be obtained as follows:

Now consider Rings' correlation scores with the other factors. There is more of a link the closer they are near 1.

There is at least some association, if not a strong one, between the morphological characteristics of mature abalones and their age. Very high correlations indicate that the modeling approach will be simple. You'll need to experiment to discover what outcomes you can get from the kNN algorithm.

Using pandas opens up a wide range of additional opportunities for data investigation. Check Read Using Pandas and Python to Explore Your Dataset for more information about data exploration with pandas.

Python Tutorial: A Step-by-Step kNN From srcatch

You will learn the inner workings of the kNN algorithm in this lesson section. You must comprehend the algorithm's two primary mathematical elements. You'll begin with a straightforward kNN algorithm tour to warm you up.

Walkthrough of the kNN Algorithm in Simple English

Comparing the kNN algorithm to other machine learning methods, it is unusual. Each machine learning model has a unique formula to be estimated, as you have already seen. The k-Nearest Neighbors approach is unique in that this formula is calculated at the time of prediction rather than fitting. Most other models operate differently.

The kNN method, as the name suggests, begins by locating the closest neighbors of a new data point when it is introduced. The values of those neighbors are then used to anticipate the value of the new data point.

Consider your neighbors as a simple illustration of how this works. Frequently, your neighbors and you have similar interests. They most likely belong to the same socioeconomic group as you. They could work in the same field as you, send their kids to your school, etc. However, this method could be more effective for other tasks. For instance, it would be absurd to try to determine your preferred color by looking at that of your neighbor.

The kNN technique is founded on the idea that you may forecast a data point's attributes based on those of its neighbors. This prediction could work in certain circumstances but not in others. The mathematical definition of "nearest" for data points will be discussed next, along with techniques for combining several neighbors into a single forecast.

Explain the "Nearest" Distance Defined Mathematically

The Euclidean distance is a mathematical desrciption of distance that may be used to locate the data points most near the point you need to forecast.

It would help if you first comprehend what the difference between two vectors meant to arrive at this definition. Here's an illustration:

This image may have two data points: blue at (2,2) and green at (4,4). Let's start by combining two vectors to get the distance between them. From point (4,2) to point (4,4), vector a travels, and from point (4,2) to point (2,2), vector b. The colorful dots represent their heads. They are at a 90-degree angle, as you can see.

The vector c, which connects the head of vector a to the head of vector b, distinguishes these two vectors. The length of vector c represents the distance between your two data points.

The norm refers to a vector's length. The magnitude of the vector is indicated by the norm, which is a positive value. The Euclidean formula may be used to determine a vector's norm:

The square root of the sum of the squared differences in each dimension and the distance are used in this formula to calculate the distance. To determine the separation between the data points in this situation, you should compute the norm of the difference vector, C.

You must realize that your data points are vectors to apply this to your data. The distance between them may be calculated by determining the difference vector's norm.

Python may calculate this using the NumPy function lining. norm(). Here's an illustration:

You define your data points as vectors in this code block. The difference between the two data points is then used to compute the norm(). In this manner, the distance between two multidimensional locations is immediately ascertained. The distance between the points is still a scalar or a single number, even though they are multi-dimensional.

For additional information on mathematics, look at the Pythagorean theorem to see how the Euclidean distance formula is created.

Locate the k Proximate Neighbours

Now that you know how to calculate the distance between any two points, you can utilize that knowledge to locate the points closest to the one on which you wish to base your forecast.

The amount of neighbors you need to find is indicated by k. K must have a minimum value of 1. This entails making the forecast with just one neighbor. The amount of data points you have is the maximum. This entails utilizing every neighbor. The user decides what its value should be. You may do this using optimization tools, as you'll learn in the last section of this lesson.

Return to the Abalone Dataset and use NumPy to locate the closest neighbors. You need first convert your pandas DataFrame into a NumPy array using the. Values attribute because, as you have seen, you need to specify distances on the vectors of the independent variables.

This code block creates two objects, X and y, which now hold your data. Your model's independent variables are called X, and its dependent variables are called Y. A capital letter is used for X, whereas a lowercase letter is used for y. Because mathematical notation typically uses a capital letter for matrices and a lowercase letter for vectors, this is frequently done in machine learning algorithms.

Now, you may use a kNN with k = 3 on a fresh abalone with the physical measurements as follows:

The NumPy array for this data point may be created as follows:

The final step is to use the following code to calculate the distances between this new data point and each of the data points in the Abalone Dataset:

The three neighbors who are closest to your new_data_point are listed below. In the following paragraph, you'll learn how to translate those neighbors into an estimate.

Multiple Neighbors Voting or Averaging Their Votes

It would help if you combined the indices of your abalone's three closest neighbors to create a forecast for your new data point after determining the indices of the abalone's three closest neighbors.

It would help if you started by learning the reality of those three neighbors:

You can now make a prediction for your new data point based on the values of those three neighbors. For regression and classification, combining the neighbors into a prediction works differently.

Standard Regression Values

The target variable in regression issues is a number. You may integrate numerous neighbors' predictions into a single one by averaging several neighbors' target variable values. As an example, you could do this:

You'll receive a prediction score of 10. As a result, your new data point's 3-nearest Neighbor prediction is 10. The same would apply to whichever many fresh abalones you want.

Classification Method

The goal variable in classification issues is categorical. As was previously said, averages cannot be calculated for categorical variables. What would, for instance, be the average of the predicted three automobile manufacturers? To state that would be impossible. On the class predictions, an average cannot be used.

Instead, you use the mode when classifying things. The value that happens most frequently is the mode. This implies that you keep the most prevalent class after adding the classes of all your neighbors. The value that occurs among the neighbors the most frequently is the forecast.

There are several potential solutions if there are various modes. A final winner might be chosen at random from the other winners. The mode of the nearest neighbors would be kept if the final decision were based on the distances between the neighbors.

You may determine the mode by utilizing the SciPy mode() method. The code below demonstrates how to determine the mode for a toy example because the abalone example is not a case of classification:

As you can see, "B" is the value that appears the most frequently in the input data, making it the mode in this case.

Using Scikit-Learn, fit kNN in Python

It could be more practical to code an algorithm from srcatch while working on a machine-learning problem, even though it's wonderful for learning. In this part, you will examine how the kNN method is implemented in scikit-learn, one of the complete Python machine-learning libraries.

Creating Training and Test Sets from Data for Model Evaluation

You'll assess the effectiveness of your abalone kNN model in this part. You had a technical focus in the earlier portions, but your point of view will now be more pragmatic and results-oriented.

There are other methods for assessing models, but the train-test split is the most used one. To evaluate a model using a train-test split, divide the dataset into two sections as follows:

1. The model is fit using training data. This implies that the neighbors for the kNN will be drawn from the training set.
2. Test data are employed to assess the model. It implies that you will estimate each abalone's ring count in the test data and compare the findings to the actual number of existing rings.

Using the built-in train_test_split() function from scikit-learn, you may divide the data into training and test sets in Python:

The test_size desrcibes how many observations you want to include in the training and test sets. If you set the test_size parameter to 0.2, the test_size will equal 20% of the original data, leaving the remaining 80% as training data.

You may get the same results every time the code is executed by using the random_state argument. The random data split produced by train_test_split() makes it difficult to replicate the findings. Therefore, random_state is frequently used. Random_state's value selection is random.

As mentioned above, you divide the data into training and test data in the code. For an objective model evaluation, this is required. Now, you can use scikit-learn to fit a kNN model to the training set of data.

Making a kNN fit Scikit-Learn Regression to the Abalone Dataset

Making a model of the appropriate class is the first step in fitting a scikit-learn model. You must now select the values for your hyperparameters as well. You must select a number for k, known as n_neighbors, in the scikit-learn implementation for the kNN method. The Python code for doing this is as follows:

You create an unfitted model with knn_model. This model will use the three nearest neighbors to predict the value of a future data point. To get the data into the model, you can then fit the model on the training dataset:

Let the model learn from the data using the. fit() function. All the information required to generate predictions on fresh abalone data points is currently included in knn_model. You need that code to fit a kNN regression in Python.

Making use of Scikit-Learn to Examine Model Fit

But more than simply fitting a model is required. You'll look at a few functions in this section that you may use to assess the fit.

You'll utilize the root-mean-square error (RMSE), one of the most often-used assessment metrics for regression. A prediction's RMSE is calculated as follows:

1. Determine the variation between each data point's projected and actual value.
2. Calculate the square root of each difference.
3. Add up each difference that is squared.
4. Determine the value's square root.

On the training data, you may first assess the prediction error. As a result, you know that the outcome should be respectably positive as you are using the training data for prediction. To get the RMSE, use the following code:

1. Determine the variation between each data point's projected and actual value.
2. Calculate the square root of each difference.
3. Add up each difference that is squared.
4. Determine the value's square root.

On the training data, you may first assess the prediction error. As a result, you know that the outcome should be respectably positive as you are using the training data for prediction. To get the RMSE, use the following code:

Explanation

You assess the mistake in data that the model still needs access to in this code block. The RMSE is higher now that it is more realistic. You may read this as having, on average, a 1.65-year inaccuracy because the RMSE gauges the average error of the expected age. It depends on the circumstances if going from 2.37 to 1.65 years is a decent improvement. At least you're coming closer to determining the age accurately.

You have just up to now utilized the scikit-learn kNN method. Hyperparameter tweaking and K's random selection still need to be done. The RMSE on the training data and the RMSE on the test data differ considerably. This indicates that the model could have generalized better since it was overfitted to the training set of data.

Right now, there's no need to be concerned about this. In the next section, you'll learn how to use various tuning techniques to optimize the prediction or test error.

Plotting Your Model's Fit

Before updating the model, look at how well it fits the data. You may use Matplotlib to see how your predictions have been produced and what the model has learned:

The arrays X_test[:,0] and X_test[:,1] are subsets in this code block to produce a scatter plot of X_test's first and second columns of X_test. Length and diameter are the first two columns, as you recall before. As you can see from the correlations table, they are highly associated.

You tell c to create a colorbar using the expected values (test_preds). The size of the points in the scatter plot is determined by the argument s. To supply the cubehelix_palette color map, use cmap. Visit Python charting With Matplotlib to learn more about charting with this library.

With the code above, you'll obtain the graph shown below:

Each point on this graph represents an abalone from the test set, and the X- and Y-axes show the abalone's actual length and diameter. The point's color depicts the expected age. As you can see, an abalone's estimated age increases with length and size. This is a good indicator since it makes sense. It indicates that your model is picking up information that looks to be accurate.

By simply substituting the variable utilized for c, you can perform the same analysis for the actual numbers to determine whether this pattern is present in the raw data.

Output:

This code generates a scatterplot with a colorbar using Seaborn. It generates the graph shown below:

Explanation:

This demonstrates that the trend your model is learning is valid.

Each combination of the seven independent factors might yield a graphic. That would be too long for this guide, but feel free to try. The scatter's defined columns are the only thing that needs to be altered.

These visualizations show a seven-dimensional dataset in two dimensions. Playing around with them can help you understand what the model is learning and, maybe, what it is not learning or is incorrectly learning.

Using scikit-learn, Tune And Improve kNN in Python

Your predictive score may be raised in a variety of ways. Although data wrangling might be used to enhance the input data, the kNN technique is the main emphasis of this session. You'll then consider approaches to enhance the algorithm component of the modeling workflow.

Improving Scikit-Learn's kNN Performances GridSearchCV use

Until now, you've always used k=3 in the kNN method, but you must empirically determine the appropriate k value for each dataset.

When using fewer neighbors than when using more neighbors, your forecast will be significantly more variable:

• If you consider one neighbor, the forecast may significantly vary from one place to another. One of your neighbors may differ greatly from the rest when considering your neighborhood. Your 1-NN forecast would be off if an outlier were nearby.
• The influence of one significantly dissimilar neighbor will be substantially smaller if you have several data points.
• The forecast of each point runs the danger of being extremely near if there are too many neighbors. Let's assume you make a forecast using data from every neighbor. Every prediction would be the same in such a scenario.
• You will use a program called GridSearchCV to determine what value of k is ideal. This tool is frequently used for fine-tuning machine-learning model hyperparameters. It will assist you in your situation by automatically determining your dataset's ideal value of k.

Scikit-learn offers GridSearchCV, which has the advantage of being applied in the same manner as the models in Scikit-learn:

Here, you fit the model using GridSearchCV. In essence, GridSearchCV iteratively fits kNN regressors on a subset of the data and evaluates the results on the remaining subset. Repeating this will accurately estimate how well each value for k predicts the future. You test the values from 1 to 50 in this example.

Finally, it will save the k value that performs the best, which you may access using.best_params_:

Explanation:

The parameters with the lowest error score are printed in this code. You can see from.best_params_ that selecting 25 as the value for k will result in the best-predicted performance. You can observe how the optimal value influences your test and training results now that you know what it is:

Using this code, you may analyze the test data and fit the model to the training data set. You can see that while the test error is better than before, the training error has improved. This indicates that your model doesn't suit the training data as well. Overfitting on the training data has been made less of a concern by using GridSearchCV to determine what k should be.

Weighted Average of Neighbors Added Considering Distance

You decreased the test RMSE from 2.37 to 2.17 using GridSearchCV. You'll learn how to enhance the performances even further in this part.

Here, you'll test if your model will do any better when making predictions using a weighted average instead of a standard average. This implies that distant neighbors will have less of an impact on the forecast.

Setting the weights hyperparameter to the value of "distance" will do this. Setting this weighted average, however, could affect the ideal value of k. As a result, GridSearchCV will once again inform you of the appropriate type of averaging to use:

Here, you use your GridSearchCV to see if switching to a different weighting makes sense. The prediction error decreased from 2.17 to 2.1634 using a weighted average instead of a conventional average. Even if this isn't a significant improvement, it's still better, therefore it's worthwhile.

Improvements to the kNN in Scikit-Learn Using bags

You may use bagging as a third stage in the kNN tuning process. A relatively simple machine learning model is fitted to many different models using an ensemble approach called bagging, which modifies each fit slightly. Decision trees are frequently used in bagging. However, kNN works just as well.

Performance-wise, ensemble approaches frequently outperform individual models. One model could occasionally be off, but on average, a hundred models should be off less frequently. The predictions will be more accurate as the mistakes of several separate models are expected to average out.

The instructions below may be used to apply bagging to your kNN regression using scikit-learn. Create the KNeighborsRegressor first, using the ideal values for k and weights from GridSearchCV:

Then, using the bagged_knn model, construct a new instance of the BaggingRegressor class from scikit-learn with 100 estimators:

The predicted error using the bagged kNN is 2.1616, which is a little less than the error you previously discovered. Although running takes a bit longer, this is fine in this case.

Conclusion

You are now prepared to begin creating effective prediction models in Python after learning everything there is to know about the kNN algorithm. From a simple kNN model to a fully optimized model requires a few steps, but the performance boost is worthwhile!

You mastered the following skills in this tutorial: comprehending the mathematical underpinnings of the kNN algorithm, programming the kNN method from srcatch in NumPy, using the scikit-learn implementation to fit a kNN with the least amount of code, and using GridSearchCV to determine the optimal kNN hyperparameters. Utilize bagging to increase the performance of kNN.