In this exercise, we will compare kNN to neural networks where we have a small dataset. We will be using the iris dataset imported from the scikit-learn library.
To begin, we will first discuss the basics of kNN. The kNN classifier is a nonparametric classifier that simply stores the training data, D, and classifies each new instance using a majority vote over its set of k nearest neighbors, computed using any distance function. For a kNN, we need to choose the distance function, d, and the number of neighbors, k: