One way to determine whether a point lies inside a polygon is to add up the angles that it makes with the polygon's vertices. If the test point is P and A and B are two adjacent polygon points, then you look at the angle ∠APB.
If you add up all of the angles between the test point and each of the polygon's edges, the result will be either 0, 2π, or –2π. If the total is 2π or –2π, then the point is inside the polygon. If the total is 0, then the point lies outside of the polygon. You can probably convince yourself that this works if you draw a few examples. For example, try placing points inside and outside of a square, draw the angles, and estimate their values.
The idea is straightforward. The hard part is calculating ...
