For both images separately are calculated:
1) Least-squares norm surface.
2) Local maxima values uniformly distributed over the norm surface.
3) Small involving windows for all local maxima points. The windows are rotated so gradient vector points to (1,0), and normalized subtracting mean and dividing by standard deviation.
The result of this step are two arrays of tie points xy, XY.
Find closest XY point to each from xy.
(Subtracting each XY from each xy and getting closest)
Find the transformation using geometry of tie points.
Cull points, which doesn't match to general transformation.
Idea: Model transformation using random pairs of tie points, find how much pairs fit to this transformation, supports the transformation, using some threshold parameter. Model the transformation for several pairs (all for best results). Use the transformation with better support. Cull non-fitted points.
*** After this step we have a transformation! Now let's find out how good is it?
Model random transformations to find the bad fit parameters.
Use the same image twice to find a good transformation fit parameters.
Test the bad and the good fit parameters against fit parameter to find if the transformation is good or not.