It looks as if you are attempting to find the radius of the unique circle passing through the three points D, E, and F. If that is the case, the center of such a circle would be the intersection of the perpendicular bisectors of the line segment DE with that of EF. That would give you two linear equations in two unknowns, namely the two cartesian coordinates of the center, which matlab can easily solve. Once you have the center coordinates you can easily find the radius as the distance of this center to any of the three points.
As to the equation of a perpendicular bisector between two points, let the points be P1(x1,y1) and P2(x2,y2), and let P(x,y) be the coordinates of an arbitrary point along the bisector. The midpoint, M, between P1 and P2 is ((x1+x2)/2,(y1+y2)/2), and segment P1P2 must be orthogonal to segment PM. Therefore the dot product of the vectors P1P2 and PM must be zero, which gives the linear equation:
(x-(x1+x2)/2)*(x2-x1) + (y-(y1+y2)/2)*(y2-y1) = 0
Alternatively, you can also obtain an equivalent pair of linear equations by requiring the three distances from D, E, and F to the center to be equal and manipulating appropriately with the resulting equations.
