It is for building extraction from aerial images. After segmentation process boundaries are zigzag-shaped with many intrusions. I would like to simplify polygons but to maintain their essential shape. It means buildings should have orthogonal angles and small extrusions should be cut off. So I am looking for some implementation algorithm in MatLab. Like this: http://img812.imageshack.us/img812/3284/exampleus.png
sush: Maybe you have a file called edge.m on your path that is hiding the toolbox edge function. What is the output of the command "which edge"? If the result isn't in the Image Processing Toolbox, you could try renaming your edge.m file.