MATLAB Answers

Identifying GUI objects from UI Prototype

1 view (last 30 days)
Manoj Bandara
Manoj Bandara on 9 Jun 2011
Hi,
I suppose to identify UI objects such as buttons,combo boxes, text boxes from given UI prototype(Image).Can anyone suggest me approach regarding this? I am planning to identify UI objects as geometric shapes such as circles(option buttons), rectangles(text boxes) etc and I also need to identify (x,y)locations of these objects

Answers (1)

Brian Cody
Brian Cody on 10 Jun 2011
Before diving into solutions, I'm curious about what problem you're trying to solve. Are you hoping to generate actual GUI code from a mock-up image? Is this more of an academic exercise? Something else entirely?
I have done something like this in the past for the purposes of finding rendering differences between two images. To accomplish that task, we used connected components to break down the image into segments onto which we could apply heuristics about what different GUI components should look like. That allowed us to ignore a set of allowable differences while pointing out any other per-pixel differences as well as higher-level layout issues.
Another approach could be to try to define templates for what these different GUI objects should look like (with some parameterization to allow for sizing, etc.), and then searching for instances within your image that adhere to these template definitions.
But here are some considerations to take into account. How well can you describe the GUI elements in your input images? For instance, if they all come from one mock-up program, then you may have a small set for which to search. However, if you are using screen shots from different platforms, from different browsers on the same platform, or from the same platform with different display settings, you will find differences ranging from subtle to dramatic. So limiting your input will be critical.
If you use connected components you will probably find that either you need to look for relationships between different connected components (because they make up part of the same GUI object) or you need to control how the connected component algorithm connects sets of pixels to make sure that you end up with the component set that you want. I bet that you will find a lot of ambiguity here. For instance, perhaps in one GUI object you want two shades of gray to be considered part of the same object, but in another situation you want those two grays to be kept in separate components.
  1 Comment
Manoj Bandara
Manoj Bandara on 10 Jun 2011
Thanks brain for your reply. Really this is an academic exercise to generating actual code from UI mock-ups(eg: UI mock-ups to java swing). Normally when developing software system, we first design UI mock-ups and then separate team working for converting these UI mock-ups to actual coding. Goal is to minimize UI engineering cost.
I am really interested second method that you suggested,
"Another approach could be to try to define templates for what these different GUI objects should look like (with some parameterization to allow for sizing, etc.), and then searching for instances within your image that adhere to these template definitions"
Can you please share some code snippets that direct me?
All UI mock-up design is done by using same prototyping tool and it need not have platform independence.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!