MATLAB tailored engineering Benchmarks for AI Models?
Hello everyone,
Today, we have a wide range of models available for problem-solving, education, and agentic coding. Picking a model for a task itself is a skill worth learning and exploring. We have a bouquet of closed models from Anthropic, OpenAI, Grok (Cursor), Mistral, and Gemini, as well as open-source models like Kimi and Gemma.
I am wondering whether, as a MathWorks community, we can come up with some benchmarks tailored for MATLAB + AI models by choosing and compiling tasks like plotting (graphing), some matrix problems, signal processing tasks, control system problems and other engineering problems, and then test AI model with Matlab-MCP server and document performance.
Reaching a consensus would really help us choose a certain model for certain tasks. That way, we can save API tokens, be smart with our choices and as a community, we can learn and help by sharing our experiences. I would love to hear your thoughts and see what this conversation could lead to.
@Hans Scharler do you have any initial thoughts to share with us?
7 Comments
Time Descending
The number of models is sure ballooning. See https://www.mindstudio.ai/models . And it seems everyone but Anthropic is leaning towards a mixture of models model, using inexpensive ones for specialized tasks and one could be an open model that just codes (and verifies?) MATLAB with an option to restrict to Opera and test in its container if the user does not have MATLAB hooked up via MCP.
Great ideas!
The more we share here, the more people will find best practices, workflows, skills, etc. to make things more streamlined. I am still learning how to best approach complex tasks.
Thank you for sharing your thoughts. It would be great if Mathworks could organize a community event or call to discuss paths or approches to comeup with model benchmarks for agentic workflows in Matlab.
This is a great idea. I agree that picking a model for a task itself is a skill worth learning and exploring. This should be a community effort and we can start sharing what we have experiemented so far. You mentioned open-weight models. Have you tried some?
Hi, thank you for sharing your thoughts and I agree that this should be a community effort. I tried Mistral Medium model couple of times and it does a good job. I still have to try and experiment with other closed and open weights models.
This is a great idea. I am currently trying out various local models on my personal Mac and I would love somethng like that. I am thinking of something similar to this? https://huggingface.co/blog/pratikbhavsar/agent-leaderboard-v2
MATLAB Answers may be a good way to create an automated eval system. Present the questions to a given model, withhold the solutions, and how closely can they produce answers accepted by humans?
Thank you so much for sharing your thoughts and for the hugging face link. Yes, if we can come up with something like that, it would be great.
I am curious to know your toughts on "MATLAB Answers for automated eval system". Could you tell us how it works?
Sign in to participate
Tags
