Matlab Grader: Complex and conditioned assessments for alternative solutions, differentiable error messages and figures

During the preparation of my courses I came along the following problems:

1. Allow alternative solutions
2. Specific check of reoccurring student errors, with precise hints
3. Check plots

which seem to be undiscussed until know. So here are some more specific examples including the solution (hope it may help you):

1)

These is locked code in the Learner Template, defining some important physical values:

% Definition of physical constants
c = 299792458; % speed of light [m/s]
h = 6.626070040e-34; % Planck's constant [Js]
e = 1.6021766208e-19; % elementary charge [C]
% Definition of units
eV = 1.6021766208e-19; % unit conversion: eV to J

The student is first requested to provide the following lines, which he has to be extract from a formulary:

% Define the photon energy E
E = [8047.8227, 8027.8416] % transition energies [eV]

Afterwards he has to calculate the wavelength in angstrom, the Reference Solution to this problem would be:

% Calculate the wavelength Lambda
Lambda = h*c./(E*eV); % wavelength [m]
Lambda = Lambda*1e10 % wavelength [A]

The two standard tests would be:

assessVariableEqual('E',referenceVariables.E)
assessVariableEqual('Lambda',referenceVariables.Lambda)

However, a lot of students provided the following solution:

% Calculate the wavelength Lambda
E = E*eV;
Lambda = h*c./E; % wavelength [m]
Lambda = Lambda*1e10 % wavelength [A]

The solution is completely correct, but the error which the first test returns since the E is redefined made the students insecure and animated them to provide further solutions which were incorrect. Thus, I replaced the first test by:

% Get reference variable
E_ref = referenceVariables.E;
eV_ref = referenceVariables.eV;
% Submitted variables can be addressed normally as long as the student defined it otherwise an error will be displayed similar to variable not declared.
 
% Conditioned comparison between the reference solution and the submitted solution
if ~exist('E','var')
    error('The submission must contain a variable named E.')
elseif E==E_ref*eV_ref % allow for the alternative definition in Joule
    assessVariableEqual('E', E); % this is a dummy test it is always true because the studends variable is compared to itself
else % check if the variable is defined in the actual requested unit electron volt
    assessVariableEqual('E', E_ref,'Feedback','If the value is incorrect check if you have selected the experimental transition energy, if you have four decimal places and if it is specified in electron volt (avoid to redefine it, exception is the unit conversion to Joule).');
end

2)

Check the example above also a lot of students calculated the wavelength as follows and did not recognized were the fault was:

% Calculate the wavelength Lambda
E = E*eV;
Lambda = h*c./E; % wavelength [m]

Thus, I replaced the second test by test which tells them if the result is completely incorrect or if they just forgot the conversion to angstrom:

% Get reference solution for x.
Lambda_ref = referenceVariables.Lambda;
% Conditioned comparison between the reference solution and the submitted solution
if ~exist('Lambda','var')
    error('The submission must contain a variable named Lambda.')
elseif Lambda==Lambda_ref*1e-10;
    error('You have calculated Lambda in meter, now convert it to angstrom to get the correct solution!')
else % check if the variable is defined in the actual requested unit electron volt
    assessVariableEqual('Lambda', Lambda_ref,'Feedback','If you passed the first test, than you entered the equation wrong or your resulting unit is something like "rainbows per unicorn" ;) Check for correct unit conversion. The wavelength is requested in angstrom.');
end

3)

The students have to load some data into the variable Voltage and plot it subsequently. The Reference Solution is (note: the line containing Fig_Voltage is locked in the Learner Template):

% Import the measured data from the file "tube_voltage.txt" to the variable Voltage
Voltage = importdata('tube_voltage.txt') % no semicolon to print structure elements
% Open figure (the assignment of the new figure to the variable Fig_Voltage is only necessary for the automated tests)
Fig_Voltage = figure(1);
% Plotting
hold on
    % Plot the data using connected markers
    plot(Voltage.data(:,1),Voltage.data(:,2),'-s')
    % Plot the axes labels
    xlabel('Voltage [kV]')
    ylabel('Intensity [cps]')
    % Plot the title
    title('Current: 20 mA')
hold off

If we apply the standard tests we can check if Voltage is correct and if the functions like plot, xlabel, etc. were used. But some students provided a solution which was wrongly identified as correct using this basic assessment:

Voltage = importdata('tube_voltage.txt') % no semicolon to print structure elements
% Open figure (the assignment of the new figure to the variable Fig_Voltage is only necessary for the automated tests)
Fig_Voltage = figure(1);
% Plotting
hold on
    % Plot the data using connected markers
    plot(Voltage.data(1,:),Voltage.data(2,:))
    % Plot the axes labels
    xlabel('Voltage [kV]')
    ylabel('Intensity [cps]')
    % Plot the title
    title('Current: 20 mA')
hold off

Thus, I used the following complex tests for checking the plotted data and the usage of connected plot markers:

Test 1: Data check

% Do not extract the values from the reference plot, as it returns a handle to a deleted figure
% Define the data which should be plotted, as the data is checked in the first test we just check if the students data is plotted correctly
% Nice side effect even if the data is compromised the student is not punished for a cascading failure
PlotData_ref = Voltage.data';
 
% Check if the correct data is plotted
Axes = findall(Fig_Voltage,'type','axes');
if size(Axes)~=[1 1]
    error('You assigned none or more than one axis to the figure Fig_Voltage!');
else
    Plots = Axes.Children
    if size(Plots)~=[1 1];
        error('You assigned none or more than one plot to the figure Fig_Voltage!');
    else
        PlotData = [Plots.XData; Plots.YData];
        assessVariableEqual('PlotData',PlotData_ref,'Feedback','If the value is incorrect, you may have switched the x and y axis or you are using the current instead of the voltage.')
    end
end

Test 2: Plot markers

Axes = findall(Fig_Voltage,'type','axes');
if size(Axes)~=[1 1]
    error('You assigned none or more than one axis to the figure Fig_Voltage!');
else
    Plots = Axes.Children;
    if size(Plots)~=[1 1]
       error('You assigned none or more than one plot to the figure Fig_Voltage!');
    else
        if ~strcmp(Plots.Marker,'none') && contains(Plots.LineStyle,'-')
            assessVariableEqual('Fig_Voltage', Fig_Voltage); % this is a dummy test it is always true because the studends variable is compared to itself
        else
            error('Either you did not connected your markers or used none.');
        end
    end
end

Tags: matlab, grader, code, test, assessment, complex, conditioned, alternative solutions, differentiable errors, figure, plot, submission, reference solution, assessVariableEqual, learner template, feedback, if, referenceVariable

Answer 1

[Martin Rudolph]

Answer is provided beside the question (see above) ! But check also the other answers below, some clever lightweight alternatives are described there.

Answer 2

[Cris LaPierre]

I like that you are not forcing students to solve the problem exactly the same way you did. There are often multiple ways of obtaining the correct solution, and students should not be penalized for using a different yet correct approach.

How you implement this can come down to personal preference. I do think your approach for grading E and Lambda can be simplified by taking advantage of MATLAB Grader.

1. To grade E, use a try-catch. The function assessVariableEqual already checks that the specified variable exists, is the expected data type, and the same size as the reference variable before comparing the values. If any of these are not the same, default error messaging will provide the appropriate feedback to the student.
2. Feedback is only shown when the solution is incorrect. You can format the feedback if you use the Feedback field in the assessment test.
3. Only show feedback for the first test that is incorrect by checking the box indicated in the image below. This allows you to write feedback assuming all previous tests are correct. This simplifies the feedback messaging for Lambda.

4. To grade Lambda, use the default test type and provide your custom feedback in the feedback field.

Here is how this approach behaves for a student who included both common differences.

And finally, here is a screenshot of how the feedback looks when both assessment tests are incorrect. Note the default error messaging about E, and that the feedback about Lambda is hidden, thereby encouraging students to focus on getting E correct first.

[Martin Rudolph]

Dear Cris,

thank you very much. The try-catch statement is indeed a very comfortable solution.

Best regards

Martin

[Cris LaPierre]

I'm trying to recreate the issue you were seeing with the student solution to the plotting, but can't recreate the same problem. Reversing X and Y does not create the plot you showed, and using the plot command you shared creates a line with negative slope. Does the text file contain data in addition to what is plotted?

[Martin Rudolph]

This seems a misunderstanding probably due to the short description of the errors. There are two frequent errors I observed when working with the students before using Grader. I only showed one. So here are both errors:

% Error one: switched x and y axes
plot(Voltage.data(:,2),Voltage.data(:,1))
% Error two switched rows and columns of the matrix
plot(Voltage.data(1,:),Voltage.data(2,:))

I showed above only error two because you directly see that something is completly wrong. There is no specific feedback for this error as assessVariableEqual returns for this error that the dimensions do not match. Whereas the first error returns, that only the value is incorrect. I did not check if students use something like set(gca,'Ydir','reverse'). This is usally a command the students do not know. Furthermore, this error would be no typo one would expect the implementation of this fault on purpose.

P.S. I provided with this answer also the data file.

[Cris LaPierre]

Ah, I had created the data in the reverse order.

Our general recommendation is to design your problems in such a way that you do not have to grade plots in MATLAB Grader. As you no doubt have experienced, it can be difficult to define what "correct" means, and then challenging to create a robust assessment.

In addition, there are often multiple ways to create a plot that look correct. However, these plots may have different properties that cause it to be marked incorrect.

You have managed to create the tests you wanted, so no need for me to recreate it. I would just provide the following recommendation. Rather than extracting properties for labels, titles, etc. and then performing text comparisons, just add an assessment test to require their use in the solution. Chances are, if they have to use them, they will use them as intended.

It may also be helpful to include an image of the correct plot in the feedback so students can see what their figure should look like.

Continuing the idea of not forcing students to solve the solution a certain way, might I recommend one minor change to your code? It is not necessary to provide the following as a locked line in the learner template.

Fig_Voltage = figure(1);

Instead, update your assessment code to use findobj instead of findall. Find the figure first, then axes on the figure, then lines on the axes.

Also, hold on and hold off are unnecessary for this plot. Removing them further simplifies the code.

[Martin Rudolph]

Thanks for the hints. I provided solutions containing also the figures and other information on our eductional platform so that the students can check there plots and solutions. Furthermore, I check always the student solutions and update tests for alternative solutions. For a lot of such problems your suggested try-statement is a really nice and easy solution. However, there remains one large problem which consumes a lot of time. Since I wanted the tests to allow for self studing with feedback I allowed the students for infinite submissions. But checking the solutions gets messy, some students need 10 up to 20 tries. The order is not always correct, sometimes newer solutions are not at the top:

Furthermore, I do not want to check always all submissions. Entering into the filter "correct" gives only the correct solutions of all students:

However, I need something like "latest", where only the last sumission of each student is visible in a list. So, that I only have to check 30 files instead of 300 or more. Because using the filter "correct" does not allow me to see the solutions of students with issues but for checking who has passed all tests this is nice.

[Cris LaPierre]

Thank you for the detailed feedback. I have passed it on internally.

[Martin Rudolph]

Thanks, please consider also to add a response to student button, were I can directly answer them with regard to the selected submission.

[Cris LaPierre]

I wanted to report back that the issue with solution order has been addressed. Please check if you are still seeing the issue. If you are, Please let MathWorks Customer Service know by reporting the issue here:

http://www.mathworks.com/support/servicerequests/create.html