Documentation

matlab.perftest.TimeExperiment.limitingSamplingError

Class: matlab.perftest.TimeExperiment
Package: matlab.perftest

Construct time experiment for specified margin of error and confidence level

Syntax

``experiment = matlab.perftest.TimeExperiment.limitingSamplingError``
``experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value)``

Description

example

````experiment = matlab.perftest.TimeExperiment.limitingSamplingError` constructs a time experiment for each test suite element, with the specified statistical objectives (such as margin of error and confidence level). This method returns an instance of `FrequentistTimeExperiment`. This syntax uses the following defaults to determine the number of sample measurements. Number of warm-up measurements: 4Minimum number of samples: 4Maximum number of samples collected in the event other statistical objectives are not met: 256Objective relative margin of error for samples: 0.05 (5%)Confidence level for samples to be within relative margin of error: 0.95 (95%) ```
````experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value)` constructs a time experiment with additional options specified by one or more `Name,Value` pair arguments. Use this syntax to override the defaults listed above.```

Input Arguments

expand all

Name-Value Pair Arguments

Specify optional comma-separated pairs of `Name,Value` arguments. `Name` is the argument name and `Value` is the corresponding value. `Name` must appear inside quotes. You can specify several name and value pair arguments in any order as `Name1,Value1,...,NameN,ValueN`.

Example: ```experiment = matlab.perftest.TimeExperiment.limitingSamplingError('RelativeMarginOfError',0.12,'MaxSamples',100)``` creates a time experiment that collects sample measurements until samples have a relative margin of error of 12%, or until it collects 100 measurements.

Number of warm-up measurements, specified as a nonnegative integer. The value defines the number of times that the test framework exercises the test code to warm it up.

Minimum number of sample measurements, specified as a positive integer. The value defines the minimum number of times that the test framework exercises the test code after any warm-up runs. The test framework exercises the test code at least `MinSamples` times, regardless of whether the experiment meets the statistical objectives.

Maximum number of sample measurements, specified as a positive integer. The value defines the maximum number of times that the test framework exercises the test code after `NumWarmups`. If the experiment does not meet the statistical objectives, the testing framework collects up to `MaxSamples`.

Objective relative margin of error for samples, specified as a positive number.

The testing framework calculates the relative margin of error for a sample `X` using the equation

`$\text{relMoE}=\frac{\text{T}\ast \text{std(X)}}{\text{mean(X)}\ast \sqrt{\text{length(X)}}}$`

where `T` is the T-score from Student's T distribution using the specified `ConfidenceLevel` and `length(X)-1` degrees of freedom.

Confidence level for the samples to be within the relative margin of error, specified as a number from 0 through 1.

Examples

expand all

In your current working folder, create a class-based test, `preallocationTest.m`, that compares different methods of preallocation.

```classdef preallocationTest < matlab.perftest.TestCase methods(Test) function testOnes(testCase) x = ones(1,1e7); end function testIndexingWithVariable(testCase) id = 1:1e7; x(id) = 1; end function testIndexingOnLHS(testCase) x(1:1e7) = 1; end function testForLoop(testCase) for i=1:1e7 x(i) = 1; end end end end```

Create a test suite.

`suite = testsuite('preallocationTest');`

Construct a time experiment with a variable number of sample measurements, and run the tests.

```import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError; result = run(experiment,suite);```
```Running preallocationTest .......... .......... .......... .. Done preallocationTest __________ ```

View the test activity for the first test. Your results might vary.

`result(1).TestActivity`
```ans = 8×12 table Name Passed Failed Incomplete MeasuredTime Objective Timestamp Host Platform Version TestResult RunIdentifier __________________________ ______ ______ __________ ____________ _________ ____________________ ___________ ________ _____________________ ________________________________ ____________________________________ preallocationTest/testOnes true false false 0.056052 warmup 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.056227 warmup 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.055969 warmup 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.054961 warmup 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.052572 sample 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.051743 sample 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.051709 sample 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 preallocationTest/testOnes true false false 0.051256 sample 05-Oct-2018 10:14:15 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 34fa8e1d-e21f-42b5-83bd-fd104ffcec12 ```

For this test, the performance testing framework collected 4 warm-up measurements (the default), and 11 sample measurements. After 11 sample measurements, the performance testing framework satisfied the default statistical objectives.

Construct a time experiment that collects two warm-up measurements and runs the tests a variable number of times to reach a sample mean with a 10% relative margin of error within a 90% confidence level.

```experiment = TimeExperiment.limitingSamplingError('NumWarmups',2,... 'RelativeMarginOfError',0.10, 'ConfidenceLevel', 0.90); result = run(experiment, suite);```
```Running preallocationTest .......... .......... .... Done preallocationTest __________ ```

View the test activity for the first test. Your results might vary.

`result(1).TestActivity`
```ans = 6×12 table Name Passed Failed Incomplete MeasuredTime Objective Timestamp Host Platform Version TestResult RunIdentifier __________________________ ______ ______ __________ ____________ _________ ____________________ ___________ ________ _____________________ ________________________________ ____________________________________ preallocationTest/testOnes true false false 0.053963 warmup 05-Oct-2018 10:21:31 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 preallocationTest/testOnes true false false 0.053086 warmup 05-Oct-2018 10:21:31 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 preallocationTest/testOnes true false false 0.052502 sample 05-Oct-2018 10:21:31 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 preallocationTest/testOnes true false false 0.05252 sample 05-Oct-2018 10:21:31 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 preallocationTest/testOnes true false false 0.052048 sample 05-Oct-2018 10:21:32 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 preallocationTest/testOnes true false false 0.052434 sample 05-Oct-2018 10:21:32 MY-HOSTNAME win64 9.6.0.966561 (R2019a) [1x1 matlab.unittest.TestResult] 22e4507c-e12c-4cac-8730-aff65e75a2e1 ```

For this test, the performance testing framework collected two warm-up measurements and nine sample measurements. After nine sample measurements, the performance testing framework satisfied the specified statistical objectives.