Thread Subject:

how to implement the "many tiny functions" paradigm?

One of the most often cited coding "best practices" is to avoid
structuring the code as one "large" function (or a few large
functions), and instead to break it up into many tiny functions.
One rationale for this strategy is that tiny functions are much
easier to test and debug.

The question is how to implement this "many tiny functions" strategy
in MATLAB.

To be more specific, suppose that I start with some "large" function
foo, implemented in a ~100-line file foo.m, and I break it down
into about ~20 tiny functions, including a new (and much smaller)
entry-point function foo that calls the remaining tiny functions,
either directly or indirectly.

The problem is that, if the new foo.m file now looks like this:


    function f = foo(a, b, c)
      ...

    % End of function foo

    function b = bar(d, e)
      ...

    % End of function bar

    ...

    function z = zzz(x, y)
      ...

    % End of function zzz

...then the "helper" functions bar, ..., zzz are not accessible from
outside foo.m, so there's no easy way to test or debug them
*directly*. (One can do so only indirectly through calls to the
entry-point function foo, and this can be quite cumbersome and
error-prone.)

(Note that it would be entirely consistent with the philosophy
described above to have, among the "tiny functions" in foo.m, some
--say, test_bar, ..., test_zzz, etc.--, whose sole purpose was to
test some other "tiny function". Even in this case, however, one
would need a way to call these testing functions from outside of
foo.m.)

So file structure shown above may not be very useful for testing
and debugging. On the other hand, it would be too much clutter
(both of directories and of the global namespace) to put all these
tiny functions each in its own *.m file (bar.m, ..., zzz.m,
test_bar.m, ..., test_zzz.m).

So my question is: is there a convenient way to structure this code
that would *both* preserve the overall decomposition into many tiny
functions *and* enable these functions to be tested?

Suggestions would be appreciated.

Thanks in advance!

PS. I am aware of the existence of MATLAB xUnit Test Framework,
etc., but my aims here is more general and fundamental, and in any
case decidedly different, from those of unit testing, and therefore
if possible I'd much prefer to avoid bringing into the picture the
whole unit testing apparatus and mind-set. I'm sure that the
question of the suitability of a unit-testing framework for what
I want to do could provide fodder for lengthy discussions, but this
is a somewhat philosophical point that I'd prefer to avoid altogether.
I hope that responders will be willing to take it on faith my
assertion that MATLAB xUnit Test Framework et al. are not what I'm
after at the moment.

On 1/9/2013 2:15 PM, kj wrote:
> One of the most often cited coding "best practices" is to avoid
> structuring the code as one "large" function (or a few large
> functions), and instead to break it up into many tiny functions.
> One rationale for this strategy is that tiny functions are much
> easier to test and debug.
>
> The question is how to implement this "many tiny functions" strategy
> in MATLAB.
...
[BUT]
...
> ...then the "helper" functions bar, ..., zzz are not accessible from
> outside foo.m, so there's no easy way to test or debug them
> *directly*. (One can do so only indirectly through calls to the
> entry-point function foo, and this can be quite cumbersome and
> error-prone.)
...
> So file structure shown above may not be very useful for testing
> and debugging. On the other hand, it would be too much clutter
> (both of directories and of the global namespace) to put all these
> tiny functions each in its own *.m file (bar.m, ..., zzz.m,
> test_bar.m, ..., test_zzz.m).
>
> So my question is: is there a convenient way to structure this code
> that would *both* preserve the overall decomposition into many tiny
> functions *and* enable these functions to be tested?
>
> Suggestions would be appreciated.
...

I don't know of any way to have both -- what I've done in the past some
(altho not religiously and I won't claim it's any great revelation) is
to write/test the helpers independently then put them into the larger
container file when they're vetted.

This (somewhat) follows the Forth idea of writing words from bottom up
but Forth doesn't have the same visibility/scoping within a file as does
Matlab unless one builds it w/ dictionaries.

It is, unfortunately, sorta' a remnant of Matlab having "just growed"
from the initial concept and not sure there's any solution w/o a major
restructuring that's probably not likely (any time _real_soon_now_ (tm)
anyway).

--

kj <no.email@please.post> wrote in message <kckj4h$m4s$1@reader1.panix.com>...
>
>
> One of the most often cited coding "best practices" is to avoid
> structuring the code as one "large" function (or a few large
> functions), and instead to break it up into many tiny functions.
> One rationale for this strategy is that tiny functions are much
> easier to test and debug.
>
> The question is how to implement this "many tiny functions" strategy
> in MATLAB.
>
> To be more specific, suppose that I start with some "large" function
> foo, implemented in a ~100-line file foo.m, and I break it down
> into about ~20 tiny functions, including a new (and much smaller)
> entry-point function foo that calls the remaining tiny functions,
> either directly or indirectly.
>
> The problem is that, if the new foo.m file now looks like this:
>
>
> function f = foo(a, b, c)
> ...
>
> % End of function foo
>
> function b = bar(d, e)
> ...
>
> % End of function bar
>
> ...
>
> function z = zzz(x, y)
> ...
>
> % End of function zzz
>
> ...then the "helper" functions bar, ..., zzz are not accessible from
> outside foo.m, so there's no easy way to test or debug them
> *directly*. (One can do so only indirectly through calls to the
> entry-point function foo, and this can be quite cumbersome and
> error-prone.)
>
> (Note that it would be entirely consistent with the philosophy
> described above to have, among the "tiny functions" in foo.m, some
> --say, test_bar, ..., test_zzz, etc.--, whose sole purpose was to
> test some other "tiny function". Even in this case, however, one
> would need a way to call these testing functions from outside of
> foo.m.)
>
> So file structure shown above may not be very useful for testing
> and debugging. On the other hand, it would be too much clutter
> (both of directories and of the global namespace) to put all these
> tiny functions each in its own *.m file (bar.m, ..., zzz.m,
> test_bar.m, ..., test_zzz.m).
>
> So my question is: is there a convenient way to structure this code
> that would *both* preserve the overall decomposition into many tiny
> functions *and* enable these functions to be tested?
>
> Suggestions would be appreciated.
>
> Thanks in advance!
>
> PS. I am aware of the existence of MATLAB xUnit Test Framework,
> etc., but my aims here is more general and fundamental, and in any
> case decidedly different, from those of unit testing, and therefore
> if possible I'd much prefer to avoid bringing into the picture the
> whole unit testing apparatus and mind-set. I'm sure that the
> question of the suitability of a unit-testing framework for what
> I want to do could provide fodder for lengthy discussions, but this
> is a somewhat philosophical point that I'd prefer to avoid altogether.
> I hope that responders will be willing to take it on faith my
> assertion that MATLAB xUnit Test Framework et al. are not what I'm
> after at the moment.

Two things come into my mind.

First, if you want to adopt the good practice of writing small functions, do it at the initial stage when you create the code. Put your immediate thoughts into lines of the code. That way, you create small functions, one by one, which accomplish the task at hand. If the code already exists, you may break it into a few functions if it comes in handy, but there is no point in setting a goal of slicing a 100-line code into 20 tiny ones. No need to do it on purpose.

Second, why are the newly created functions not accessible from outside the original foo.m? What prevents you from saving them in separate files? You are going to debug each one of them anyway…

kj <no.email@please.post> wrote in message <kckj4h$m4s$1@reader1.panix.com>...
>
>
> One of the most often cited coding "best practices" is to avoid
> structuring the code as one "large" function (or a few large
> functions), and instead to break it up into many tiny functions.
> One rationale for this strategy is that tiny functions are much
> easier to test and debug.

I would be careful when applying blindly this "best practice". MATLAB have huge running performance penalty due to function overhead (as opposed to other programming language as C, fortran, ...)

You will slow down the code by putting simple (scalar) arithmetic expressions in function. If the expression is vectorized and supposes to operate on big array, then it's OK to put it in the functions.

Bruno

In <kckj4h$m4s$1@reader1.panix.com> kj <no.email@please.post> writes:

>The question is how to implement this "many tiny functions" strategy
>in MATLAB.

>To be more specific, suppose that I start with some "large" function
>foo, implemented in a ~100-line file foo.m, and I break it down
>into about ~20 tiny functions, including a new (and much smaller)
>entry-point function foo that calls the remaining tiny functions,
>either directly or indirectly.

>The problem is that, if the new foo.m file now looks like this:

> function f = foo(a, b, c)
> ...

> % End of function foo

> function b = bar(d, e)
> ...

> % End of function bar

> ...

> function z = zzz(x, y)
> ...

> % End of function zzz


OK, one possibility (still rather imperfect) is to replace the
above with something like this:

    classdef foo
        methods (Static)
            function f = run(a, b, c)
                % the function formerly known as foo;
                % it calls one or more of foo.bar, ..., foo.zzz
                ...
            end

            function b = bar(d, e)
                ...
            end
            ...

            function z = zzz(x, y)
                ...
            end
        end
    end

Now all the functions are accessible from other files, and yet they
do not pollute the global namespace.

The one drawback I see is that, instead of calling foo(a, b, c),
one must call foo.run(a, b, c).

Are there other drawbacks I'm not seeing?

In <kclpkc$65b$1@newscl01ah.mathworks.com> "Bruno Luong" <b.luong@fogale.findmycountry> writes:

>kj <no.email@please.post> wrote in message <kckj4h$m4s$1@reader1.panix.com>...
>>
>>
>> One of the most often cited coding "best practices" is to avoid
>> structuring the code as one "large" function (or a few large
>> functions), and instead to break it up into many tiny functions.
>> One rationale for this strategy is that tiny functions are much
>> easier to test and debug.

>I would be careful when applying blindly this "best practice". MATLAB have huge running performance penalty due to function overhead (as opposed to other programming language as C, fortran, ...)

>You will slow down the code by putting simple (scalar) arithmetic expressions in function. If the expression is vectorized and supposes to operate on big array, then it's OK to put it in the functions.


To be sure, the benefits of using many tiny functions over a few
big ones need to be weighed against the performance costs of the
additional function calls. This is always true.

In practice, however, I've found it is possible to narrow down
significantly the performance-critical parts of the code, outside
of which, the added cost of the additional function costs becomes
negligible. This usually creates a fairly large scope of applicability
for the "tiny functions" strategy. (I.e., a 100-line function may
be boiled down to ~10-20 2-3 line functions, and one 15-line
performance-critical computational "core".

And even when performance argues against breaking down a function
any further, my practice has been to go ahead and code the
slow-but-testable implementation, structured around tiny functions,
and use this slow version to check the correctness of the fast,
production version of the code. So even when performance is a
consideration I find the tiny-functions paradigm extremely useful.

"kj" <no.email@please.post> wrote in message
news:kcn2qd$ld5$2@reader1.panix.com...
> In <kclpkc$65b$1@newscl01ah.mathworks.com> "Bruno Luong"
> <b.luong@fogale.findmycountry> writes:
>
>>kj <no.email@please.post> wrote in message
>><kckj4h$m4s$1@reader1.panix.com>...
>>>
>>>
>>> One of the most often cited coding "best practices" is to avoid
>>> structuring the code as one "large" function (or a few large
>>> functions), and instead to break it up into many tiny functions.
>>> One rationale for this strategy is that tiny functions are much
>>> easier to test and debug.
>
>>I would be careful when applying blindly this "best practice". MATLAB have
>>huge running performance penalty due to function overhead (as opposed to
>>other programming language as C, fortran, ...)
>
>>You will slow down the code by putting simple (scalar) arithmetic
>>expressions in function. If the expression is vectorized and supposes to
>>operate on big array, then it's OK to put it in the functions.
>
>
> To be sure, the benefits of using many tiny functions over a few
> big ones need to be weighed against the performance costs of the
> additional function calls. This is always true.
>
> In practice, however, I've found it is possible to narrow down
> significantly the performance-critical parts of the code, outside
> of which, the added cost of the additional function costs becomes
> negligible. This usually creates a fairly large scope of applicability
> for the "tiny functions" strategy. (I.e., a 100-line function may
> be boiled down to ~10-20 2-3 line functions, and one 15-line
> performance-critical computational "core".

Depending on what you're doing, there are three possibilities that you may
want to explore.

If all these "tiny functions" are only used by your application, and your
tests will live in the same directory, make your helpers private functions.

http://www.mathworks.com/help/matlab/matlab_prog/private-functions.html

If they are used by many files in different directories, or if your tests
must live in a different directory, consider putting your helpers in a
package directory.

http://www.mathworks.com/help/matlab/matlab_oop/scoping-classes-with-packages.html

While this documentation is in the object-oriented section of the
documentation, packages are not limited to use only with MATLAB classdef
files. You can store plain functions in a package and use them as such.

A third, somewhat "hacky" way to test many small subfunctions in your main
file is to have a syntax that causes the main function to return a struct
array whose fields contain function handles to the subfunctions. Your tests
can then invoke the subfunctions using those function handles even though
the subfunctions are not in scope. As long as the subfunction was in scope
when the function handle was _created_ it doesn't matter if the subfunction
is out of scope when the function handle is _executed._

http://www.mathworks.com/help/matlab/matlab_prog/applications-of-function-handles.html#brf_1tn-1

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

In <kcn6p7$m1r$1@newscl01ah.mathworks.com> "Steven_Lord" <slord@mathworks.com> writes:

>Depending on what you're doing, there are three possibilities that you may
>want to explore.

>If all these "tiny functions" are only used by your application, and your
>tests will live in the same directory, make your helpers private functions.

>http://www.mathworks.com/help/matlab/matlab_prog/private-functions.html

>If they are used by many files in different directories, or if your tests
>must live in a different directory, consider putting your helpers in a
>package directory.

>http://www.mathworks.com/help/matlab/matlab_oop/scoping-classes-with-packages.html

>While this documentation is in the object-oriented section of the
>documentation, packages are not limited to use only with MATLAB classdef
>files. You can store plain functions in a package and use them as such.

>A third, somewhat "hacky" way to test many small subfunctions in your main
>file is to have a syntax that causes the main function to return a struct
>array whose fields contain function handles to the subfunctions. Your tests
>can then invoke the subfunctions using those function handles even though
>the subfunctions are not in scope. As long as the subfunction was in scope
>when the function handle was _created_ it doesn't matter if the subfunction
>is out of scope when the function handle is _executed._

>http://www.mathworks.com/help/matlab/matlab_prog/applications-of-function-handles.html#brf_1tn-1

Thanks, that's very useful info. (I got to admit I loved the last
one, hacky as it may be.)