Path management for different versions of projects

9 views (last 30 days)
Imagine that you develop a large Matlab program, which uses hundrets of functions. To keep the different modules independent from eachother, the corresponding functions are store in separate folders, e.g.:
During the development your create different version of e.g. the GUI and the export functions and get this hierarchy:
Which tools are you using (do you recommend) to manage these folders in your MATLAB path? Do your tools consider dependencies, e.g. export1.03\ needs gui1.02\ or higher?
Due to the different folders GENPATH does not match. If you're working on project2\, project3\, etc., using the PATHTOOL and storing the different path strings does not work also, because the versions of the different projects cannot be set independently.
I've written a tool for this job, but it is far too specific and complex to be published. In addition it includes folders of the compiled MEX files for the different Matlab versions and patchs of the toolbox functions (e.g. a TYPECAST and BSXFUN for Matlab 6.5). Using a standard source control system helps to manage the files, but I did not find a comfortable method to care about the set of folders in the MATLAB path. Some of my projects are used for clinical studies and the users must be able to reproduce the results for 10 years after a publication. "Sorry, I cannot get my old source control system to run under Window 14" is not an option.
I think this is a common problem and therefore I'm interested in your solutions.
  1 Comment
Jan on 7 Jul 2011
I'm goind to accept the answer: "Such a function will be included in Matlab 2012a, TMW". :-)

Sign in to comment.

Accepted Answer

Bjorn Gustavsson
Bjorn Gustavsson on 7 Jul 2011
In my MyStartup.m I have a call to a function LOADPROJECTS that lets me choose projects, then the paths required gets added to the matlab path. To my LOADPROJECTS I also have EDITPROJECTS, ADDPROJECTS, etc. So I can set up the paths for Project6p5 then unset and set the paths for Project7p2p.
Maybe you're asking for something more extensive, but this covers my current needs to keep separate versions apart. I'd be very interested in hearing what others do.
Alvery on 23 May 2012
We have set up SVN to automatically generate a .m file to instantiate folder paths when you checkout a workspace. The path includes the "primary" folder + a series of dependent folders. The folder dependencies are stored in a SVN property, and the script recursively looks for the SVN property in all svn:externals folders. svn:externals is a general purpose mechanism within the SVN tool for linking libraries and the like, and links to a specific revision of the library if you want to.
The result is that you can check out the MATLAB code + Simulink models to any location on your computer, and have the path correctly configured.

Sign in to comment.

More Answers (6)

Titus Edelhofer
Titus Edelhofer on 7 Jul 2011
Hi Jan,
interesting question. And there will be no definite answer ;-). I can only describe my approach:
  • generally I'm no fan of having different folders with the same files. I sometimes do it as well (with folders project_xyz_2011_04_01), but at some time it will just fail.
  • In my opinion using a good version management system is the only way out. When I have a new version of a project finished, I check out the project, zip it and have it stored separately.
  • I admit, that one problem I usually don't have: the integration problem you mention (GUI 1.02 with Compute 2.37). Most of the projects have a coherent versioning.
  • This way the folder structure does not change often, and I have no big problems with path setting ...
I'm also curious what others say.
  1 Comment
Jan on 7 Jul 2011
@Titus: Thanks for this answer. The demands for a version control grow with the size of the projects.

Sign in to comment.

Daniel Shub
Daniel Shub on 7 Jul 2011
Your folder naming does not seem to fully capture the power of modern version control systems. If you think of your project as having a version and not the individual components, then you will never have to have folders like gui1.01.
My projects tend to rely on a set of common export and calculations folders. These I version control separately. When I start a new project I import the most recent version of my calculations "project." In this way each project has its own stable copy of the calculations project.
Jan on 7 Jul 2011
The SVN wrapper sounds very interesting. I'll take a deeper look into the psychotoolbox.
Automatic updates are not wanted by some users of my software: a full test needs 4 to 6 weeks. Therefore the code is freezed and it is preferred to find workarounds for known bugs.

Sign in to comment.

Arthur on 21 May 2012
I manage a similar issue with a product I maintain that uses a lot of common in-house tools that are also used by some unrelated analysis functions. The product and analysis code are kept in separate git repositories. I have packaging & release functions checked in with the product that I use to create reproducible builds for delivery; they wrap a call to git that determines the version number from the latest tag (git describe), build documentation with the version number embedded, run MCC, zip and copy files to a folder for burning to a CD. The product itself looks for git when not isdeployed, and self-reports the current commit ID when running from MATLAB. (It also compares the data files we ship to precomputed MD5 hashes, because we have to worry about "power users" overwriting data files and producing untraceable results.)
I am planning to migrate the shared functionality to separate repositories and use git subtree to bring them in to each repository that needs the functionality. This way, analysis functions will not depend on the product directory. Once that is complete, I will have duplication between the repositories, but there will be no run-time dependencies between them. Changes to the common functions will be propagated between the repositories only on request, so there will not be any surprise updates.

Jason Ross
Jason Ross on 8 Jul 2011
Using a version control system is only part of the answer. You need to also have a release process to produce the finished software package and a means of setting your environment appropriately to run only the files of that release.
One approach could be as simple as copying the files for the "released" project1 and then having a file that sets this environment correctly. Then, project1 is effectively "frozen" in time and you can return to it at any point in the future (*). Of course, in a modern version control system, you can also add in branches, tags, etc to know when each was released and also use it as a means to re-create and patch older versions as needed.
You should also take into account other dependencies you have -- for example, compilers, web servers, drivers and other software bits you might need to re-create the environment at release time. Depending on the complexity and number of interactions in the various software modules, the communatorics of testing changes for past versions may make trying to keep up what version works with what other version to be too complex an undertaking -- for example, if you make a change to package A that's at version 1.0.2 and that is supposed to be compatible with three old releases, you need to test all of those to make sure they still work, give desired outputs, fail in the same way and so on. This can be daunting depending on the size and complexity of your code base.
And, of course, you also need to actually test your releases in a "sterile" environment to ensure that you don't have some dependency that you didn't capture.
  • - assuming that the OS you are running can still run the executables. You may need to preserve an appropriate OS image by some means (for example, making an image of the OS and keeping it with your release -- but this opens another can of worms). At some point, the OS vendor will drop support for an old OS, or newer versions of an old OS will no longer keep backwards compatability to an old one. Microsoft has been fairly decent about keeping compatibility, via one means or another -- but the list of caveats gets larger with each older release as new technologies, OS architecture and chip families emerges. I've found that virtualization offers a nice and convenient way to capture OS images and also largely sidesteps the potential hardware/driver issues that exist, since the OS only has to use the virtualization system's drivers, and you don't run the risk of too-modern equipment not having supported drivers that work with a very old OS.

Jan on 11 Jul 2011

K E on 23 May 2012
There is a nice presentation on smart ways to manage directories for a scientific project here. It lays out a consistent approach to naming directories and deciding what should go in them. [Not what your question asked, but it still may be of interest].

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!