Can I generalize a folder lookup location so that data can be accessed in more than one way?

I'm finding this issue difficult to describe, so bear with me...

So a colleague created a structure that looks for files in specified location, for example if I call the structure.get_image() function, it looks in folder '/media/Server/user/...'

However, the data itself has been copied to different servers and is accessed from different workstations (sometimes with different operating systems), and not all of those workstations and servers have the same file structure. For example, in one it may need to look in '/data/DifferentServer/user...' - everything past the server name should be the same for everyone and every type of access, but not so with the server name itself and what comes before it. It may also need to look in a windows computer at a location like Z:\media\Server\user\...' which possibly poses a different problem.

So my question is, is there a way to generalize the folder location to work in all of these cases?

Structure example (omitted some things, but all of the omitted things are universal regardless of how the data is accessed):

algorithm: 'omitted'
parameters: [1×1 struct]
nJobs: 2
comment: []
folder: '/media/Cube/omitted/' % The problem
refScan: 1
uuid: (leaving this out)
patient: []
study: [1×1 study]
sliceFolder: '/media/Cube/omitted/omitted/' % Also the problem
% There is a lot more to these structures, but I think this gives the idea

Error received if accessed from a server that instead has the folder located at '/data/Sphere/omitted/':

Cannot find file
"/media/Cube/omitted/01_deformed.nii".

Currently people have to go in and do this if they want to use the data from another location:

set(thing.subThing,'folder','data/Sphere/omitted');
set(thing.subThing,'sliceFolder','data/Sphere/omitted');
% About 5 more things to set in addition to these, but you get the idea

And then they can't save those changes or else it may mess it up for other people accessing the data. And sometimes we want to work with dozens of these structures at once so we would have to do these sets for each one (and the "omitted" part is different for each one, so it can't be looped). Ideally the folder location would be instead something like:

folder: '.../omitted/'
% Same for sliceFolder and all the other paths

Where the "..." is determined based on where "omitted" is actually located for that user.

Is this sort of generalized pointer possible?

Apologies again for how difficult this has been for me to explain, please let me know if I can provide any additional clarification!

[Stephen23]

"Is this sort of generalized pointer possible?"

One simple approach is to get the user to browse to that location and then use relative filepaths:

https://www.mathworks.com/help/matlab/matlab_env/specify-file-names.html

[Ryan Andosca]

Ah so that works, but my concern there would be someone getting frustrated if I push that update to all of the data and they are used to accessing it without browsing to the location. May be just something people have to get over if we want the access to work, but if there's another (even more complicated) way of doing this that maintains the current functionality entirely, it would be preferable.

Also, would this fix the issue with users accessing the data from a Windows machine? I'm currently the only user not on Linux, but others have expressed interest. (I currently have the issue fixed by using a mklink junction, which is unlikely to be a solution other users will implement)

[dpb]

Hardcoding the specific server name with the rest of the folder is the root of the problem; those should be two different pieces of building the full file name, not one. Then, build the path via fullfile using the particular user's server path.

Ideally, the app should have a separate configuration file for each user that lets them customize their install to match their workstation and work habits. Then, the data aren't global and wouldn't have the server embedded into a dataset that subsequently gets moved somewhere else.

One temporary workaround while working out a more generic solution that would alleviate a fair amount of the issues raised above would be to put the file browsing call inside a try...catch block that only prods the user to browse to the correct location if the present server path doesn't exist or the file isn't at that location. Then those for which the present is correct won't know there's anything different.

And, of course, if one does have to change the server, then there should be a script/function/app that does all the needed changes given the updated server info. That would/will be much simpler once you get the server specification into its own datafield.

It should be essentially transparent as to OS with MATLAB IFF you use fullfile() and other high-level MATLAB functions to do the filename manipulations; it handles the format for each OS. You may have some issues with existing names hardcoded into the data files; I'm not sure how forgiving Linux is on the forward vs backwards slash and other details...

[Stephen23]

"Ideally, the app should have a separate configuration file for each user"

Another option is to specify the root path as a function input argument, and let the user manage their own path as they see fit. If you have competent MATLAB users than that might be a good option.

In any case, one way or another if every computer has the required data stored in a different location then you need to have a way to provide that location to your code. What approach you use is more of a design decision than a technical one: what user experience do you want to offer your users? Does your code already use a configuration file? What skill level do your users have? Do you provide your code as functions/methods to be called by the user, or as a turn-key blackbox? Is the code installed along with other code (possibly with other languages/apps) or consists of MATLAB functions only?

"I'm not sure how forgiving Linux is on the forward vs backwards slash and other details"

Not at all forgiving: linux accepts only the /, whereas windows accepts both / and \. So to write paths that work on both, use /.

[Steven Lord]

So to write paths that work on both, use /.

To write folder paths that work on Linux, Windows, and Mac use fullfile. Or if you can't for some reason, use filesep.

[Ryan Andosca]

Going to be looking into these solutions today, but to answer some questions and provide more information:

The users are fairly adept users, but they already have a bunch of code that uses this data. They want to be able to access the data from our new server that we uploaded the data to, but the location the data is stored in in the new server (call it Sphere) doesn't match the old server (call it Cube). So I'm trying to provide a way that users can run their code on the new server and have it work out-of-the-box so to speak, rather than them having to do something like alter the folder locations themselves.

I wrote a very simple function based on an earlier suggestion that users can run when they open up the data that will update the folder locations locally (since some people still use the data on the old server) that people can use for now, but if we have new people come in to work with this data it would be easier if it didn't require they run a function based on how they're accessing the data (I wrote one for each server that has the data). Though I get it if that's just what is required.

Appreciate all of the responses and I'll be looking into each one more today!

Answer 1

[Jan]

The core of the problem is a messed up storage of files in a bunch of folders. Collecting the files in one shared folder would be much cleaner. A database is even better to store a bunch of data.

It is not hard to write a workaround: The folders needed by a user are defined in a function e.g. called "PersonalDataFolder" in the personal user path. Then calling the sharded function "FindFile" requests the list of the folders and searchs for the specific file in the set of given folders:

function File = FindFile(FileName)
FolderList = PersonalDataFolder();  % E.g. {'/media/Server1/user1', '/media/server2/user1'}
File = '$File not found$';
for iFolder = 1:numel(FolderList)
   aFile = fullfile(FolderList{iFolder}, FileName);
   if isfile(aFile)
       File = aFile;
       break;
   end
   % Alternatively: Check the other folders also to find ambiguities
end
end

[Ryan Andosca]

I'm concerned that I'm not understanding this solution correctly, though I'd like to try it.

So each object (aThing) has a bunch of data associated with it (aThing.get_data, aThing.get_otherData, aThing.subThing.get_data, etc) and each of those "get_data" functions look for that data in different folders associated with that object instance. When I run a get_data on a place other than the original server, it looks in the "wrong" folder as described. Is the suggestion that insead of delineating folders for each object instance, all the data should be stored in a single folder and the proposed FindFile function should be used in "get_data" to pull the correct file?

Worth mentioning I didn't make these objects and I sort of inherited this issue. I just use the data the most, so it's become my job to ensure it's accessible from any of the servers that it is stored on. I'm an intermediate MATLAB user at best!