Load/save python list of list with MATLAB

Most of my work is done in matlab, but I currently have something I needed to do in python. Basically there is a package that I have to use in python. Some issue has raised with transferring data between matlab and python.

My matlab data are saved as matlab v7.3 '.mat' files. Which I then load them in python using a library called 'mat73'. This step is fine and I have managed to do all the neccessary things I do in python. Issue arise when I try to saveout python objects as '.mat' file using the scipy library. The error I got is something like this.

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (29,) + inhomogeneous part.

I believe it is mad at me because scipy cannot deal with list of lists (where the inner list is of different length). I was wondering if I can avoid this problem by changing the libraries I used to load/save data (or some other methods that I currently cannot think of).

A few things to note:

1. I could avoid this by padding the list of list to the same length within python. In fact I have already done that for some of the data I created in python. However this problem also come when some of the matlab cell arrays as loaded as python list of list, but cannot be saved out in the same way.
2. You might ask from the previous point: then why don't you use the same library to save '.mat' file in python? It is because 'mat73' is only made for loading data but not saving.
3. I can't really locate where this error would occur. By this I mean I know where it is wrong in this file, but I don't know where it would be wrong when I run another file. Unfortunately each of my data is not organized perfectly and some files have an extra field than some other files. And as I mentioned I work with big nested structs so it would be pretty hard to find a good way to locate the cell array (or list of list). After all I don't really know what should I be searching for...
4. I have thought about saving the data in pickle files, and then potentialy find some way to load pickle file in matlab. Is that possible and if so will the same issue still occur?

With that said advice is welcomed. Even just some incomplete thought. Thanks a lot!!

[Stephen23]

https://www.squash.io/how-to-fix-valueerror-setting-an-array-element-with-a-sequence-in-python/

Answer 1

[Nipun]

Hi Yuxin,

I understand that you are facing an error while using Python's list of lists in MATLAB. I assume that the list data types are different from one another and size of the lists varies in the list of lists.

List of lists are stored as a collection of row or column vectors in MATLAB, called matrices. Since MATLAB supports concatenation operation for similar-sized lists only, it is not possible to import list of varying length lists from Python as they are imported as a matrix.

One possible workaround for this problem is to create structs with list as fields. You may consider importing to JSON format and reading the JSON file. The following code shows how to load JSON format files in MATLAB:

In Python, do:

import json

% Data to be imported to MATLAB
dat = {'k1':{'k11':[1,2,3],'k12':[[1,2],[1],[2,2]]},'k2': []}

% Save a JSON file
with open('dat.json', 'w') as f:
    json.dump(dat, f)

In MATLAB, use the following command-line code to load:

>> dat = jsondecode(fileread('dat.json'))

dat = 

  struct with fields:

    k1: [1×1 struct]
    k2: []

>> dat.k1

ans = 

  struct with fields:

    k11: [3×1 double]
    k12: {3×1 cell}

>> dat.k1.k12

ans =

  3×1 cell array

    {2×1 double}
    {[       1]}
    {2×1 double}

Hope this helps.

Regards,

Nipun