File Exchange

image thumbnail

JSONlab: a toolbox to encode/decode JSON files

version 1.5 (40.7 KB) by

JSONlab is an open-source JSON/UBJSON encoder and decoder for MATLAB and Octave.

4.55102
63 Ratings

535 Downloads

Updated

View License

Editor's Note: This file was selected as MATLAB Central Pick of the Week

** JSONLAB v1.5 (Nominus - alpha) is released on 01/02/2017.**
JSONlab is a component of the "iso2mesh" toolbox (http://iso2mesh.sf.net). For the latest information regarding JSONlab, please visit its homepage at http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab
JSON (JavaScript Object Notation) is a highly portable, human-readable and "fat-free" text format to represent complex and hierarchical data. It is as powerful as XML, but less verbose. JSON format is widely used for data-exchange in applications, and is essential for the wild success of Ajax and Web2.0.

UBJSON (Universal Binary JSON) is a binary JSON format, specifically optimized for compact file size and better performance while keeping the semantics as simple as the text-based JSON format. Using the UBJSON format allows to wrap complex binary data in a flexible and extensible structure, making it possible to process complex and large dataset without accuracy loss due to text conversions.

We envision that both JSON and its binary version will serve as part of the mainstream data-exchange formats for scientific research in the future. It will provide the flexibility and generality achieved by other popular general-purpose file specifications, such as HDF5, with significantly reduced complexity and enhanced performance.

JSONlab is a free and open-source implementation of a JSON/UBJSON encoder and a decoder in the native MATLAB language. It can be used to convert a MATLAB data structure (array, struct, cell, struct array and cell array) into JSON/UBJSON formatted strings, or to decode a JSON/UBJSON file into MATLAB data structure. JSONlab supports both MATLAB and GNU Octave (a free MATLAB clone).

JSONlab provides two functions, loadjson.m -- a MATLAB->JSON decoder,
and savejson.m -- a MATLAB->JSON encoder, for the text-based JSON, and
two equivallent functions -- loadubjson and saveubjson for the binary
JSON. The savejson, loadubjson and saveubjson functions were written by
Qianqian Fang, while the loadjson.m script was derived from the previous works by the following people:

- Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713
            date: 2009/11/02
- François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393
            date: 2009/03/22
- Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565
            date: 2008/07/03

Please find detailed online help at http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc

Comments and Ratings (179)

To fix the 'st' issue with 2017a, replace l.211 with the following snippet (also described in https://github.com/fangq/jsonlab/issues/37):

           tmpobj=eval(arraystr);
           assert(iscell(tmpobj)); %Since matlab 2017a ["GML", "XML"] produces a valid string array
           object = tmpobj;

Maglev

Maglev (view profile)

next code gives an error:
>> loadubjson(saveubjson('', 0.000132))

found the rootcause in saveubjson at line 300:
numtxt=regexprep(regexprep(matdata2ubjson(item,level+1,varargin{:}),'^\[',''),']','');

I fixed this with (add '$' after ']'):
numtxt=regexprep(regexprep(matdata2ubjson(item,level+1,varargin{:}),'^\[',''),']$','');

Andrew Smith

Calls that formerly worked in Matlab 2016b and earlier, don't work in Matlab 2017a. From savejson I now get:

Undefined function or variable 'st'.

Error in savejson>matlabobject2json (line 448)

txt=struct2json(name,st,level,varargin{:});

Sadly it seems to fail with enumerated types. Seems to work ok otherwise

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Paul, the best place to start is to run "help savejson" and "help loadjson" and read the format. you may also edit the demo script and see the calling examples.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Marcel, a=[1;2;3] is not a row vector. it is a column vector in matlab.

If you change a to [1,2,3], then jsonlab will give you the desired output.

Marcel Keyser

Is there any way that a row vector like a=[1;2;3] remains a 1D array in the json output?

{"a": [1,2,3]} instead of {"a": [[1],[2],[3]]}

Paul Fishback

I'm a little confused as to how the toolbox works. I've downloaded it, added it to my path directory, and ran the demo_jsonlab_basic. But I'm not sure what the output represents.

For my own use, I have json format files that describe certain "social networks" and I wish to obtain from them the corresponding adjacency matrices. Can I accomplish this with your package and, if so, how do I go about doing so. Many thanks.

anaston

I am sorry for the previous comment, I typed it in the wrong tab...

anaston

Serge Kogan

It's great!
Thank you very much!

Joe

Joe (view profile)

@Qianqian On the JSON page it states that:

"A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes."

It also states that a a character is:

"any-Unicode-character-except-"-or-\-or-control-character"

These control characters are the following:

\"
\\
\/
\b
\f
\n
\r
\t
\u four-hex-digits

This I understand should be interpreted to mean that a forward slash <by itself> shouldn't be interpreted as an escape character, but rather only when it's preceded by a back slash, as it is this combination that is explicitly stated to be an escape character. In the flowchart describing a string (http://www.json.org/string.gif) it also shows that all control character are preceded by a backslash.

The issue is that the save function seems to be interpreting the forward slash of the directory (which is saved as a string on matlab) as an escape character. Additionally, when using loadjson function on this newly created file, it interprets the \/ in the directory as a forward slash, which would be a second peculiarity.

I would take it that the problem arises in the usage of regular expressions when verifying for escape characters, but specifically how I couldn't say. Either way, this seems like an unintended behavior of the implementation if it's adhering to the JSON specification.

If it makes any difference, I'm working with matlab2014b running on OSX El Capitan, but that shouldn't be an issue when using regular expressions on matlab as far as I know.

Hope my point is clear!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Joe, I believe "/" is one of the escape characters in a JSON string. please check http://www.json.org/ and see the "string" specification.

Joe

Joe (view profile)

Hello, I have an issue when using savejson. If I have a name/value pair on matlab which is supposed to be read as this:

“Directory”:”/my/home/directory”

when I use the savejson function, it produces a JSON with the name/value pair of:

“Directory”:”\/my\/home\/directory”

Due to the forward slash not being primarily an escape character this seems like an odd behavior for the function to have. I went through the escapejsonstring and I'm not sure why this happens exactly. If I remove ('\/') from the escapechars list, then I get the desired name/value pair, but this seems like it might be problematic somewhere along the line, though sufficient for my current goal of writing a directory as a value in a JSON file.

Besides not being able to use '\/' as an escape character, would this produce any other issues? Is this a sufficient fix for now?

Thank you!

Juntao Qu

It works!
Thank you so much Qianqian!!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Juntao, matlab won't replace the "i" and "j" in data.link_i_j by the value you defined. field name "link_i_j" is simply a string.

to use i/j to dynamically create a field name, you should use the field name operator:

data.(sprintf('link_%d_%d',i,j))

give it a try.

Juntao Qu

Hi, I have a problem when loading the data.
Let's say my json file name is filejson. It works well as follows:

>> loadjson('filejson')

ans =

    link_16_14: [1x1 struct]
    link_16_15: [1x1 struct]
    link_16_16: [1x1 struct]
......
>> data.link_16_14

ans =

          MD_capac: 6.0000e+03
         term_node: 8
    MD_flow_minute: [1x120 double]
          link_idx: 16
......

But it errors when I want to use a defined variable instead of value:
>> i = 16;
>> j = 14;
>> data.link_i_j
Reference to non-existent field 'link_i_j'.

Can anyone tell me why and how to fix it? Because I need to use a for loop to call the data from json file.
Thanks a million!

Vinit

Vinit (view profile)

Very good job
I am able to convert JSON to sldd but can not convert base workspace data to JSON.

Jonn Lantz

Hi

Found a bug in savejson at line 256-258. an empty struct produces cell output. Replace { } with [ ] on these rows

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Jonathan, not sure what you were referring to when you say "still unable to parse". Setting FastArrayParser does allow it to be parsed and converted back and forth. try this:

a.foo = { 1, [2,3] }; a.bar = [1,2,3];
a1=loadjson(savejson('',a),'FastArrayParser',0)
a2=loadjson(savejson('',loadjson(savejson('',a),'FastArrayParser',0)),'FastArrayParser',0)
isequal(a1,a2)

The error when calling loadjson without FastArrayParser=0 is a known issue, and is now fixed, see

https://github.com/fangq/jsonlab/issues/5

Jonathan

Jonathan (view profile)

@Quianqian Converting every array to a cell array does not solve the problem, it avoids it. This parser is still unable to parse simple structures as:

a.foo = { 1, [2,3] }; a.bar = [1,2,3];

Qianqian Fang

@Jonathan, short answer, use this:

res=loadjson(savejson('',foo),'FastArrayParser',0)

a detailed answer was replied to this issue tracker on github:

https://github.com/fangq/jsonlab/issues/27

Lance

Lance (view profile)

very good toolbox, it's just what I want, Thanks

Jonathan

Jonathan (view profile)

@Qianqian Fang, here is a short example, not sure how to format it, is there a GitHub repo / SO tag for this toolbox?

>> foo.bar = {1,2,3}

foo =

    bar: {[1] [2] [3]}

>> savejson('',foo,'FileName','foo.json');
>> !cat foo.json
{
"bar": [
1,
2,
3
]
}
>> baz = loadjson('foo.json')

baz =

    bar: [1 2 3]

Qianqian Fang

@Gabriel Zienert, what you said was correct. specifying FloatFormat changes a logical to a floating point string first before 0-1 replacement. I will think about a better way to do this.

Qianqian Fang

@Jonathan, can you give me a real example?

Jonathan

Jonathan (view profile)

The reader cannot parse the output of the writer correctly! (ie, the transformation is not reversible)

Example saving and loading cells of scalar numbers; the writer converts the cell to an array with numeric entries in JSON output (I would expect an array with singlet array entries), which is then parsed as a numeric array when loading back into Matlab.

Setting 'SingletArray' for save and unsetting 'SimplifyCell' for load, or any combination of setting/unsetting, results in something different than a cell of scalar numbers.

I have a problem with savejson (v1.2) when saving floats and logicals. Try this:

savejson('',struct('float',pi,'logical',true),'FloatFormat','%.2f','ParseLogical',1)

and you will get:

{
"float": 3.14,
"logical": true.falsefalse
}

It seems that logicals are formatted as floats and then '0' and '1' are replaced with 'true' and 'false'.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Julia, if you want to output 1x2 cell from both cases, you should try 'SimplifyCellArray', try this:

s=loadjson( '{"s":{"chType":["None","None"]}}', 'simplifycellarray',true)

s.s.chType

s=loadjson( '{"s":{"chType":["None","Non"]}}', 'simplifycellarray',true)

s.s.chType

Hi,
I have a problem with laodjson when using the option 'SimplifyCell' at true.
Here are the contents of 2 json :
Test1.json : {"s":{"chType":["None","Non"]}}
Test2.json : {"s":{"chType":["None","None"]}}
If I do mysUser1 = loadjson('test1.json','SimplifyCell',true); I obtain a cell 1x2 as usual but when I do loadjson('test2.json','SimplifyCell',true); I obtain a char 2x4. It seems it's only when the two strings have the same number of letters.
Have you already noticed this bug? How can I fix it (I need the option SimplifyCell for other reasons)?
Thanks in advance
Julia

Jie Huang

Natarajan

Hi Qianqian, this is a great toolbox and works really well. I have a question - can you make an update to incorporate Map containers (like a hash map), which is very similar to a json. Right now, it tries to write it as a matrix and errors out.

Shaoxin Li

Thanks for the util. It works just fine for me.

Zaki

Zaki (view profile)

Thanks for the util. Could you debug this simple issue?

K>> loadjson(savejson('',{'a','b';'c','d'},'hey.json'))

ans =

    {1x2 cell} {1x2 cell}

For others who need a short-term fix with encoding matrices, please use http://vision.is.tohoku.ac.jp/~kyamagu/software/json/

Excellent utility to quickly make MATLAB data shareable with Python scripts with built-in importer/exporters.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Eric

thanks for the feedback, do you mind submitting one or more trackers for these issues on github? it will help us to document the changes, as well facilitate further discussions. here is the link

https://github.com/fangq/jsonlab/issues

Eric

Eric (view profile)

Hi, I would like to suggest a fix for valid empty JSON such as:
{}
[]
[{},{},{}]

To loadjson.m
line 66 - should be ".*" instead of ".+"
fixed 66: if(regexp(fname,'^\s*(?:\[.*\])|(?:\{.*\})\s*$','once'))

line 123&132: a '{}' in JSON is and empty structure in Matlab. Should be "struct([])" instead of "[]". The empty structure then needs to be indexed (see line 132).
fixed 123: object = struct([]);
fixed 132: object(1).(valid_field(str))=val;

line 207&208: an assert can be inserted here to detect objects. A "{}" will eval by Matlab to an empty cell but it should be a structure.
added after 207 (inside try):
assert(isempty(regexp(arraystr,'\{','once')),'Skip eval if the JSON contains objects.');

To savejson.m:
bug on line 216 where a cell is put inside the txt cell (should be a cell array of strings)). The txt cells should be concatenated.

is 216:
txt{end+1}=obj2json(name,item{i,j},level+(dim(1)>1)+(len>bracketlevel),varargin{:});

should be 216:
       txt=[txt obj2json(name,item{i,j},level+(dim(1)>1)+(len>bracketlevel),varargin{:})];

line 258 should be "{}" not "[]" for an empty object.
is 258: txt={padding0, '[]'};
should be 258: txt={padding0, '{}'};

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Eoin, I believe you just need an extra pair of curly brackets, like this:

>> savejson('',struct('tags', {{'somename', 'somelongername'}}, 'param1', 1000, 'param2', 'closed'))

ans =

{
"tags": [
"somename",
"somelongername"
],
"param1": 1000,
"param2": "closed"
}

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Jana, the support for exporting matlab objects was recently contributed by Sertan Senturk in the following two pull requests:

https://github.com/fangq/jsonlab/pull/10
https://github.com/fangq/jsonlab/pull/11

I personally haven't tested this extensively.

Do you mind opening a Github issue for this? providing a sample script to show the issue would be great. thanks

https://github.com/fangq/jsonlab/issues

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Paul, thanks. This was also pointed out by Jon Dufour earlier. A fix was committed on Jan 7 (see my reply on that date).

Paul

Paul (view profile)

Hi, nice toolbox, just a quick bugfix:

savejson('',struct.empty(1,0))

gives and error which can be fixed by inserting the following line after line 259:

txt = sprintf('%s',txt{:});

Eoin Murray

I would like to save some metadata to be sent to a REST API in use in my lab:

I have some tags in a cell array, however I cannot figure out how to get the matlab struct into a standard json-compliant format.

For example:

```
savejson('', struct('tags', char({'somename'; 'somelongername'}), 'param1', 1000, 'param2', 'closed'))
```

gives:

```
{
"tags": [
"somename ",
"somelongername"
],
"param1": 1000,
"param2": "closed"
}
```

you can see that in the tags array there extra whitespace
on the "somename" string, this is because matlab arrays are need to have fixed size elements.

However if I leave the tags as a cell array like so:

```
savejson('', struct('tags', {'somename'; 'somelongername'}, 'param1', 1000, 'param2', 'closed'))
```

I get:

```
[
[
{
"tags": "somename",
"param1": 1000,
"param2": "closed"
},
{
"tags": "somelongername",
"param1": 1000,
"param2": "closed"
}
]
]
```

Which is obviously not suitable to sent to a typical webservice on the internet.

Is there anyway for jsonlab to create a standard json array from a cell array without needing the extra text padding

Jana

Jana (view profile)

Hi, it's a terrific tool, I find it very useful. Thanks for that!! I only have one problem: it works perfectly with structures, but not with objects. If I try to export an object, it gives me an error: "Maximum recursion limit of 500 reached. Use set(0,'RecursionLimit',N) to change the limit. Be aware that exceeding your available stack space can crash MATLAB and/or
your computer." Is there any reasonable way how to fix it, without putting the computer in danger?

Cheers,

Jana

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Robin, try this

>> reloadedstruct = loadjson(jsonstruct,'SimplifyCell',1);

>> class(reloadedstruct)

ans =

struct

The SimplifyCell flag enables jsonlab to convert cell arrays to matrix/struct arrays when supported.

Robin

Robin (view profile)

Thanks for the good tool, I am using it successfully!
However, I have a problem with struct-arrays.
In the following code, I convert a struct array to json and back, but i finally get a cell array back.
I didnt yet find where exactly this occurs but i am still looking for a way around it.

substruct1 = struct('a',1,'b',2);
substruct2 = struct('a',3,'b',4);

teststruct = struct([substruct1,substruct2]);

jsonstruct = savejson('',teststruct);

reloadedstruct = loadjson(jsonstruct);

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

thanks Jon. for some reason, this slipped through all my regression tests. A patch is committed here, it should fix the problem.

https://github.com/fangq/jsonlab/commit/e07e91f5ec8f7caf210722348af9603f2145a409

Jon Dufour

Just started using this but seems like a great tool so far, thanks! One thing I noticed is that I get an error when trying to savejson() with a 0x1 struct. Of course 0x1 structs are weird and probably don't come around often, but thought I would let you know! Try this piece of code if you want to recreate (I'm using Matlab 2015b):

s.Field1 = 'Test';
s(1) = [];
savejson(s)

Thanks again!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

FYI, a new release (v1.2) is uploaded to FileExchange. Enjoy and happy new year!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

For everyone, I was able to improve the speed of savejson by a factor of 2.3 by eliminating string concatenations, following the insight provided by @Samuel.

See github Issue#17 for details

https://github.com/fangq/jsonlab/issues/17

and download the latest code from github.

@Qianqian, I created the issue #17 in github with the files. Cheers.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Samuel, we have an issue tracker for speeding up loadjson for large JSON file structures on github

https://github.com/fangq/jsonlab/issues/9

but we haven't spent time optimizing speed for savejson.

can you create a separate Issue on github and post patches or pull request so I can review the changes?

also, there is a built-in speed benchmark in JSONlab

https://github.com/fangq/jsonlab/blob/master/examples/jsonlab_speedtest.m

wondering how your patch changes the performance.

Very useful, yet, in last version the conversion to JSON of of complex datasets is still very slow, even in fast computer, mostly because internally the whole mechanism resides in string concatenation, sometimes even using infix concatenation which is very inefficient. By complex datasets I mean cell arrays with thousands of entries, each entry with structures with 30 or more fields, some of them with large 2D arrays. Because the large number of individual objects, the string concatenation becomes a pain as more objects are added.

I wrote a sister function savejson_fastfile that replaced most of sprintf to fprintf(fid... to redirect the output, I had to change a bit there and there to avoid infix concatenations, but it was not that hard. The only string concatenations I left was around the function matdata2json, mostly because the output of sprintf(formatstr,mat') truly requires some search/replace to produce a formal JSON string.

In my case, the savejson_fastfile requires 53s vs 150s with save_json to save a JSON stream with 49,000 objects, including all sort of mixed size of data (single values, strings, small matrices and large matrices). So roughly a 3X speed up. The test computer is a Dual Xeon E5-2640 @2.5 GHz (12 cores in total) and 32 GB RAM. The files produced by both functions are identical.

Let me know if you want to take a look of the new function. Thanks a lot for sharing your library.

sri user

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Hüseyin, for your question regarding handling of empty arrays, structs and cells, yesterday, I committed a patch to control brackets for single-element cells/arrays,

https://github.com/fangq/jsonlab/issues/8#issuecomment-164044337

I found it also made a difference to empty arrays, cells and structs. With this patch, you can now get the following results

>> obj={[],{},struct()};
>> savejson(obj)

ans =

{
"obj": [
null,
[],
{
}
]
}

this is not a robust solution though, because more loadjson/savejson iterations will convert all of them to nulls.

The root of the problem is the ambiguity and one-to-many mapping between JSON data and matlab structures.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Hüseyin, thanks for reporting this issue. This was also reported by another user on github. I posted a fix to this issue yesterday, see my replies at

https://github.com/fangq/jsonlab/pull/16

please checkout the latest version from github, and try again.

git clone https://github.com/fangq/jsonlab.git jsonlab

Hüseyin

First of all: Thanks for the nice ToolBox! :-)

OK here some problems using it with Matlab 2015b:

> Problem saving Json to file with the option 'FileName',<string>.
e.g. savejson('',json2data,'FileName','myFile.json')

In varargin2struct.m you use lower() for setfield(), so you have opt.filename and all other options are registered with lowercase.

In savejson.m line 147 you have the problem:
>> fid = fopen(opt.FileName, 'wt');

you try to access opt.FileName, which does not exists, since it is case-sensitive.

So fix this please with:
>> fid = fopen(opt.filename, 'wt');

and check for other options since all are defined in lowercase.

PS: Errors might be also in
>> Line 99, new: opt=struct('filename',varargin{1});
>> Line 144, new: fid = fopen(opt.filename, 'wb');

Hüseyin

There is a problem with "null" output in JSON files, which makes it incompatible, since i expect {} instead of "null".

See loadjson.m function parse_object() line 123:

You use: object = [];
output --> "mykey": null,

BETTER: object = struct();
output --> "mykey": {},

This quick-hack replaces the "null" entries by {}

Can you fix this or at least add as an option ?

Chris FUNG

Jason

Jason (view profile)

Saved my life with your toolbox! Thanks a lot!

Rob Campbell

Rob Campbell (view profile)

Does the job. Thanks!

@Qianqian

I am using the current version from mathworks.
I will move further discussions to github.
Some modifications of mine lead to a major speedup of your parser on the first run, so maybe we can contribute.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Timo

can you confirm you were using the latest release, v1.1? I had made some updates in this release to speed up progress bar and the jstruct2array function. see commit log

https://github.com/fangq/jsonlab/commit/2acca6745b7d6f47f3fb397fd598f51db7106353#diff-e4e4bc9fba80ed2220686f019637f986L203

@Qianqian

this is exactly what I did.
I profiled and timed your json2data() function and the recursive part is the most exhaustive one.
The second is the progress bar, if one turns it on.
Maybe you should reduce the calls by e.g. updateing the bar every 5%, and not on every loop.

Anyway, the time spend in kernelspace defently correlates with the depth of the JSON objects.
I've tested it with different structured files.

At last, the webread() function calls decodeJSON() (or decodeJSONstream()) in the end, which parses the data in a recursive fashion, as well as your code does.
Thus, no C conversion is supported.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Timo

I am not sure how you drew that conclusion. If I were you, I will take a 1MB sample out of the data and run the profiler. Parsing time is closely related to the complexity of the JSON data. You never know what was the bottleneck unless you profile it.

As I mentioned in a previous post, JSONlab performs 30% faster than Matlab 2015 webread(). I don't know if webread is non-recursive, but I believe it is compiled.

The runtime and the processing time spent in kernelspace correlates with the nesting depth of the JSON objects.
It is mainly caused by the utilization of recursive functions in your parser.
Is there a way to built a non-recursive parser?
This would also help to convert your code to mex functions to speed the parsing up.

Great work so far but I have a issue here.
I am loading JSON files of ~1GB size.
While it is still parsing (since 12h), I observe under Linux, that the process works for about 30% in kernel space.
What happens here?
Are there additional file IOs or system calls in you toolkit?

I call MATLAB from Bash in the following way:
matlab -nodisplay -r "opt.SimplifyCell = 0;opt.FastArrayParser = 1;opt.ShowProgress = 0;addpath('./jsonlab');json2data=loadjson('myJsonFile.json', opt);save('myJsonFile.mat');exit"

I run r2014b.
Do you have any recommendations, how to speed up the parsing?

Oliver Woodford

Thank you! This worked great.

Perfect - Thanks!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Christopher: yes, this can be done by defining a logical variable and set the 'ParseLogical' flag. here is the sample code:

a.field=logical(1);
savejson('a',a,'ParseLogical',1)

Thanks for putting this very helpful toolbox together. Due a peculiarity of the program I am running, I need to insert a string into my json file without the quotes. When I assign field = 'true', it prints the json file as "field" : "true", but I need it to print "field" : true

Is this possible? Thanks!

Frank

Frank (view profile)

Very good submission! I have one comment about saving cell arrays with one element into json.

At the moment, savejson only adds square brackets, if the cell aray has at least 2 elements and ignores the square brackets altogether, if the cell array has one element. In some applications this can be annoying, when filtering for a specific json structure with square brackets.

This can be adjusted by altering the subfunction "cell2json". Just replace the criterion "len>1" by "len>0" at three instances, namely line=183,201 and 214 in "savejson.m".

Perhaps this is something that could be improved in future, to make the user decide whether or not to use square brackets in case of cell arrays with one or more elements.

Wes Baldwin

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

just to let you know I've uploaded JSONlab 1.1. Please go ahead and give it a try. It is supposed to be faster (for large files, significantly faster).

here is the full changelog:
http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab/Doc/ChangeLog

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Vlad, can you give me an example what is a "real pre-pended JSON file"? I will be happy to put that support back if it does not conflict with other core features.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Jeremy, sorry for the delay. have been busy with deadlines.

For your first question, yes, I do use both JSON and UBJSON to store (part of) my research data. They worked quite nicely for me. Still many of my research data were stored in other formats (raw binary, text,csv etc), I am slowly migrating to JSON/UBJSON because many of my research projects involves home-made software for processing instead of matlab. I will have to add JSON/UBJSON support in my software first in order to convert everything.

For Q2, JSONLab 1.0 or earlier supports Draft 9. In the new version I am about to release, v1.1, the string format is updated to Draft 12. see

https://github.com/fangq/jsonlab/commit/4e88f2f60fb6aa85135e977de638dd62fea5856a

Regarding your comments on using JSON/UBJSON to store scientific data, I totally agree with you that these are excellent formats for your purposes. In fact, I am currently working on a format specification, based on JSON/UBJSON, specifically for storing scientific data (partly funded by an NIH grant). I call it the "JData Specification", here is the incomplete draft:

http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc/JData

In the future, JSONlab will likely stay compatible to this specification once it is finalized.

Vlad Atanasiu

Vlad Atanasiu (view profile)

Nice that we now have a waitbar! Version 2014/09/17 had a useful capability: to real pre-pended JSON files. Maybe we can get it back in a future version. Thanks!

Jeremy

Jeremy (view profile)

Jeremy

Jeremy (view profile)

Hi Qianqian, I had a few questions:
1. Do you use ubjson or json to store your data?
2. What UBJSON draft version does jsonlab implement?

I am asking because I want to find a format to store scientific data. Pure json would be more portable, but performance is much poorer (in precision/filesize and encoding/decoding time) than ubjson.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Jan. I agree. there are many places can be improved. Particularly, the main structures of loadjson was derived from prior works (see the Acknowledgements section above). While it works, it was obviously not optimized. I have been polishing/optimizing loadjson, started from the most time consuming parts and move downwards. I haven't got to the low-priority issues like you suggested yet, but it will get there soon (probably now is a good time).

by the way, speaking of speed, some people have complained about the slow parsing speed of loadjson. I noticed the same recently with a user supplied large (>2MB) JSON file. I did some optimization to the code, now loadjson is significantly (>20x) faster for certain type of files - it is even faster than MATLAB 2015's built-in webread() JSON parser!

For those who are interested, checkout the latest version from github. I will release JSONlab 1.1 soon.

https://github.com/fangq/jsonlab

For parsing a JSON file containing 150k unstructured small nodes, webread took 99 seconds, the git version of loadjson took 72 seconds on the same computer. it is nearly 30% faster.

Jan Simon

Jan Simon (view profile)

There are still some bad programming methods in the code, e.g. globals with standard names as "pos" or "len" and fields are created by the ugly "eval( sprintf('object.%s = val;', valid_field(str)));", although dynamic field names "object.(valid_field(str))=val" works for over 10 years now.

STRMATCH is surprisingly slow and outdated in addition. Using STRCMP would be more useful.

The code is easier to read and the JIT acceleration is better supported, when a line contains one command only. I do not see a reason to move this to 1 line: "nstr = length(str); switch inStr(pos)".

The submission contains an exhaustive and useful help section. I'd suggest to improve the mentioned programming techniques to allow for a reliable working in an productive environment.

Elkin David

Very Thanks!!!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Elkin, your question is similar to an earlier question posted by Kyle (on Jan 14, 2015), you have to use an additional parameter, "FastArrayParser", to prevent collapsing of the cell arrays,

loadjson('test.json','FastArrayParser',2)

by default, jsonlab simplifies cell arrays into arrays by calling cell2mat. Setting FastArrayParser to a positive number will set the starting-level of the grouping.

Setting FastArrayParser to 0 disables the grouping, every element in a JSON array becomes a cell.

Elkin David

I had a problem import this json

%%%%%%%%%%--start--%%%%%%%%%%%%%%%%%

{
"array":[

     [
        [1,2],
        [1,2],
        [1,2]
     ],
     [
        [1,2],
        [1,2]

     ]

]
}

%%%%%%%%%%%end%%%%%%%%%%%%

when i saved happened this

%%%%%%%%%% start %%%%%%%%%

{
"array":[

     
        [1,2],
        [1,2],
        [1,2],
        [1,2],
        [1,2]
     
      ]
}

%%%%%%%%%%%%%%%%end%%%%%%%%%%%%%%%

=(

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@gvoysey, no, UBJSON is a new specification, structured quite differently from BSON. In my opinion, it is simpler, and easier to work with. You can find more details at http://ubjson.org

However, it is still under development. There are various versions of the specs, for example, Draft 8,9,10,12... Unfortunately, these specs are not backward compatible.

JSONlab 1.0 was based on UBJSON Draft 9. However, the latest JSONlab dev branch was updated to Draft 12. In addition, I used an extended syntax to represent ND array, and proposed this to the UBJSON developer, however, it has not made into the specs as of today

https://groups.google.com/d/msg/universal-binary-json/tgMCEbOmhes/s7JlCl58hvQJ

the error you saw may likely caused by these known compatibility issues. do you see a specific error message when it fails?

gvoysey

Hi Qianqian,

I'm confused about UBJSON. Is this another name for BSON ? I'm having some trouble inserting serialized output from your toolbox into MongoDB using their community supported matlab drivers, and haven't been able to find a good guide.

I think that I'm doing everything "right", but giving the database data in a format it's not expecting.

Thanks for the clarification!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Michal, by "very slow", what did you compare jsonlab against?

did your timing include file IO? if yes, I would recommend you to simply return the output as a string, instead of writing to a file. The IO cost for writing a large file can be significant, especially on non-solid-state disks.

very slow save function for large json files (> 100k lines)

Samaksh Behl

Can one create markups/annotations in a PDF using JASONlab?

Thanks!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Graham, your input, json, is not exactly an object. it is a string. based on JSON grammar, double quotes inside a string needs to be escaped, thus the backslashes, see http://www.json.org/

to convert json to an actually matlab object, you need to do

jsonobj=loadjson(json);

then you can save jsonobj to a JSON string:

savejson('',jsonobj)

if all you want to do is to save an already converted JSON string to a file, you can just use the standard file IO functions - fopen/fprintf/fclose, no need to use savejson.

Graham Warner

Why does savejson add backslashes to the string? I have an object "json" which reads:
>> json

json =

{"Duration":16420.479075,"Num_Samples":40000,"Frequency":13449312,"Num_Channels":17}

and then when I try to save it, I get backslashes:
>> savejson('',json,'/space/jazz/1/users/gwarner/jason.mat')

ans =

"{\"Duration\":16420.479075,\"Num_Samples\":40000,\"Frequency\":13449312,\"Num_Channels\":17}"

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Pinghung, first of all, the loadjson function was derived from parse_json.m that you used as the reference. I am sure there is a simple tweak to restore the old behavior. Have you profiled both codes (profile on; loadjson(...); profile viewer)? it can tell you right away where was the difference.

Secondly, it is fairly easy to optimize for a special input to get good speed (again, use the matlab profiler is a good start), IMO, what is harder to achieve is to optimize the code so that it can get good speed for most types of inputs. This often needs a balanced solution. The changes in loadjson from the prior works were focused on accelerating the parsing of large arrays, which are most commonly encountered by matlab users. Try running the built-in speedtest benchmark and you will see the difference.

https://github.com/fangq/jsonlab/blob/master/examples/jsonlab_speedtest.m

thirdly, try a different versions of matlab, I found the later versions of matlab are ~10x faster when running loadjson.

PS: feel free to email me (fangqq at gmail.com) your input JSON file so I can find out exactly what happened with loadjson.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Laurent, you can disable the "FastArrayParser" by
setting it to 0, for example:

a = { 8 , 9 };
loadjson(savejson('',a),'FastArrayParser',0)

this will force treating all containers (array/object) to cells.

Laurent

this is great!
but one thing: cell arrays turned into matrices; how can I avoid this behavior ?
ex.:
a = { 8 , 9 };
savejson('',a,'f.mat');
b = loadjson('f.mat');
-> b is now the 1-x-2 matrix [ 8 9 ]
big thanks for the help

Mark Mikofski

Mark Mikofski (view profile)

Pinghung, have you tried the Java implementation org.json or the .NET implementations json.NET and fastJSON. I explain how to use them in this Undocumented MATLAB post: http://undocumentedmatlab.com/blog/json-matlab-integration and in this Gist: https://gist.github.com/mikofski/2492355

Pinghung

Much slower than other implementations like parse_json.m. We need to quickly deserialize datasets fetched from a JSON API. Will need to find a faster way to deserialize in Matlab.

Benchmark 1:
deserialize 500 JSON datapoints (each with nested data)

parse_json.m 0.785s
loadjson.m 3.914s

Benchmark 2:
deserialize 5000 JSON datapoints (each with nested data)

parse_json.m 7.837s (scales linearly)
loadjson.m stopped test after 3 minutes and 100% CPU consumption.

Iain Robinson

Iain Robinson (view profile)

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Alexander, you need to use the SimplifyCell flag, like this:

loadjson(savejson('',structarray),'SimplifyCell',1)

I have noticed that when I save a struct array in Matlab and then load the json file back again, it turns into a cell array. Is there a way to prevent it from doing this?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Kyle, you need to use the following command to reproduce the input:

y=loadjson(myjson,'FastArrayParser',2)

if a branch can be collapsed (by cell2mat), the FastArrayParser controls the depth of the output cell array. 0 disables this feature. Try setting it from 0 to 3 to see the difference.

In the future, I agree that loadjson should be more intelligent on which level can be simplified.

Kyle

Kyle (view profile)

Hi Qianqian Fang,

I cannot recover a cell containing numbers from your software. For example:

x = {[1],[1,2],[1,2,3]};
myjson = savejson('data',x);
y = loadjson(myjson);

In the example above, y does not equal x. I would like for y to be a cell just like x, but instead the numbers within x are collapsed into an array. Would you please change your code to make the input and output equivalent?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Calos, I think what you need is a vector transpose, see my example:

 a=rand(100,1);
 savejson ('myvar',a')

does this work for you?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi everyone, JSONlab 1.0 is here! enjoy

Carlos

Carlos (view profile)

Hi Qianqian Fang. I have a var which is a 100x1 double and I would like convert it to JSON in the following form:

{
  "myvar": [num1, num2, num3...],
}

I tried converting "myvar" to char and to cell but i don't achieve this form.
Thanks for your efforts.

Paul Macey

This just works - I had started writing my own decoder when I came across JSONlab, saving me a lot of effort. Thanks for the contribution.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

a progress bar is now added to loadjson, see the latest svn commit:

http://sourceforge.net/p/iso2mesh/code/452/

use loadjson(...,'ShowProgress',1) to activate this feature.

b

b (view profile)

Is it possible to output the record number on the prompt as it loads each record in the datafile? If the number of records are too large, can the cursor give any indication or percentage of the amount of file read or number of records read? Thanks.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

Unfortunately, JSON comment using // or % is not part of the specification. A common workaround is to put comments into an element with special names, for example "_help" or "::comment::" etc, load the entire object, but not to use those values. JSONlab can handle special characters in the name fields, so you can name a comment quite distictively.

b

b (view profile)

Hello Qianqian,

How to use % or // or anything else to comment a particular field in a json file so that loadjson will not give an error:
Error using loadjson>error_pos (line 431)
JSONparser:invalidFormat: String starting with " expected at position 3267:
30:02.091Z",

Alternately, if we want skip a field while loading (say, "_id" : ), is there a way to do this?

Thanks.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@tmartin

Compact JSON output is now supported, see my latest commit:

http://sourceforge.net/p/iso2mesh/code/449/

add ('Compact',1) to the savejson parameter list to enable this feature.

the only formatting white space left now is the space after the colon. I will remove them in my future updates.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Nicolas

I added the requested help info in the latest commit, please see

http://sourceforge.net/p/iso2mesh/code/448/

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Alessandro

thanks for pointing out this issue. It is indeed a bug related to the fast array parser that I added. I don't really have a good fix for this as it is related to how matlab eval() command handles an input string. However, I just added an option to loadjson that allows you to skip the fast parser for some array dimensions.

See my changes at:

http://sourceforge.net/p/iso2mesh/code/448/

your example can be properly handled using

B = loadjson('temp.json','FastArrayParser',2)

let me know if you see any addition issues related to this change.

Nicolas

Thanks a lot for your work.
Would it be possible to get some information on the parsing options that can be passed as input arguments to the loadjson function?
I've had a look on the toolbox home page but couldn't find anything.
Thanks for your help.

Timo Dörsam

tmartin

Is there a convenient way to produce a compact form of the JSON file?

T Lankhorst

I am amazed by the speed and usability of this software.

T Lankhorst

Alessandro

Hi Qianqian,

thank you for making available this very useful library.

I want to report what it looks like to be a bug. I noticed that if a cell-array contains matrices having same number of columns, and save/load it, the library concatenates vertically the matrices, outputting a single matrix. Please look at the code here below:
A = {rand(2,4), rand(3,4)};
savejson('', A, 'temp.json');
B = loadjson('temp.json');
whos A B

which outputs:
  Name Size Bytes Class Attributes
  A 1x2 384 cell
  B 5x4 160 double

Joerg Buchholz

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Richard

feel free to share your patch/update by sending me an email: fangqq <at> gmail.com. I will review it and commit it to the code repository (and add acknowledge to your contribution in the documentation).

just to understand your request, by "serialization", did you mean adding data as additional parameters? In the past, I would often do it by using a cell array "{obj1, obj2, obj3 ...}"

Richard

Excellent library, really useful. I have a feature request though - adding support for serialisation of Matlab objects. I have modified the code so that struct2json is called with an object, which works correctly. Is there anywhere where I can contribute these changes back?

Evgeny

Evgeny (view profile)

Was really useful for me for a simple task. One command and you're done

Evgeny

Evgeny (view profile)

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

that's not true. high dimensional arrays are supported. what's not supported is the high-dimensional cell arrays.

Dmitry

Dmitry (view profile)

You can't save multidimensional arrays with dimensionality larger than 2, is this true?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Kai

can you try converting your input "review" (a char array) to unicode first by using native2unicode()?

not sure if it helps, in the last test in jsonlab/sample/demo_jsonlab_basic.m, I have a utf-8 encoded Chinese string. loadjson/savejson seem to work well with this example (my locale is en_US.UTF-8).

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

sorry, not in jsonlab_selftest.m, but demo_jsonlab_basic.m

Kai

Kai (view profile)

Works great with Romanized characters. However, I ran into some character encoding issues. I am trying to read in script-type languages (Arabic, Urdu, Dari, Farsi, etc.) and the imported text is

{"text":"حمایت اتحادیه میوه فروشان و سبزی فروشان افغانستان از تیم اصلاحات و همگرایی؛\nحوالی ساعت 8 امشب 27 ماه حوت سال روان... http://t.co/AAcvfHxYAN","isoLanguageCode":"fa"}

Resulting text stored in the array:

حمایت اتحادیه میوه فروشان و سبزی فروشان افغانستان از تیم اصلاحات و همگرایی؛ حوالی ساعت 8 امشب 27 ماه حوت سال روان... http://t.co/AAcvfHxYAN

I can get the characters to show up properly from a text file if:
 a. I set my local Language Settings for
Persian/Arabic and

b. I specify UTF-9 encoding during fopen

>> file = fopen('tweet.txt','r','n','UTF-8');
>> review = fread(file,'*char')';
>> fclose(file);

Any thoughts?

Augustin

A typical example is a large music library. That being said it does not seem to be the use this toolbox was intended for.

Augustin

Pros : it's easy to use

Cons : 1) poor documentation (what are the options of the parser ?)

2) This toolbox appears to be optimized for large arrays, but it is not efficient when fields are small (e.g. strings, numeric values) but the json file is still big.

Albert

Albert (view profile)

Very useful, thank you. Have been using it for months. Quite fast.

Works great. Performs the conversions between string and struct like a champ.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

based on the error, it sounds like you had an invalid json file. have you tested your json file by one of the json validators? for example:

http://jsonlint.com/

J

J (view profile)

(someone please delete my previous question I have found the solution)

Problem with input:

data = loadjson('data.json')

I get error:

Error using loadjson>error_pos (line 415)
JSONparser:invalidFormat: Outer level structure must be an object or an array: /<error>

Error in loadjson (line 70)
            error_pos('Outer level structure must be an object or an array');
 

J

J (view profile)

I've attempted to use loadjson.m but I keep getting the error at line 47 saying "input file does not exist". All I am doing is "data = loadjson('data(5).json')" and that data file is within the same directory. Please help (or redirect me to where I should type my question if this is not the appropriate place). Many thanks!

Mark Mikofski

Mark Mikofski (view profile)

@Jan, I've been using a JSON.jar [1] I made from the java-json source [2] and it works perfectly. I don't know how native java speed compares with native MATLAB though.

[1] http://dl.dropbox.com/u/19049582/JSON.jar
[2] http://json.org/java/

Leah

Leah (view profile)

I would have liked if there was an option to return as a dataset. Added these lines

data=data.data;
data=struct2cell(data);
data=cell2mat(squeeze(data)');
data= dataset({data(:,1), 'TimeEPOC'}, {data(:,2), 'Value'});

Anssi Klapuri

Correction to my previous post: The mentioned line should read "currstr=inStr(pos:min(pos+100,end));" of course.

Anssi Klapuri

Excellent contribution. For large json files (mine was >100MB), you can make the code >100 times faster by replacing the line "currstr=inStr(pos:end);" with "currstr=inStr(pos:pos+100);" in parse_number().

Useful, fast parser. Much appreciated!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Chuck37

the issue you had reported is related to the cell2mat call on line#248 of loadjson. you can compare the following:

  a=struct('one',1,'two',2);
  b=[a a];
  c={a,a};
  class(b)
  class(c)
  class(cell2mat(c))

in many other cases, this kind of conversion is beneficial because it simplifies the data by looking into the internal regularity. However, it does come with the side-effect you have described.

To solve this issue, I just committed SVN#384, accepting a flag to control cell array->struct array conversion, see

http://iso2mesh.svn.sourceforge.net/viewvc/iso2mesh?view=revision&revision=384

you may check out this version and run the following to compare:

  d=loadjson(savejson('',c))
  class(d)
  d=loadjson(savejson('',c),'SimplifyCell',1)
  class(d)

by default, loadjson no longer converts a cell array to a struct array.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

@Arnout: 2D cell is currently not supported by jsonlab. I hope this can be added in the future versions. Contribution on this is welcome!

Thanks a bunch for JSONLab, it is saving me a lot of time writing my own serializer...

One question: I notice that 2-D cell arrays of strings get squashed to a 1-D JSON array of strings. Is it possible to keep the dimensionality of the cell array?

Chuck37

Thanks for making this. One issue I have is that sometimes the reader converts JSON files to cells and sometimes to structures. If I understand right, this is a function of whether the fields are exactly the same. For my application, this is a real pain because sometimes the fields are out of order (cell) and sometimes they aren't (array) and my processing code can't work with both without a lot of work. Is there a way I can force it to always make cells? I know this may not be as fast, but I need to know what I'm going to get.

Chuck37

Jonathan Lister

Works great. Seems really fast for my use. I wanted to save structs in human readable format. At first I went for XML tools, but found them lacking. JSON is less bloated than XML. I like it.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Mark

I just uploaded a new JSONLAB release, v0.9.0. Aside from various bug fixes, one of the main features of this release is the better round-trip consistency you suggested earlier. A leading underscore is now converted to x0x5F_; all hex codes generated by loadjson are automatically converted back to string forms in savejson.

A remaining issue is the handling of multi-byte characters in matlab. If this is the case, you may have to run feature('DefaultCharacterSet','latin1') in matlab to get consistent results.

Let me know if you are happy with this new release. (download from http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab/Download )

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Mark

taking out the try-catch block can give you very verbose output. For a simple array input, the output becomes a cell array with each element a cell. It becomes very difficult to read and process.

I think the difficulty we are facing comes from the ambiguity when mapping a matlab data structures to JSON. For example, when you say "foo", you can map it to "foo", {"foo"} or ["foo"], or even {["foo"]}. I chose the simplest one for now, because I don't know what we would gain from having the brackets (differentiate from an array origin or a cell origin?)

For the underscores, a unique prefix/suffix seems to be the safe way to go. If I were to implement this feature, I would add something like JSONLAB_LEADING_UNDERSCORE_ to both loadjson/savejson, and use opt.LeadingUnderscore as an input option.

Mark

Mark (view profile)

I should have been clearer with my first question: What I would like is for savejson to keep cell arrays of strings as cell arrays (i.e., to keep the single brackets around them). That is, I would like
O.test = {'x'};
s = savejson('',O)
to output:
s = '{"test":["x"]}'

You said, "both loadjson and savejson were designed to eliminate a JSON array that has a single element by default." - what was your reason for making this choice? And is there some way I can hack around in the code to get the behavior I want out of it without having 289 other things go wrong?

Also: I found that if I comment out the try-catch-end block you mentioned in your response to Nancy (loadjson.m lines 247-253), I can ensure that bracketed json objects (even with only one element: '["x"]') are interpreted as cell arrays ({'x'}) (which I want). Is making that change going to get me in trouble elsewhere in the code?

As to the underscores - well, I was thinking about a unique prefix you could use (or post-fix? two trailing underscores is legal (and uncommon) in matlab...?) But I can work around this myself if you'd rather not change both functions. Thanks!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Mark

regarding your first question, no, the NoRowBracket only applies to "numerical elements". I don't see any problems for processing array of strings though.

For your second question, it is not difficult for me to add a non-standard option to do the conversion. However, the current approach to deal with a leading underscore is to prepend an "x", similar to the behavior of MATLAB function genvarname(). However, once converted, a name in the form of "x_..." becomes difficult to separate from the genuine ones. A conversion from "x_..." to "_..." may have false positives. I am not sure if this is the best way to do, unless I define a unique prefix in the loadjson for this scenario.

Mark

Mark (view profile)

Hi Qianqian,

Thanks for the excellent code!

Can you please clarify your answer to Nancy's question above? When you wrote,

For savejson, there is a flag "NoRowBracket" controls if you want to force a [] around a single numerical element. By default, [] is not printed. Although this was not implemented for string arrays, a similar behavior is probably preferred to be consistent.

...did you mean that you intend to implement NoRowBracket functionality for arrays of strings? (And if so, might I add some encouragement for you to do so?)

Also, though I understand the complexity of achieving full round-trip invariance with loadjson and savejson, I wonder if you could be convinced to make one special case of round-trip invariance:

Leading "_" characters are commonly used in python variables, and are necessary for "_id" and "_rev" field names if you are working with json objects in couchdb. Is there any chance you could ensure that savejson un-did the string substitutions made to leading underscores in loadjson?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Cyril

thanks for pointing out this bug. I just committed a fix in the SVN:

http://iso2mesh.svn.sourceforge.net/viewvc/iso2mesh?view=revision&revision=369

Please check it out. Your example will have a output of

{
"test": {
"foo": null,
"bar": 0
}
}

a struct element with either {} or [] will be mapped to null.

Let me know if this works out for you.

Qianqian

Cyril Davin

Hi Qianqian,

Thanks for your nice work.
Here is a small bug :
s.foo={};s.bar=0; savejson('test',s)
The foo field and empty cell array disappear.

regards,
Cyril.

Mark Mikofski

Mark Mikofski (view profile)

FYI: for those interested in using Java org.json in MATLAB

Demo:
https://gist.github.com/2492355
git clone git://gist.github.com/2492355.git orgJSONdemo

JSON in Java (org.json):
http://www.json.org/java/index.html

Using Java Collections in Matlab:
http://undocumentedmatlab.com/blog/using-java-collections-in-matlab/

Storing MATLAB structs in Java Objects

Java SE 6 (1.6) API:
http://docs.oracle.com/javase/6/docs/api/

Java HashMap
http://docs.oracle.com/javase/6/docs/api/java/util/HashMap.html

Java ArrayList
http://docs.oracle.com/javase/6/docs/api/java/util/ArrayList.html

Passing Data to a Java Method:
http://www.mathworks.com/help/techdoc/matlab_external/f6425.html

Handling Data Returned from a Java Method:
http://www.mathworks.com/help/techdoc/matlab_external/f6671.html

Bringing Java Classes and Methods into MATLAB Workspace: Simplifying Java Class Names:
http://www.mathworks.com/help/techdoc/matlab_external/f4863.html#f46341

Working with Java Arrays:
http://www.mathworks.com/help/techdoc/matlab_external/f15351.html

Concatenating Java Objects:
http://www.mathworks.com/help/techdoc/matlab_external/f4873.html#f48488

javaArray:
http://www.mathworks.com/help/techdoc/ref/javaarray.html

Mark Mikofski

Mark Mikofski (view profile)

Thank you Q. Fang for this excellent program. As an alternative anyone could just use the org.json files directly from java. they are here:
https://github.com/douglascrockford/JSON-java
and directions are here:
http://www.json.org/java/index.html
You need to compile them or just export them to a jar file which requires jdk. I have compiled it already for you here:
http://dl.dropbox.com/u/19049582/JSON.jar
Before you can use the org.json library you have to add to your java class path (either dynamic or static) by following the directions here:
http://www.mathworks.com/help/techdoc/matlab_external/f4863.html
For example: javaaddpath('full-path\JSON.jar') will add the jar file to your dynamic path.
Then you can use java which is native to MATLAB to use the org.json library.
For example:
jsonObj = org.json.JSONObject('{myKey:myVal}')

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Nancy

both loadjson and savejson were designed to eliminate a JSON array that has a single element by default. In loadjson, line#248, the call cell2mat() attempts downgrading a 1x1 cell array to an array of the element type (unless there are mixed types). Test class(loadjson('["x"]')) with and without that try/catch block, you can see the difference.

For savejson, there is a flag "NoRowBracket" controls if you want to force a [] around a single numerical element. By default, [] is not printed. Although this was not implemented for string arrays, a similar behavior is probably preferred to be consistent.

To keep round-trip invariance is not likely, partly because JSON syntax has redundancies. What it might be achievable is to keep loadjson/loadjson->savejson->loadjson, or similarly, savejson/savejson->loadjson->savejson, giving similar answers. But it is still a quite challenging task.

Qianqian

Nancy

Nancy (view profile)

Hi Qianqian,

I noticed that when I have only 1 value in a json array, a roundtrip load/save doesn't give me the original json string (square brackets are left out).
Example:
>> jsonStr1 = '{"keys" : ["value1"]}';
>> savejson('',loadjson(jsonStr1))

ans =

{
"keys": "value1"
}

When there is more than 1 value in the json array, there is no problem.
Example:
>> jsonStr2 = '{"keys" : ["value1", "value2"]}';
>> savejson('',loadjson(jsonStr2))

ans =

{
"keys": [
"value1",
"value2"
]
}

Also, will there be future updates to convert the illegal character substitutions created by loadjson, when using savejson? For example, '_0x2F_' will be converted back to '/' when using savejson.

This is great though, thanks!

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Francois

both loadjson and savejson are expected to handle the input items in a sequential order.

For savejson, I am sure an numeric array/cell/cell array/string array are saved by element indices.

For struct inputs, savejson encodes each sub-field based on its "defined" orders. That means
  savejson('root',struct('f1',1,'f2',2))
and
  savejson('root',struct('f2',2,'f1',1))

give you different outputs. If you want to order it in a deterministic way, you can use orderfields() function to treat the struct first.

I do want to mention one thing, currently jsonlab can not handle high-dimensional cell/struct arrays. It converts it to 1D array based on matlab's column-major ordering. For example:
 a=struct('f1',1,'f2',2);
 a=repmat(a,2,2);
 savejson('',a)

I expect to improve this in the later versions. Any contribution on this is welcome!

Qianqian

Promising work !!
I will try to implement it in my researchs where we have a lot of heterogenous data as parameter of our simulations.

@Fang : do we have the insurance to always have the same order of nodes in the json file so I can use diffs and version control over json parameters files ?

Regards

Sally

Sally (view profile)

Release 0.8.1 solved the problem I had on 27 Feb. The bracket depth on my updated json remains identical to the depth in the input file.

We can now use jsonlib to record steps in our image processing pipeline. Thank you for adjusting this and doing it so quickly.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

I don't see problems for either case on my side:

>> works = struct('root', struct('root2', struct('root3', [1 2]')));
>> works_out = loadjson(savejson('' , works));
>> works_out.root.root2.root3

ans =

     1
     2

>> doesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
>> doesnt_work_out = loadjson(savejson('' , doesnt_work));
>> doesnt_work_out.root.root2.root3

ans =

     1 2

I even tested on octave, no problem at all. Can you double check and make sure you have downloaded the latest version (version 0.8.1)?

Oscar

Oscar (view profile)

I'm sorry again(!). Here is how the example should be.

works = struct('root', struct('root2', struct('root3', [1 2]')));
works_out = loadjson(savejson('' , works));

doesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
doesnt_work_out = loadjson(savejson('' , doesnt_work));

Oscar

Oscar (view profile)

I'm sorry, my previous post didn't appear for some reason. Here is it:
works = struct('root', struct('root2', struct('root3', [1 2]')));
works_out = loadjson(savejson('' , meta2));

donesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
donesnt_work_out = loadjson(savejson('' , meta2));

Oscar

Oscar (view profile)

also, in the case that works. The ouptut 'works_out' is not exactly the same as the input. In fact, it has turned into 'doenesnt_work'.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Oscar

I tested the following commands, it works fine with the current release of jsonlab:

a=struct('f1',[1 2 3 4])
savejson('',a)
a2=loadjson(savejson('',a))

I am happy to consider your patch, but it would be really helpful if you can give me your test script that failed earlier.

Oscar

Oscar (view profile)

hi Qianqian,

I found the problem. If the struct contains row-vectors, they don't get properly encoded. I added these simple lines to the mat2json subfunction, that convert the row-vectors to column-vectors, and now it works like a charm.

if ((size(item, 1) == 1) && size(item, 2) > 1)
  item = item';
end

thanks

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Oscar

please post your sample script and data so we can debug.

Qianqian

Oscar

Oscar (view profile)

In my experience, this doesn't work for all kinds of object, and remarkably, I couldn't load the output json struct with 'loadjson' that was created by 'savejson'. I wonder if there might be fundamental problems / ambiguities with a json matlab - struct interface?

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Sally, I just updated jsonlab to 0.8.1. A change in this release is to avoid forcing a "root" object in savejson.m. To do this, you will need to use savejson('',obj) to save your structure.

So, download the new version and give it a try. Let me know if this is what you want.

Sally

Sally (view profile)

My json gets an extra set of curly brackets on each loadjson/savejson round trip. This effect is shown in each example on your nice page: http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc/Examples.

Is there a way to avoid this? I would like to use matlab to add an element to an existing json file, without nesting the content deeper in the result file.

thanks, sally

Jan Simon

Jan Simon (view profile)

@Fang: (I hope this is your "1st" name) Thanks for your answer. I will try to test the speed improvement of my suggestions soon.
The current version number starting with 0 implies, that further development is planned. I expect, this is going to be a 5 star submission finally.

Qianqian Fang

Qianqian Fang (view profile)

  • 1 file
  • 535 downloads
  • 4.55102

hi Jan

thanks for your comments. The issues you pointed out in loadjson.m are largely inherited from the previous authors (except for the OCTAVE_VERSION one). I will consider your comments and improve the loading speed in the future. You are also welcome to submit patches and profiling results to help me out.

Also, please go to http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab/Download and download version 0.5.1, released yesterday. I realized I made a mistake in 0.5.0 and my optimized code on loading large data was not enabled. With the new code, loading a large matrix from JSON file can be 30x faster (by passing the parse_number calls in parse_array).

Hopefully version 0.5.1 will be updated on Matlab File Exchange soon.

Jan Simon

Jan Simon (view profile)

The excessive usage of global variables reduces the speed remarkably and might interfer with other programs.
exist('OCTAVE_VERSION', 'builtin') is a better test, because it does not search the file "OCTAVE_VERSION.m" in the complete Matlab path.
With pre-allocations the code could be accelerated very much. Dynamic fieldnames are faster than GETFIELD. STRMATCH is extremly slow and deprecated.
"eval(sprintf('object.%s = val;', valid_field(str)))" is cruel. Better: "object.(valid_field(str)) = val".
MLint suggests to omit some FINDs.
This function is very useful, because JSON files are highly efficient for a certain kind of data. If the speed is improved furtherly, this is a very good submission.

Updates

1.5

-use Big-endian format to store floating points (d/D) in saveubjson
-speedup parsing large unstructured data
-make parsing independent of whitespace
-allow to parse array of homogeneous elements
-fix a bug that prevents saving to a file in savejson

1.2

- 2x speed in savejson (Issue#17)
- fix FileName case bug (SVN rev#495)
- add SingletCell/SingletArray flags (Issue#15,#8)
- fix file name bug (Pull#14)
- fix bug for cell with transposed data (Pull#12)
- export matlab object to JSON (Pull#10,#11)

1.2

- 2x speed in savejson (Issue#17)
- fix FileName case bug (SVN rev#495)
- add SingletCell/SingletArray flags (Issue#15,#8)
- fix file name bug (Pull#14)
- fix bug for cell with transposed data (Pull#12)
- export matlab object to JSON (Pull#10,#11)

1.2

- 2x speed in savejson (Issue#17)
- fix FileName case bug (SVN rev#495)
- add SingletCell/SingletArray flags (Issue#15,#8)
- fix file name bug (Pull#14)
- fix bug for cell with transposed data (Pull#12)
- export matlab object to JSON (Pull#10,#11)

1.2

- 2x speed in savejson (Issue#17)
- fix FileName case bug (SVN rev#495)
- add SingletCell/SingletArray flags (Issue#15,#8)
- fix file name bug (Pull#14)
- fix bug for cell with transposed data (Pull#12)
- export matlab object to JSON (Pull#10,#11)

1.1

-massively accelerating loadjson
-parse logicals in loadjson
-make options case insensitive
-reading unicode encoded json files
-allow \uXXXX in a unicode in a string
-handle escape characters in a string
-implement the UBJSON Draft12 new name format

1.15

ChangeLog:
- update documentation, finalize 1.0
- fix a bug to strictly respect NoRowBracket in savejson

1.13

ChangeLog:
- fix UBSJON data endianness for all integer types
- fix several compatibility issues for GNU octave
- support 2D cell and struct arrays
- escape special characters in a JSON string
- fix a bug when saving UBSJON files

1.12

- update to jsonlab 1.0 beta release
- fix file read and write bugs in loadubjson and saveubjson

1.11

update to 1.0 alpha release, included functions (saveubjson and loadubjson) for writing and reading Universal Binary JSON files

1.10

ChangeLog

- 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson
- 2012/06/01 support JSONP in savejson
- 2012/05/25 fix the empty cell bug
- 2012/04/05 savejson can save to a file

1.9

ChangeLog:

 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS
 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson

1.7

ChangeLog:
-2012/01/13 speed up loadjson by 20 fold when parsing large data arrays in matlab
-2012/01/11 remove row bracket if an array has 1 element
-2011/12/22 accept sequence of 'param',value input
-2011/11/18 fix struct array bug

1.3

ChangeLog:

 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration
 2011/10/20 loadjson supports JSON collections: appendable JSON objects

MATLAB Release
MATLAB 7.4 (R2007a)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video