JSONlab: a toolbox to encode/decode JSON files in MATLAB/Octave

JSONlab is an open-source implementation of a JSON encoder and a decoder/parser for MATLAB/Octave.

Description

JSONlab was a component developed for the iso2mesh project (http://iso2mesh.sf.net). For the latest information regarding JSONlab, please visit its homepage at http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab

JSON (JavaScript Object Notation) is a highly portable, human-readable and "fat-free" text format to represent complex and hierarchical data. It is as powerful as XML, but less verbose. JSON format is widely used for data-exchange in applications, and is essential for the wild success of Ajax and Web2.0. With the fast advance of web-based technologies, we envision that JSON will serve as a mainstream data-exchange format for scientific research in the future, and fulfill part of the roles achieved by HDF5.

JSONlab is a free and open-source implementation of a JSON encoder and a decoder in the native MATLAB language. It can be used to convert a MATLAB data structure (array, struct, cell, struct array and cell array) into JSON formatted text, or to decode a JSON file into MATLAB data. JSONlab supports both MATLAB and GNU Octave (a free MATLAB clone).

JSONlab provides two functions, loadjson.m -- a MATLAB->JSON decoder, and savejson.m -- a MATLAB->JSON encoder. The loadjson.m script was derived from the previous works by the following people:

- Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713
            date: 2009/11/02
- François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393
            date: 2009/03/22
- Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565
            date: 2008/07/03

Please find detailed online help at http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc

• MATLAB release: MATLAB 7.4 (R2007a)
• Other requirements: JSONlab is platform independent.

[Jan Simon]

The excessive usage of global variables reduces the speed remarkably and might interfer with other programs.
exist('OCTAVE_VERSION', 'builtin') is a better test, because it does not search the file "OCTAVE_VERSION.m" in the complete Matlab path.
With pre-allocations the code could be accelerated very much. Dynamic fieldnames are faster than GETFIELD. STRMATCH is extremly slow and deprecated.
"eval(sprintf('object.%s = val;', valid_field(str)))" is cruel. Better: "object.(valid_field(str)) = val".
MLint suggests to omit some FINDs.
This function is very useful, because JSON files are highly efficient for a certain kind of data. If the speed is improved furtherly, this is a very good submission.

[Qianqian Fang]

hi Jan

thanks for your comments. The issues you pointed out in loadjson.m are largely inherited from the previous authors (except for the OCTAVE_VERSION one). I will consider your comments and improve the loading speed in the future. You are also welcome to submit patches and profiling results to help me out.

Also, please go to http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab/Download and download version 0.5.1, released yesterday. I realized I made a mistake in 0.5.0 and my optimized code on loading large data was not enabled. With the new code, loading a large matrix from JSON file can be 30x faster (by passing the parse_number calls in parse_array).

Hopefully version 0.5.1 will be updated on Matlab File Exchange soon.

[Jan Simon]

@Fang: (I hope this is your "1st" name) Thanks for your answer. I will try to test the speed improvement of my suggestions soon.
The current version number starting with 0 implies, that further development is planned. I expect, this is going to be a 5 star submission finally.

[Sally]

My json gets an extra set of curly brackets on each loadjson/savejson round trip. This effect is shown in each example on your nice page: http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc/Examples.

Is there a way to avoid this? I would like to use matlab to add an element to an existing json file, without nesting the content deeper in the result file.

thanks, sally

[Qianqian Fang]

hi Sally, I just updated jsonlab to 0.8.1. A change in this release is to avoid forcing a "root" object in savejson.m. To do this, you will need to use savejson('',obj) to save your structure.

So, download the new version and give it a try. Let me know if this is what you want.

[Oscar]

In my experience, this doesn't work for all kinds of object, and remarkably, I couldn't load the output json struct with 'loadjson' that was created by 'savejson'. I wonder if there might be fundamental problems / ambiguities with a json matlab - struct interface?

[Qianqian Fang]

hi Oscar

please post your sample script and data so we can debug.

Qianqian

[Oscar]

hi Qianqian,

I found the problem. If the struct contains row-vectors, they don't get properly encoded. I added these simple lines to the mat2json subfunction, that convert the row-vectors to column-vectors, and now it works like a charm.

if ((size(item, 1) == 1) && size(item, 2) > 1)
  item = item';
end

thanks

[Qianqian Fang]

hi Oscar

I tested the following commands, it works fine with the current release of jsonlab:

a=struct('f1',[1 2 3 4])
savejson('',a)
a2=loadjson(savejson('',a))

I am happy to consider your patch, but it would be really helpful if you can give me your test script that failed earlier.

[Oscar]

also, in the case that works. The ouptut 'works_out' is not exactly the same as the input. In fact, it has turned into 'doenesnt_work'.

[Oscar]

I'm sorry, my previous post didn't appear for some reason. Here is it:
works = struct('root', struct('root2', struct('root3', [1 2]')));
works_out = loadjson(savejson('' , meta2));

donesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
donesnt_work_out = loadjson(savejson('' , meta2));

[Oscar]

I'm sorry again(!). Here is how the example should be.

works = struct('root', struct('root2', struct('root3', [1 2]')));
works_out = loadjson(savejson('' , works));

doesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
doesnt_work_out = loadjson(savejson('' , doesnt_work));

[Qianqian Fang]

I don't see problems for either case on my side:

>> works = struct('root', struct('root2', struct('root3', [1 2]')));
>> works_out = loadjson(savejson('' , works));
>> works_out.root.root2.root3

ans =

     1
     2

>> doesnt_work = struct('root', struct('root2', struct('root3', [1 2])));
>> doesnt_work_out = loadjson(savejson('' , doesnt_work));
>> doesnt_work_out.root.root2.root3

ans =

     1 2

I even tested on octave, no problem at all. Can you double check and make sure you have downloaded the latest version (version 0.8.1)?

[Sally]

Release 0.8.1 solved the problem I had on 27 Feb. The bracket depth on my updated json remains identical to the depth in the input file.

We can now use jsonlib to record steps in our image processing pipeline. Thank you for adjusting this and doing it so quickly.

[Francois Rongère]

Promising work !!
I will try to implement it in my researchs where we have a lot of heterogenous data as parameter of our simulations.

@Fang : do we have the insurance to always have the same order of nodes in the json file so I can use diffs and version control over json parameters files ?

Regards

[Qianqian Fang]

hi Francois

both loadjson and savejson are expected to handle the input items in a sequential order.

For savejson, I am sure an numeric array/cell/cell array/string array are saved by element indices.

For struct inputs, savejson encodes each sub-field based on its "defined" orders. That means
  savejson('root',struct('f1',1,'f2',2))
and
  savejson('root',struct('f2',2,'f1',1))

give you different outputs. If you want to order it in a deterministic way, you can use orderfields() function to treat the struct first.

I do want to mention one thing, currently jsonlab can not handle high-dimensional cell/struct arrays. It converts it to 1D array based on matlab's column-major ordering. For example:
 a=struct('f1',1,'f2',2);
 a=repmat(a,2,2);
 savejson('',a)

I expect to improve this in the later versions. Any contribution on this is welcome!

Qianqian

[Nancy]

Hi Qianqian,

I noticed that when I have only 1 value in a json array, a roundtrip load/save doesn't give me the original json string (square brackets are left out).
Example:
>> jsonStr1 = '{"keys" : ["value1"]}';
>> savejson('',loadjson(jsonStr1))

ans =

{
"keys": "value1"
}

When there is more than 1 value in the json array, there is no problem.
Example:
>> jsonStr2 = '{"keys" : ["value1", "value2"]}';
>> savejson('',loadjson(jsonStr2))

ans =

{
"keys": [
"value1",
"value2"
]
}

Also, will there be future updates to convert the illegal character substitutions created by loadjson, when using savejson? For example, '_0x2F_' will be converted back to '/' when using savejson.

This is great though, thanks!

[Qianqian Fang]

hi Nancy

both loadjson and savejson were designed to eliminate a JSON array that has a single element by default. In loadjson, line#248, the call cell2mat() attempts downgrading a 1x1 cell array to an array of the element type (unless there are mixed types). Test class(loadjson('["x"]')) with and without that try/catch block, you can see the difference.

For savejson, there is a flag "NoRowBracket" controls if you want to force a [] around a single numerical element. By default, [] is not printed. Although this was not implemented for string arrays, a similar behavior is probably preferred to be consistent.

To keep round-trip invariance is not likely, partly because JSON syntax has redundancies. What it might be achievable is to keep loadjson/loadjson->savejson->loadjson, or similarly, savejson/savejson->loadjson->savejson, giving similar answers. But it is still a quite challenging task.

Qianqian

[Mark Mikofski]

Thank you Q. Fang for this excellent program. As an alternative anyone could just use the org.json files directly from java. they are here:
https://github.com/douglascrockford/JSON-java
and directions are here:
http://www.json.org/java/index.html
You need to compile them or just export them to a jar file which requires jdk. I have compiled it already for you here:
http://dl.dropbox.com/u/19049582/JSON.jar
Before you can use the org.json library you have to add to your java class path (either dynamic or static) by following the directions here:
http://www.mathworks.com/help/techdoc/matlab_external/f4863.html
For example: javaaddpath('full-path\JSON.jar') will add the jar file to your dynamic path.
Then you can use java which is native to MATLAB to use the org.json library.
For example:
jsonObj = org.json.JSONObject('{myKey:myVal}')

[Mark Mikofski]

FYI: for those interested in using Java org.json in MATLAB

Demo:
https://gist.github.com/2492355
git clone git://gist.github.com/2492355.git orgJSONdemo

JSON in Java (org.json):
http://www.json.org/java/index.html

Using Java Collections in Matlab:
http://undocumentedmatlab.com/blog/using-java-collections-in-matlab/

Storing MATLAB structs in Java Objects

Java SE 6 (1.6) API:
http://docs.oracle.com/javase/6/docs/api/

Java HashMap
http://docs.oracle.com/javase/6/docs/api/java/util/HashMap.html

Java ArrayList
http://docs.oracle.com/javase/6/docs/api/java/util/ArrayList.html

Passing Data to a Java Method:
http://www.mathworks.com/help/techdoc/matlab_external/f6425.html

Handling Data Returned from a Java Method:
http://www.mathworks.com/help/techdoc/matlab_external/f6671.html

Bringing Java Classes and Methods into MATLAB Workspace: Simplifying Java Class Names:
http://www.mathworks.com/help/techdoc/matlab_external/f4863.html#f46341

Working with Java Arrays:
http://www.mathworks.com/help/techdoc/matlab_external/f15351.html

Concatenating Java Objects:
http://www.mathworks.com/help/techdoc/matlab_external/f4873.html#f48488

javaArray:
http://www.mathworks.com/help/techdoc/ref/javaarray.html

[Cyril Davin]

Hi Qianqian,

Thanks for your nice work.
Here is a small bug :
s.foo={};s.bar=0; savejson('test',s)
The foo field and empty cell array disappear.

regards,
Cyril.

[Qianqian Fang]

hi Cyril

thanks for pointing out this bug. I just committed a fix in the SVN:

http://iso2mesh.svn.sourceforge.net/viewvc/iso2mesh?view=revision&revision=369

Please check it out. Your example will have a output of

{
"test": {
"foo": null,
"bar": 0
}
}

a struct element with either {} or [] will be mapped to null.

Let me know if this works out for you.

Qianqian

[Mark]

Hi Qianqian,

Thanks for the excellent code!

Can you please clarify your answer to Nancy's question above? When you wrote,

For savejson, there is a flag "NoRowBracket" controls if you want to force a [] around a single numerical element. By default, [] is not printed. Although this was not implemented for string arrays, a similar behavior is probably preferred to be consistent.

...did you mean that you intend to implement NoRowBracket functionality for arrays of strings? (And if so, might I add some encouragement for you to do so?)

Also, though I understand the complexity of achieving full round-trip invariance with loadjson and savejson, I wonder if you could be convinced to make one special case of round-trip invariance:

Leading "_" characters are commonly used in python variables, and are necessary for "_id" and "_rev" field names if you are working with json objects in couchdb. Is there any chance you could ensure that savejson un-did the string substitutions made to leading underscores in loadjson?

[Qianqian Fang]

hi Mark

regarding your first question, no, the NoRowBracket only applies to "numerical elements". I don't see any problems for processing array of strings though.

For your second question, it is not difficult for me to add a non-standard option to do the conversion. However, the current approach to deal with a leading underscore is to prepend an "x", similar to the behavior of MATLAB function genvarname(). However, once converted, a name in the form of "x_..." becomes difficult to separate from the genuine ones. A conversion from "x_..." to "_..." may have false positives. I am not sure if this is the best way to do, unless I define a unique prefix in the loadjson for this scenario.

[Mark]

I should have been clearer with my first question: What I would like is for savejson to keep cell arrays of strings as cell arrays (i.e., to keep the single brackets around them). That is, I would like
O.test = {'x'};
s = savejson('',O)
to output:
s = '{"test":["x"]}'

You said, "both loadjson and savejson were designed to eliminate a JSON array that has a single element by default." - what was your reason for making this choice? And is there some way I can hack around in the code to get the behavior I want out of it without having 289 other things go wrong?

Also: I found that if I comment out the try-catch-end block you mentioned in your response to Nancy (loadjson.m lines 247-253), I can ensure that bracketed json objects (even with only one element: '["x"]') are interpreted as cell arrays ({'x'}) (which I want). Is making that change going to get me in trouble elsewhere in the code?

As to the underscores - well, I was thinking about a unique prefix you could use (or post-fix? two trailing underscores is legal (and uncommon) in matlab...?) But I can work around this myself if you'd rather not change both functions. Thanks!