Thread Subject: NI-DAQ: 12 bits saved as double?

I have made some Matlab software for recording from National Instrument DAQ boards and saving data to disk. I record up to 10 channels at 20KHz over up to ? hour. The files becomes >1 gb, which makes working with them slow.
The .daq files show that NativeDataType is 'double', thats 64 bits per sample. However the acquisition board only has an 12 bit AD converter, so it could be saved in e.g. 16 bits, that is 1/4 of the space I'm using now.

Does anybody have a soultion ? E.g. is there a way that I can change the nativeDataType of an analog input object for a NI DAQ board?

It is possible to change the nativeDataType for an analog input object connected to the soundcard, by changing the ai.BitsPerSample, however, this attribute seems to be specific for WINDSOUND devices.

Hope some one can help
Thanks in advance!
  
 

An update:

When using the old NIDAQ 7.4.4 (Not mx) driver sampled data are saved in int16, as they should be. However, using a 2 years old driver which is soon to be phase out of Matlab is not a super good solution.

If any one have a better solution I'll be very interested.

Hi Kristian,

You are correct that WINSOUND is the only adaptor for which you can change the NativeDataType.

A few words about the data size returned, then we'll talk about the slowness you mentioned.

Data Acquisition Toolbox (DAT) works with the data in the same size as the vendor's hardware drivers. The NI-DAQmx drivers return double, no matter what the resolution of the board is, so DAT will give you doubles back. NI-DAQmx now does calibration of the data before returning it. This means that the drivers can now return better data than they could before. Better data is preferable to smaller data. As you discovered, Traditional NI-DAQ did return a smaller data size, but as the resolution of devices increases so does the size of the data returned.

In modern systems with ample RAM and disk reducing the data size is not as important as it use to be. Let's say a device can acquire 500,000 samples/second. To store one second of data as a double will take 4,000,000 bytes. To store one second as a uint16 will take 1,000,000 bytes. Now on my PC, the MEMORY command indicates that I have 1.587e+009 bytes available for all arrays. Therefore the difference between storing one second of data in uint16 versus double is:

>> (4000000 - 1000000) / 1.587e+009 * 100

ans =

    0.1890

Which means that for this device we're talking .2% of the available memory for one second of data at the maximum data rate of the card.

Lastly, MATLAB does much of its computation in double precision anyway. A lot of functions will work on the native data type of the array, but a lot will promote to double. For example, if you want to do an FFT, you're now working with double precision complex numbers.

In terms of addressing the slowness you speak of with 1GB files, we would need to program around this. If you're talking about files I assume you're logging to disk and reading the files with DAQREAD. Have you considered logging to memory, using GETDATA and storing the data in smaller sized MAT files? This way you would have much more control and could create smaller files.

All the best,
Sherryl

Hi Sherryl,
Thanks For Your quick answer

It seem that there is no getting around this problem when using the "logtodisk" then. However, I'm a bit puzzled:

How can the quality/resolution of a 16 bit AD converter be improved? Is this not just a conversion from int16 to double with the use of a scaling factor and an offset (the variables are already stored in the .daq file format), which daqread is already capable of applying when reading in data. (It would be must faster to read native 16bit data from a slow HD and then convert it to doubles when the data was already in the much faster RAM.)

In Labview it is possible to select which format one want to return from a NIDaq device, so to me it seems as the drivers support the return of other formats than double. Maybe the two programs do not have access to the same drivers..

The old M series DAQ and my new E series DAQ are 12bit and 16bit AD converters, respective , however, in both cases a sample should be capable of being saved in 2 bytes/16 bits.

The problem is not as much RAM consumption, as I'll have to use double when performing calculations, as you say. My problem is that it takes very long time to load data from the HD, some times 5 min. The read time is related to the size of the file, to my experience. As I can not understand how I would get any improved in data quality by having 4 times overhead in file size, I would like to reduce the file sizes.

Until now I've been very happy with the integrated "logtoDisk" Matlab rutine. However as my recordings has gotten longer there has been an increased in file size making it a slow experience, when reading data again. I like that it is possible to extract a specific channel and also specify which samples, and I would like to avoid spending to much time in making rutins for loading and saving data, when they are already available, and have been tested and optimized by Matlab staff.
At the moment I'm considering to convert my .daq files to 16 bits wave files, as a practical solution.

Regards,
Kristian

Hi, I just ran into a similar problem. I'm sampling the outputs of photomultiplier tubes of an astronomical instrument at 4 MHz with a 16 bit NI DAQ board. Since I'm sampling 4 channels simultaneously, I should end up with 100 GB/hour of data. Saving 16 bit data as a 64 bit double is obviously a problem, as 400 GB/hour is excessive. NI has told me that they can stream int16 values at the expected rate through LabView, so why is it that Matlab is only able to use doubles?

Thanks!

Sloane

"Sherryl Radbil" <sherryl.radbil.dontspamme@mathworks.com> wrote in message <gkq1rn$pb0$1@fred.mathworks.com>...
> Hi Kristian,
>
> You are correct that WINSOUND is the only adaptor for which you can change the NativeDataType.
>
> A few words about the data size returned, then we'll talk about the slowness you mentioned.
>
> Data Acquisition Toolbox (DAT) works with the data in the same size as the vendor's hardware drivers. The NI-DAQmx drivers return double, no matter what the resolution of the board is, so DAT will give you doubles back. NI-DAQmx now does calibration of the data before returning it. This means that the drivers can now return better data than they could before. Better data is preferable to smaller data. As you discovered, Traditional NI-DAQ did return a smaller data size, but as the resolution of devices increases so does the size of the data returned.
>
> In modern systems with ample RAM and disk reducing the data size is not as important as it use to be. Let's say a device can acquire 500,000 samples/second. To store one second of data as a double will take 4,000,000 bytes. To store one second as a uint16 will take 1,000,000 bytes. Now on my PC, the MEMORY command indicates that I have 1.587e+009 bytes available for all arrays. Therefore the difference between storing one second of data in uint16 versus double is:
>
> >> (4000000 - 1000000) / 1.587e+009 * 100
>
> ans =
>
> 0.1890
>
> Which means that for this device we're talking .2% of the available memory for one second of data at the maximum data rate of the card.
>
> Lastly, MATLAB does much of its computation in double precision anyway. A lot of functions will work on the native data type of the array, but a lot will promote to double. For example, if you want to do an FFT, you're now working with double precision complex numbers.
>
> In terms of addressing the slowness you speak of with 1GB files, we would need to program around this. If you're talking about files I assume you're logging to disk and reading the files with DAQREAD. Have you considered logging to memory, using GETDATA and storing the data in smaller sized MAT files? This way you would have much more control and could create smaller files.
>
> All the best,
> Sherryl

Hi Kristian - I noticed this problem with the slowness of the daqread command (when I started developing a daq-file-reading GUI with a 370 Mb data file as my test subject).

There are (at least) two reasons that daqread is so slow.

First is that because the .daq file is written on the fly, it has no centralized directory of the data blocks. When daqread is called, it first compiles a 'chart' of the file by inching its way through the file (this is done by localReadChart within daqread.m). This is very time-consuming, and it has to be done EACH TIME that daqread is called. I tinkered with this so that I can cache a copy of the chart after it is compiled, and then skip this step on subsequent calls for reading the same file.

Second is that when reading a subset of the data (e.g. specified samples), the localReadData function does not jump directly to the correct place in the file (using the chart that was just compiled), but it inches its way, block by block, from the beginning of the file (kind of like using an old-fashioned serial tape-reader rather than a random access disk). So the later the target data in the file, the longer it takes to read it (even if you're only reading 10 values!). I fixed this for the type of data that I'm recording, but I'm not sure if the solution is robust. (I added an index of the startsample for each data block to the cached chart, and then use this to jump to the correct place in the file.)

I can see why mathworks chose not to cache the file directory (chart), because it makes the calls to daqread more cumbersome and less foolproof (however, it would be possible to cache the chart in a persistent variable, and to validate it at the start of each call). But I'm not sure why the localReadData function was done so inefficiently.


"Kristian Nielsen" <krauhe@hst.aau.dk> wrote in message <gkq7n0$8hl$1@fred.mathworks.com>...
> The problem is not as much RAM consumption, as I'll have to use double when performing calculations, as you say. My problem is that it takes very long time to load data from the HD, some times 5 min. The read time is related to the size of the file, to my experience. As I can not understand how I would get any improved in data quality by having 4 times overhead in file size, I would like to reduce the file sizes.
>
> Until now I've been very happy with the integrated "logtoDisk" Matlab rutine. However as my recordings has gotten longer there has been an increased in file size making it a slow experience, when reading data again. I like that it is possible to extract a specific channel and also specify which samples, and I would like to avoid spending to much time in making rutins for loading and saving data, when they are already available, and have been tested and optimized by Matlab staff.

I posted a copy of the improved version of daqread on the File Exchange. Please note (in my comments on the file) that it has not been tested for all types of .daq files.

daqreadCached: http://www.mathworks.com/matlabcentral/fileexchange/27289-daqreadcached