Big Data and MATLAB: Dig Software

Dig Technology provides services to the oil and gas industry, including carbon capture and storage, by leveraging advanced workflows for subsurface interpretation. Our team has developed smaller, project-specific applications using MATLAB^®. However, as our projects grow in number and data volume, we face challenges in scalability and efficiency. To address this, we are creating an integrated software solution that consolidates current and future workflows.

The team has significant experience in software development and backend programming in MATLAB, and we had previously compiled Java libraries for integration into other systems. However, our team lacked Java expertise and wanted to eliminate dependence on external programming support. This raised the question: could we achieve everything we need using only MATLAB?

Having moved from megabytes, to gigabytes, to hundreds of gigabytes and possibly terabytes of data, we were at first skeptical if we would be able to achieve a good enough performance doing it by ourselves in MATLAB. But after months of research and testing, we were encouraged by the results and committed to developing Dig Software using MATLAB App Designer and object-oriented programming.

This presentation focuses on the core of Dig Software, which is how we manage visualization and predictions on large 3D volumes. Users must be able to efficiently navigate through these volumes and generate instant profiles, slices, and horizon plots. Key challenges include data loading, memory management, and high-performance plotting. In this talk, we will share how we tackle these obstacles, including implementing custom HDF5 file formats and using what we call the ""surf-light"" function.