Data saving and loading overview

From Endrov

Getting existing data into Endrov

Endrov supports a variety of formats produced by common software. Optimal are the file-formats where everything goes into one file; these usually keep all the original data. The absolutely worst is when the files are dumped into individual files, one for each plane. We recommend storing in OME-TIFF.

Storing image data as TIFF is generally a bad idea if other single-file formats exist because the metadata is not standardized. OME-TIFF is the only exception.

Avoid common photo image formats such as JPEG, PNG, BMP as these do not store any metadata.

If you are stuck with individual files that need to be combined then we have a importer plugin.


Associated data

Image pixel data is going to be a small part of the data in the near future, and already now you should make sure that basic description of the data is stored with it:

  • Resolution, how large is a pixel in reality?
  • What is it a picture of?
  • Who took the picture and when?
  • How large are the time steps if it is a recording?

Some formats such as TIFF do not normally keep this information. Microscopy formats support the above, but what about new experimental variables?

  • Expression pattern
  • Temperature
  • Movement of specimen
  • Cell lineages
  • etc

OME is standardizing much common metadata but very specialized data is not covered; Endrov will store an additional OSTXML-file alongside the image file with all our extra data.

We have also developed an OST-container for image data but because OME is doing a good job we do not recommend this solution anymore.

Compression

You should always use compression for best performance and space usage. There are two types:

  • Lossy compression (e.g. JPEG) - When you are willing to trade quality for smaller files. Perfectly fine for qualitative data but you have to tune the quality settings to your liking. You should however only ever use lossy compression if you have a space limitation, and you are really sure what you are doing!
  • Lossless compression (e.g. PNG) - No trade for quality, all data is stored intact, but in less space. Recommended for channels for quantification, unless you know what you are doing.

Performance is faster with compressed images and will become even higher. Moore's law and related state that the time to read the hard disk will become a larger and larger bottleneck. If you can reduce the amount of data you have to read from the hard disk by increasing the workload on the CPU (decompression) then it is typically a good deal.