Big Data in Earth Observation – Calvalus Processing System for massive parallel processing and rapid prototyping
Observing the health of the Earth’s environment by satellites is called Earth Observation (EO). EO missions provide a unique dataset of observational data which is rapidly growing and producing multiply terabytes of data every day. Data processing, calibration, algorithm development, validation and reprocessing of the derived products are indispensable tasks for an efficient exploitation of EO data and form the basis for reliable scientific conclusions. Processing and reprocessing are demaning in terms of data throughput and processing performance, and in spite of its importance, the cal/val and algorithm development work is often hindered by insufficient means to access data, time consuming work used to identify suitable in-situ data matching the EO data, incompatible software and limited possibilities for a rapid prototyping and testing of ideas. In view of the amount of data produced specifically by Europe’s Sentinel satellites, a very efficient technological backbone is required to maintain the ability of ensuring data quality and processing performance. Brockmann Consult has developed such a backbone based on leading edge technologies: Calvalus is a processing system that capitalises on the power of Hadoop, utilises the map-reduce programming model with a distributed file system, integrates easily typical EO processing work flows and provides maximum flexibility whereby a high processing performance is achieved. Calvalus scales automatically to the size of the task, and can be efficiently used by large data processing centres, operational organisations, research institutes and value adding industries. Among others, Calvalus is used within the Copernicus Collaborative Ground segments of Germany (https://code-de.org/) and Finland (https://www.environment.fi/en-US/Envibase), by the European Space Agency (e.g. in the Urban TEP, https://urban-tep.eo.esa.int/), by the national aeronautics and space research centre of the Federal Republic of Germany (DLR) and is supporting the production of Land Cover, Fire Disturbance, SST and Ocean Colour Essential Climate Variables for the ESA Climate Change Initiative.
- Development of the concept; continuous improvements
- Software architecture, implementation and testing
- Integration into user‘s environment
- Development of user defined workflows (consultancy service)
- Training courses