December 11, 2013

GAIA brings CNES into the age of Big Data

CNES is well used to processing large amounts of data, but for the GAIA mission it was time to turn to Big Data technologies.

CNES is responsible for processing science data from the European Space Agency’s GAIA astronomy mission. GAIA is set to map our galaxy in three dimensions, with the extremely ambitious goal of locating and characterizing more than a billion stars. And in terms of the number of operations involved, GAIA is going to process the most data ever seen in the history of astrometry, recording star positions, distances, motions, magnitudes and many more parameters besides. In all, more than 80 billion objects will be surveyed to yield a catalogue of more than 1.3 billion stars.

During the conceptual phase of the mission, it quickly became clear that for GAIA the limits of conventional database technologies had been reached and that CNES would have to find other solutions to assure the mission’s success. The massive amounts of data to be processed and above all the huge number of objects to be handled led the agency in 2011 to take a technological leap and test a range of different products. Finally, it decided to adopt the Hadoop system, based on the Big Data technologies now used by the leading Web players like Facebook, Google, Amazon and eBay.

Astrometry is not the only field facing growing volumes of data at CNES. As a result, the use of Big Data systems will be extended to future projects to which the agency is contributing. For example, in Earth observation the six Sentinel satellites of Europe’s Copernicus environmental monitoring programme and the French-U.S. Surface Water and Ocean Topography (SWOT) mission will be sending back one to two terabytes (1012 bytes) of data from each satellite every day.

With respect to Big Data, Geneviève Campan, in charge of CNES’s information system, said: “CNES is upgrading its information system to be ready to cope with this deluge of data. The agency’s space project managers are working closely with the Information System Directorate to address this issue vital to the success of many future missions.”

CNES press contacts

Alain Delrieu Tel. +33 (0)1 44 76 74 04
Pascale Bresson Tel. +33 (0)1 44 76 75 39
Julien Watelet Tel. +33 (0)1 44 76 78 37