Pinpointing the

Milky Way

Gaia.qxd 12/12/07 1:33 PM Page 26

Gaia

In the course of Gaia’s 5-year astronomical survey, the equivalent of around 20 000 DVDs of raw information on our Galaxy

will be harvested and transmitted to Earth.Sophisticated processing is needed to distill thisflood of complex data into the final GaiaCatalogue of about 1000 million celestialobjects. A group of more than 300 Europeanscientists and software developers is rising tothe challenge: the Data Processing and AnalysisConsortium is already preparing for Gaia’slaunch in 2011.

IntroductionGaia is a pioneering astronomy missionset to revolutionise our view of theGalaxy, the Milky Way, with a preciseand detailed survey of the 1000 millionbrightest celestial objects. With high-accuracy ‘astrometry’, Gaia will pin-point the position of each star and trackits movement across the sky. Measuringits light spectrum will reveal how fast thestar is approaching or receding (its‘radial velocity’). Gaia will also gather‘photometric’ data, measuring thebrightness of a star in a few dozencolours. This array of data will reveal a

Jane Douglas, Jos de Bruijne, Karen O’Flaherty& Timo PrustiResearch and Scientific Support Department,Directorate of Scientific Programmes, ESTEC,Noordwijk, The Netherlands

William O’Mullane & Uwe LammersGaia Science Operations, Directorate ofScientific Programmes, ESAC, Villafranca, Spain

François MignardObservatoire de la Côte d’Azur, Nice, France

Ronald DrimmelINAF Osservatorio Astronomico di Torino, Turin,Italy

esa bulletin 132 - november 2007 27

The Formidable

Challenge of Processing

Gaia’s Data

Gaia.qxd 12/12/07 1:33 PM Page 27

moving 3-D Milky Way map ofunprecedented scope and precision, aswell as providing profiles of the physicalproperties of each star, including temp-erature and elemental composition.

By surveying all celestial bodies downto the very faint magnitude of +20, Gaiawill cover a representative fraction ofthe Milky Way’s population, providingscientists with the material to tackleunanswered questions about our homegalaxy, potentially revealing its history,current state and future.

This catch-all survey will naturallyinclude many objects besides stars. Gaiawill spot several thousand brown dwarfs(failed stars, too small to ignite) andextrasolar planets, and map out ourimmediate neighbourhood in greatdetail, detecting hundreds of thousandsof minor Solar System bodies.

Beyond the Milky Way, Gaia willobserve bright objects like supernovaeand quasars, as well as many distantgalaxies.

As a complete sky survey without pre-programmed targets, the discovery

potential of Gaia is profound. Whoknows what it might find?

Through sophisticated processing andanalysis, the raw data will be translatedinto the mission’s final product: theGaia Catalogue, an extensive galacticcensus, rich in scientific content. Theunprecedented accuracy and unbiasednature of this full-sky survey will provevaluable, even revolutionary, to a hugerange of scientific disciplines besidesgalaxy studies. Gaia’s wealth of data willinform and invigorate scientific areas asdiverse as the life cycles of stars, thedistribution of ‘dark matter’ andEinstein’s theory of general relativity.

Selected as an ESA Cornerstonemission in 2000, Gaia is now movingfrom its design phase into development;launch is set for late 2011.

Gaia continues a European traditionin pioneering astrometry, building onthe expertise generated by the very firstspace-based astrometry mission,Hipparcos. Flying between 1989 and1993, Hipparcos mapped 99% of starsdown to magnitude +11 with unprece-

dented accuracy. Its astrometricachievement is still unchallenged. Gaiawill outdo its predecessor by orders ofmagnitude in terms of accuracy,sensitivity and quantity. WhileHipparcos followed a programme ofpre-selected objects to observe, Gaia’ssurvey is complete and unbiased.

Advanced astrometric instrumentsand processing will enable Gaia todiscern the positions of stars and theirminute movements with incredibleaccuracy: around 25 microarcseconds(about 0.00000001º) for stars ofmagnitude +15 – roughly the ability todiscern a single human hair from adistance of 500 km. A magnitude +15star is already extremely faint, around4000 times dimmer than the fainteststars visible to the human eye. Gaia’scensus will be complete for all celestialbodies down to magnitude +20 (about400 000 times dimmer than the faintestnaked-eye stars), which equates toaround 1000 million objects.

Following launch, Gaia will travel1.5 million kilometres from Earth, awayfrom the Sun, to circle the second

Science

esa bulletin 132 - november 2007 www.esa.int28

Gaia’s launch and operations. (ESA/Astrium)

Gaia.qxd 12/12/07 1:34 PM Page 28

Lagrangian point of the Sun-Earthsystem. Known as L2, this is a point inspace where the combined gravitationalpulls of the Earth and Sun balance out,allowing a spacecraft to hold position.Shaded from the light of the Sun, Moonand Earth by a large shield, the satellitewill revolve slowly, with one fullrevolution every 6 h, thereby scanninggreat circles across the sky.

Inside the satellite, Gaia’s instrumentsare mounted on a hexagonal opticalbench. Two telescopes sharing a focalplane look out through apertures in the

payload housing. The two viewingdirections are separated by a fixed basicangle. Light from a celestial objectenters through an aperture, and strikesthe large primary mirror opposite(either M1 or M’1).

The light is bounced by a series ofmirrors along a total focal length of35 m, with the two paths meeting at theM4/M’4 beam combiner before finallyreaching the shared focal plane. Havingtwo different lines of sight allows Gaiato measure the relative separations ofthe thousands of stars simultaneously

esa bulletin 132 - november 2007www.esa.int 29

Gaia

The complex arrangement of Gaia’s mirrors. LOS is the line-of-sight, the viewing directions of the telescopes. (EADS Astrium)

The focal plane assembly: a large mosaic of 106 CCDs amounting to nearly 1000 million pixels. (EADS Astrium)

Gaia deploys its sunshield. (ESA/C. Carreau)

The ‘scanning law’ that allows Gaia to view the entire sky in aseries of overlapping great circles

Gaia.qxd 12/12/07 1:34 PM Page 29

present in the combined fields of view.These wide-angle measurements buildup a rigid network of relative starpositions, contributing to Gaia’s except-ional accuracy. The ‘folded-up’ focallength of 35 m is required to attainGaia’s high resolution.

At the focal plane is a large mosaic ofsophisticated, custom-built chargecoupled devices (CCDs), light detectorsof essentially the same kind as found ina digital camera. Containing 106 CCDs,the focal plane assembly comprises atotal of nearly 1000 million pixels (a‘gigapixel’), compared to the few millionof a typical digital camera.

As Gaia slowly rotates, the image of acelestial object crosses the focal plane.The path it takes is recorded as astro-metric data. Red and blue photometryprisms between the beam combiner andthe focal plane enable Gaia to measurethe brightness of an object in a numberof ‘colours’, or bands of wavelength.

A radial velocity spectrometergrating, also located between the beamcombiner and focal plane, will performspectrometry: measuring an object’sspectrum of light, in this case specifi-cally to observe how spectral lines areshifted, from which the object’s radialvelocity can be inferred.

This arrangement means that the focalplane used for astrometry also serves thephotometric and spectroscopic instru-ments, with certain CCDs assignedspecifically to photometry and spectro-

scopy. The instruments steadily scan thesky as Gaia spins and gradually‘precesses’ (the spin axis itself moves in acircle every 63 days). In this way, thewhole sky is eventually covered, witheach part being observed around 70times in the course of Gaia’s lifetime.

The Gaia Data ChallengeBy its very nature, Gaia will acquire anenormous quantity of complex,extremely precise data, representing the

observations of 1000 million diverseobjects dozens of times each by a‘double vision’ instrument that isspinning and precessing. These data willbe transmitted daily to a ground station,either Cebreros in Spain, or New Norciain Australia. Both have dishes 35 m indiameter to catch the faint radio whisperfrom space. Downlink speeds andstation visibility constraints will result ina daily raw telemetry volume of roughly50 Gbytes. By the end of Gaia’soperational life, around 100 terabytes(1014 bytes) will have been channelled toEarth: the equivalent of more than20 000 DVDs, and some 1000 times theraw volume from Hipparcos.

In its original format, Gaia’s data isunintelligible. Not only because it issqueezed into packets by numericalcoding but also because of the way Gaiascans the sky, picking up fragments ofinformation at each transit of the1000 million objects.

It is the combination of data complex-ity and sheer volume that presents sucha demanding task: extensive, sophistica-ted treatment is required to yieldmeaningful results, by piecing togetherthe fragments and translating them into

Science

esa bulletin 132 - november 2007 www.esa.int30

DPAC membership is distributed across more than 15 countries

The satellite with payloadhousing removed to reveal theinstruments inside (EADS Astrium)

Gaia.qxd 12/12/07 1:35 PM Page 30

real positions, motions and otherphysical information.

In terms of dedicated computingpower alone, Gaia’s requirements areconsiderable. The centre of an imagemoving across the focal plane needs tobe determined to within 1% of a pixelsize. With several thousand millionimages registering over the course of the5-year lifetime, if each one took a meremillisecond to process, it would stillrequire about 30 years to deal with theentire set one image at a time. With theaim of releasing a final Gaia Catalogueonly a few years after the satellite hasfinished collecting data, the need fordistributed or parallel processing isclear.

The data treatment, as well as theinstruments themselves, ensures thatGaia performs as accurately as intended.The orientation of the spacecraft mustbe known to microarcsecond accuracyabout all three principal axes. Not onlymust Gaia be extremely physically stable,but the data processing must also workto this highly precise scale.

Microarcsecond-astrometry raises afurther unique challenge: at this level,ordinarily negligible ‘relativistic’ effectscome into play. According to generalrelativity, the path of light is bent by thegravitational attraction of a massivebody such as the Sun. This bendingappears to shift a star from its trueposition. At Gaia’s level of accuracy,this is constantly significant, distorting

the apparent positions of celestialbodies across the entire sky. Highlyprecise relativistic corrections musttherefore be calculated and introducedinto the data processing.

It is clear that the Gaia data challengeis an enormous task in terms ofexpertise, effort and dedicatedcomputing power. In late 2006, ESA’sAnnouncement of Opportunity forGaia’s data processing was released,calling for a proposal to build andoperate a single processing pipelineleading to the intermediate and finalmission products. The announcementexpected the system to be developed as acollaboration between ESA’s GaiaScience Operations Centre and a broadscientific community supported bynational funding agencies.

In response, a large pan-Europeanteam of expert scientists and softwaredevelopers submitted their proposal fora comprehensive system capable ofhandling the full size and complexity ofthe Gaia data. In May 2007, ESA’sScience Programme Committeeapproved the proposal put forward bythe Data Processing and Analysis

Consortium (DPAC). At this point,DPAC became officially responsible forGaia data processing and analysis.

DPACDPAC is a collaboration drawing itsmembership from all over Europe,including a diverse community of morethan 300 scientists and softwareengineers, spread throughout more than15 countries, and six large DataProcessing Centres. The consortiumbrings together skills and expertise fromacross the continent; its internationalnature and cooperative spirit reflectsthat of ESA itself.

The consortium is divided intospecialist units known as CoordinationUnits (CUs). These are the buildingblocks of DPAC, with each unitassigned a unique set of processingtasks. They are supported by the DataProcessing Centres (DPCs), the centreswith the actual computer hardware.While the CUs are structured fordeveloping the software, each is closelyassociated with at least one DPC, wheretheir ‘algorithms’ are actually used.

Besides the technical challenge facedby DPAC, the sociological challenge ofcoordinating the efforts of 300 peopleundertaking an unprecedented feat indata processing for astronomy is not tobe underestimated. Effective communi-cation, organisation and maintainingmotivation for the large consortium asessential.

esa bulletin 132 - november 2007www.esa.int 31

Gaia

The DPAC organigram shows the relation of the Coordination Units, the Data Processing Centres and the DPAC executive committee (DPACE)

Gaia.qxd 12/12/07 1:35 PM Page 31

The Gaia Data FlowGaia’s processing is broken down intocomponents to facilitate distributeddevelopment, identifying the majorparts of the system that may operaterelatively independently. This approachis also driven by the fact that the systemis to be developed in many countries bya number of teams, each with differentlevels and fields of expertise.

Over the 5-year mission, Gaia willyield a total uncompressed data volumeof around 100 Tbytes. The satellite willhave contact with the ground stationonce a day for an average 11 h, duringwhich an uncompressed volume ofroughly 50 Gbytes will be downlinked ata rate of a few Mbits per second.

Within an hour, the Mission ControlSystem at the Mission OperationsCentre in Germany will receive Gaia’shousekeeping telemetry that reveals thehealth of the craft and its payload.Meanwhile, on the Science OperationsCentre (SOC) side in Spain, the sciencetelemetry is received for preliminaryprocessing in the Initial Data Treatment.This is the first step in unravelling thecomplex scientific data; this preliminaryprocessing decodes and decompressesthe telemetry, initially matchesobservations to known objects, andworks out a ‘rough’ (sub-arcsecond)satellite attitude.

The SOC also carries out ‘First Look’processing – this is Gaia’s regular healthcheck. While the Mission ControlSystem does basic system monitoring,the First Look indicates any anomaliesin the science output that can becorrected on board. The Flight ControlTeam resolves any anomalies throughcommands to the satellite.

DPAC focuses on processing the datainto science products rather than thepresentation of the final catalogue.Production of the actual catalogue willbe covered by a future Announcementof Opportunity. The ninth CoordinationUnit, ‘Catalogue Access’ is a place-holder, to be activated at a later date.

Scientific ProcessingAfter the complex raw data from Gaia’s

instruments have been untangled, theintermediate data requires scientificprocessing as the next step towards thefinal data product. Basic meaningfulscientific data will be extracted via‘iterative processing’, with the output of

one cycle forming the input of the next,gradually moving closer to the result.Unlike the near-realtime processingoutlined above, each iteration will takearound 6 months or more. This willcontinue after the satellite has finished

Science

esa bulletin 132 - november 2007 www.esa.int32

Gaia.qxd 12/12/07 1:36 PM Page 32

its job, with each CU sending its outputto the overarching Main Database.

The Main Database is the hub of allthe data in the system. It is planned thatthis database will be ‘versioned’ every6 months or so, with the final Gaia

Catalogue being derived from by about2020.

Since the science processing is itera-tive, each new version of the databasewill be derived from the precedingversion. On detecting something ofimmediate interest to the broaderscientific community, DPCs may issueoccasional ‘science alerts’ in the courseof the processing. These will be sent tothe SOC for immediate dissemination.

One of the most intensive tasks is the‘astrometric core solution’, which is keyto unlocking the data potential of Gaia.This produces the calibration data andattitude solution needed for all the othertreatments, as well as the accurate posi-tions and motions of about 100 millionwell-behaved primary objects. After thisis performed and the products stored inthe central database, the more special-ised tasks can begin: photometric andspectroscopic processing, and thetreatment of all the difficult objects,such as multiple stars.

Further specialised analyses deal withthe types of variable stars and retrievingstellar astrophysical information such astemperature and chemical composition.

The Astrometric Core SolutionThe vital astrometric core solution isobtained via the Astrometric GlobalIterative Solution (AGIS). This is thefoundation of DPAC’s scheme forreconstructing results from raw data, aprocedure that iteratively adjustsunknown parameters to converge grad-ually on an optimal solution. Variants ofthe Global Iterative Solution will beused in the photometric and spectro-scopic processing.

AGIS will be applied to around 10%of the objects – that means 100 millionobjects, and hundreds of millions ofunknowns. AGIS begins by calculatingastrometric parameters (positions andmotions) for the predictable, well-behaved primary objects by firstlyassuming that attitude and calibrationparameters are known. Assuming thenthat the astrometric and calibrationparameters are known, the attitude isestimated. Then, with the assumption

that astrometric and attitude parametersare known, calibration parameters canbe estimated.

The whole sequence will be repeatedseveral times in the course of theprocessing, perhaps once every 6 monthsduring the initial accumulation ofobservations. Iteration is necessary asmany times as it takes to converge upona solution that fits the data.

ConclusionDPAC has a pivotal role in fulfillingGaia’s potential. Excellent progress hasalready been made and further challeng-ing work still lies ahead. In the course ofthe processing, intermediate datareleases with valuable scientific contentare expected. When the Gaia Catalogueis finally published in around 2020,DPAC’s work will be complete, andGaia’s processed data will be made freelyavailable for investigation by the world’sentire scientific community. e

esa bulletin 132 - november 2007www.esa.int 33

Gaia

Further information on Gaia and its mission can befound at http://www.esa.int/science/gaia

Key terms

Astrometry: the precise measurement of the positions

and motions of celestial bodies.

Photometry: the measurement of the brightness of a

celestial body over wide bands of wavelength.

Spectrometry: the measurement of the spectrum of

light emitted by a celestial body.

Radial velocity: the speed at which a celestial body

approaches or recedes from the observer.

Magnitude: used here to mean ‘apparent magnitude’,

this is the brightness of an object as seen by the

observer. The scale is negative and logarithmic: the

higher the number, the dimmer the object appears.

The Sun has an apparent magnitude of –26.7; the

next brightest star in the sky, Sirius, is –1.4. The

dimmest stars our eyes can see are around +6.

Dark matter: hypothetical matter of uncertain

composition that potentially accounts for the

majority of mass in the observable Universe. Its

presence is inferred from gravitational effects on

visible matter; neither emitting nor reflecting light, it

cannot be observed directly.

Quasar: an extremely bright, active core of a very

distant young galaxy.

Supernova: an exceptionally bright explosion at the

death of a massive star.

An artist’s impression of our Galaxy (R. Hurt/JPL-Caltech/NASA)

Gaia.qxd 12/12/07 1:36 PM Page 33