Numerical Simulations of Colliding Black HolesGeneral Motivation: why is this so important now?Understanding Einsteins TheoryGravitational Wave AstronomySolving Einsteins EquationsCoupled Elliptic-Hyperbolic systemDriver for Computational Physics Dev.Fine interplay between analytic and numerical methods is crucialTwo Examples: New physics enabled by Numerical RelativityPure Gravitational WavesBlack HolesEd SeidelLSU &Albert-Einstein-Institut+ lots of collaborators...

Einsteins Equations and Gravitational WavesTwo major motivations/directions for numerical relativityExploring Einsteins General RelativityWant to develop theoretical lab to probe this fundamental theoryFundamental theory of Physics (Gravity)Among most complex equations of physicsDozens of coupled, nonlinear hyperbolic-elliptic equations with 1000s of termsBarely have capability to solve after a centuryPredict black holes, gravitational waves, etc, but want much moreExciting new field about to be born: Gravitational Wave AstronomyLIGO, VIRGO, GEO, LISA, ~ $1 Billion worldwide!Fundamentally new information about UniverseA last major test of Einsteins theory: do they exist?Eddington: Gravitational waves propagate at the speed of thought1993 Nobel Prize Committee: Hulse-Taylor Pulsar (indirect evidence)20xx Nobel Committee: ??? (For actual detection)One century later, both of these developments happening at the same time: very exciting coincidence!

High Capacity Computing: Want to Compute What Happens in Nature!

Yes, but when will that ever happen?Black Holes form in many waysFrom collapse of large stars, perhaps after massive SN explosionMany systems in binaries, but these may not lastGlobular Clusters may be good factories for creating binary BHsBinary Objects will slowly coalesce due to emission of wavesCarry away energy and angular momentumDoes not happen very much in your backyardBut, look into large volume of UniverseEvent rate goes up as distance D3Estimates of events indicate could see a few per year in range of LIGOBinary BHs considered one of strongest known sourcesNews Flash!!

Einsteins Equations for Numerical RelativityBasic object is the metricPythagorean Theoremds2 = dx2 + dy2 + dz2Special Relativityds2 = -dt2 + dx2 + dy2 + dz2 = -dt2 + S gijdxidxjGeneral Relativityds2 = -a2dt2 + gij(dxi + bidt)(dxj + bjdt) a, bj are gauge quantities (chosen at will) gij is the metric of the time slice, determined by Einsteins equationsWe speak of curved space becausein a gravitational field the geometry is distortedGeneralize Newtons TheoryBut much more complexEddington: Whos the third?dxdyds dx2 + dy2 + dz2

Numerical RelativityMaxwells equationsConstraint Equations: Div B = Div E = 0Evolution Equations: E/t = Curl B, B/t = - Curl EGauge Condition: A ---> A + LEinstein Eqs.: Gmn(gij) = 8pTmn Ignore matter!Shakespeare: Much ado about nothingConstraint Equations: 4 coupled elliptic equations for initial data and beyondFamiliar from Newton: 2 f = 4pr12 evolution equations for gij, Kij (gij /t)Like a wave equation 2f/dt2 -2 f = Source (f, f2, f)4 gauge conditionsSimple Solution: ds2 = -(1-2M/r)dt2 + dr2/(1-2M/r) + r2 dW2Schwarzschild solution for spherical, static black holeWant much more than this!

Einstein Equations (in axisymmetry!!!)

Important Remarks about Evolution EquationsMajor research directionsConstraints:You are NOT free to impose initial dataThey must be solved somehow(how?)They must be preserved during evolution(how?)Evolution equations:Very ugly, unrecognizable form, but sort of wave-likeMany different ways to write them, all analytically equivalent:Physics: Interpretation very difficult since evolved quantities are coordinate and gauge dependent!Very difficult art: requires blend of theoretical physics, applied mathematics, computational experiment

Computational Needs for 3D Numerical Relativity Get physicists + CS people together Initial Data: 4 coupled nonlinear elliptics Choose Gauge Evolution hyperbolic evolution coupled with elliptic eqs. 100-200 3D arrays, TByte, Tflop crucial Interpret PhysicsMajor Goal: make this all possible for non-numerical relativist

Scientific Computing Power Increase: Astounding!Prehistoric Supercomputing ~19801 Professor + 1 HP calculator = 1 FLOP and 1 Byte memory

Traditional Supercomputing ~19901 Postdoc + 1 Cray X-MP = 1MFLOP and 1 Mbyte memory

Teraflop Supercomputing ~20001 grad student + 1000 node IA-64 cluster, Cray, etc

2010 and beyond??? Use your imagination...Students take note: almost all previous numerical relativity was done on machines orders of magnitude smaller than this!!

Grand Challenge CollaborationsScience and Eng. Go Large Scale: Needs Dwarf Capabilities NSF Black Hole Grand Challenge8 US Institutions, 5 years, $4MSolve problem of colliding black holes (try)Examples of Future of Science & EngineeringRequire Large Scale Simulations, and collaborationsBring together cross disciplinary teamsNew Era for each fieldSolving problems simply beyond reach until now

Building Communities through CodesCactus FrameworkGoddardPenn StateWash UAEITACTbingenSouthamptonSISSAThessalonikiClimate Modeling(NASA,Utrecht+)Chemical Engineering (U.Kansas)Bio-Informatics(Chicago/Canada)Astrophysics(Zeus/Innsbruck)EU AstrophysicsNetworkNumerical RelativityOther ApplicationsPortsmouthRIKENPittsburghQuantum Gravity(Hamilton)LSUTexas, BrownsvilleCFD(KISTI)GarchingLinear Algebra(Lebanon)Prototype App(Many CS Projects)BenchMarkingUNAMPlasma Physics(Princeton)Texas, Austin????

Two Little Lessons in Black Hole PhysicsBlack Hole Perturbation TheoryWhat happens when a BH is perturbed?How can this be used in numerical work?Going beyond linear theory through numerical relativity...Black Hole HorizonsEvent vs. Apparent HorizonHow to find/use in numerical relativity?New Tool for studying physics provided by numerical relativity...

What happens when a black hole is perturbed?Take existing black hole (Schwarzschild solution)Analytic solution known since 1916!Perturb by tossing pebble into itIs it stable?Perturb metric tensor gmuds2 = -(1-2M/r) dt2 + dr2/(1-2M/r) + r2 (dq2 + sin2qdf2)by adding ehmn(Ylm), expand to first order in e...Discover (after 40 years study!) Scattering off potential barrierFind resonance frequencies of this potential barrier y ~ exp(iw(t-x)) w resonances are complex: get damped sinusoidsFrequency depends on mass and spin of BH, thats all! In all theoretical studies, these modes are excitedCollapse of matter to a BH, Distorted, Colliding BHs...

Black holepebbleHorizon

Black Hole Normal ModesRinging modes: damped sinusoid ~eiwt Measure this, can determine mass and spin of BH that created it Wavelength, damping uniquely determine spin, mass of BH

Questions, Questions: what happens when we go beyond linear theory?

Black Hole HorizonsEvent Horizon: outgoing light rays that never escape, never hit singularityApparent Horizon: surface of light rays that are instantaneously neither expanding nor contracting timespaceSingularityWhat is shape of BH?Can measure their properties:Area: M2 = Area/(16p) GeometryCurvatureEmbedding ShapeHorizon generatorsEtcLots of physics in here!Now have means to study dynamic BHs for first time

Event Horizons of Dynamic Black Holes:Can now compute, study numericallyPair of Pants of colliding black holes.Sketched by Hawking 25 years ago, now computed quantitatively...Rotating BH Horizon: time sequenceColor map brings out gaussianCurvature, shape can determine rotation rate!

Evolving Pure Gravitational Waves:Probing Einsteins theory with a supercomputerEinsteins equations nonlinear, so low amplitude waves just propagate away, but large amplitude waves mayScatter off themselvesCollapse on themselves under their own self-gravityCollapse and actually form black holes Must use numerical relativity: Probe GR in highly nonlinear regimeForm BH?, Critical Phenomena in 3D?, Naked singularities? Little known about generic 3D behaviorTake Brill Wave Initial datads2 = y4 (e2q(dr2+dz2)+ r2df2)q = A f(r,z,f), choose time derivative Kij (take time symmetry for now)Large amplitude: get BH to form! Find horizon, measure its geometry, find normal modes, etcBelow critical value: disperses and can evolve forever as system returns to flat spaceWe see hints of critical phenomena, known from nonlinear dynamics

Subcritical Waves: Everything radiates away...Newman-Penrose Y4 (showing gravitational waves)with lapse a underneathAlcubierre, ES, et al, results Could not do this before 1998

New formulations of EEs crucial for this.

Collapsing Gravitational (Brill) WavesProbing Einsteins Theory with SupercomputersY4 showing collapse to BH, Apparent Horizon with gaussian curvature

Fourier Decomposition of Waves emitted:Full 3D simulation shows Quasinormal mode of BH after formationl=2,m=2 wave extractionQNM fit to two lowestmodes for BH ofappropriate massBH formingFormed BH RingingNumericalwaveform

Why are Black Holes so Difficult?How Standard Numrel breaks down.Standard Numrel:ADM formulations unstableGauge conditions unstudiedStretching of hypersurfaces leads to pathologies in metric functions

Why not evolve just the exterior?Horizon is Causal BoundaryPhysics information only ingoingHorizon

Colliding BH Roadmap:A patchwork to successTimePost NewtinspiralFinal PlungeRingdownt ~ 500+ Mt ~ 100 Mt ~ 30 MMerger PhaseTime ScaleTechnique

Start with Traditional ApproachColliding Black Holes: First 3D SimulationsHead on, equal mass BHs at rest: Misner dataThe classic numerical rel. problemHahn & Lindquist 1964Smarr PhD thesis 1970s2D NCSA/WashU 1990sNew Physics: waveforms, horizons, perturbation theoryBut, long way to go...Axisymmetric version

How to evolve these systems?I. Full Numerical RelativityProvide Cauchy data for gij, KijEvolve2D3DFull CoalescenceII. Linearized TheoryProvide Cauchy data for y, y-dot Zerilli/Regge-WheelerTeukolskyEasy, but limited validity

Approach I: Full 3D Numerical EvolutionHead-on, Equal Mass, BH Collisions (Misner Data) in 3DEvent Horizon shown in green.Y4 (representing gravitational waves) shown in blue-yellow

Approach II. Comparison of Close Limit Perturbative Approach and full Numerical RelativityTake initial data as Cauchy problem for perturbation theoryEvolve it using perturbative equationsCompare to full numerical approach

Amazing!!Pert theory breaks down if BHs farther apartWe understand Misner extremely wellWell extend this later...

Grazing Collisions of 2 BHs:Extract physics, predict rotation and mass of final BHConvergence of waveformFit to Kerr QNMst/Mt/M

Initial Data for Orbits1st Time ever: study orbital initial data via full 3D simulation

Cook-Baumgarte dataInnermost Stable Circular Orbit(ISCO, labeled by 0)Other circular Orbits, tooPre-ISCO 1-10Focus on Pre-ISCO 3Major Issue: BH data sets do not want to orbit!Numerical shows inadequacy of dataCoalesce just after free fall time

Evolution of Pre-ISC0 3

Are these BHs Orbiting? No!

Coalescence Time for Pre-ISCO Sequence

Coalescence Times for Pre-ISCO SequencePI-0123PI-4

Analysis of Final BH for ISCO caseHorizon AnalysisStudy circumfernece geometryDetermine rotationCompute area ---> MassQNMs excitedCompare final BH properties to total mass, ang. mom. of spacetimeCompute energy, ang mom radiatedFind few % Mtot

Many different BBH ISCOs!BaumgarteISCOMeudonISCODamour et al

Merger Times for Simulations as of 18 Months AgoOnly 1-2 groups could do thisPre-ISCO sequence w/abovePrevious uncertain data reveal transition regionConsistent w/ Bernd et al, Meudon ISCOHigh Resolution, AMR, understanding of gauges requiredBrgmann result

Meudon Orbits: Koppitz PhD Thesis

Interacting with Running JobsRemote Viz dataHTTPStreaming HDF5AutodownsampleAmiraAny Viz Client:LCA Vision, OpenDXChanging any steerable parameter Parameters Physics, algorithms PerformanceRemote Viz data

Global Grid Testbed Collaboration5 continentsNorth America, Europe, Asia, Africa, Australia (almost had South America)Over 14 countries, including China, Japan, Singapore, S. Korea, Egypt, Australia, Canada, Germany, UK, Netherlands, Czech, Hungary, Poland, USAAbout 70 machines, thousands of processors (~7500) Many hardware types, including PS2, IA32, IA64, MIPS, IBM Power, Alpha, Hitachi/PPC, SparcMany OSs, including Linux, Irix, AIX, OSF, True64, Solaris, HitachiMany organizationsDOE, NSF, MPG, Universities, VendorsAnd yet: all run same Grid infrastructure, and many can be used for applications

The Grid Testbed and Statushttp://scb.ics.muni.cz/static/SC2002/Testbed.htmlhttp://scb.ics.muni.cz/static/SC2002/status/history/status.html

See http://sc2002.aei.mpg.de

Recent Progress in Community: Exploring Orbits!Frans Pretorius (PRL 95:121101, 2005 and gr-qc/0602115) uses a generalized harmonic formulation, collapsing scalar field initial data, compactified coordinate system placing the outer boundaries at spatial infinity, AMR, moving excision.Diener et. al (gr-qc/0512108, PRL) uses the BSSN formulation, puncture initial data, fixed excision, adaptive comoving coordinates, a slightly modified -driver shift and fixed mesh refinement. Baker et. al (gr-qc/0511103 and gr-qc/0602026) and Campanelli et. al (gr- qc/0511048 and PRD 73:061501, 2006) independently came up with very similar schemes to evolve moving punctures using no excision, BSSN and slightly modified 1+log and -driver gauge conditions. Baker et. al uses AMR (Paramesh).

Pretorius ResultsEvolutions of collapsing scalar field data with different initial boosts of the scalar field distributions. The orbital characteristics and waveforms are extremely sensitive to the initial data boost.

BrownsvilleD3.0 with an initial proper separation of L=9.32. Estimated common AH time is T=125MRemarkable agreement between the plunge part of the QC0 and D3.0 waveforms.

LSU/AEI ResultsOrbital dynamics depend sensitively on gauge and resolutionIt is possible to reconcile apparently different results at high enough resolutionResolved discrepancy with results of Bruegmann, LSU/AEI

LSU/AEI ResultsDuration of last orbit before AH formation is 59M. Common apparent horizon formation at T=124M. Angular velocity at merger agrees with half quasinormal mode frequency of the final BH: period of 15.9M corresponds to =0.395 (pointed out by Steve Detweiler).

Goddard ResultsSurprisingly good overlap between late time orbital dynamics as well as the shifted waveforms.

The FutureGravitational wave astronomy almost here: must be able to solve Einsteins equations to understand the new observationsNumerical treatment will be a most important tool for studying Einsteins theory (and other theories)Need new computational and numerical algorithmsComputational Science, Grids (www.gridforum.org)Synergies with many analytic methodsNew Physics Now, much more coming...Collapse of 3D gravitational waves to BHPatching together 3D simulations for Gravitational Wave AstronomyHorizon dynamics, physics3D Neutron Star simulations, relativistic astrophysics are just now possible: bright futureNumerical Relativity in early maturity phase:Simple ideas, coupled with advanced computational infrastructure, still have large impactjean-luc.aei.mpg.de for more info

Demonstrate httpd, parameter steering, iPaq, OpenDX. Need several people to do this: me, to give talk,Denis, to show OpenDX (extra screen), someone elseTo play with iPaq and pass around.