Steven Bamford

An introduction to scientific programming with

http://stevenbamford.com/pythoncourse

•  Part 1: Introduction •  Why (and why not) to use a modern, high-level, scripting language •  Why Python is awesome •  Introduction to the language

•  startup, syntax, constructs, functions, classes, getting help •  Good programming practice versus 'thinking aloud’ •  Python 2.x versus 3.x

•  Part 2: Scientific programming in Python •  Arrays – Numerical Python •  Using arrays wisely •  Plotting •  Scientific Python

•  optimisation, interpolation, statistics, filtering, integration, … •  Other tools

•  GNU Scientific Library, R, … •  An example analysis program

•  Part 3: Python for observers •  Handling FITS files •  PyRAF – scripting IRAF with Python

Part 1: Introduction

An introduction to scientific programming with

•  Modern scripting languages: •  Python, Perl, Ruby, IDL, … •  High-level •  Interactive interpreter

•  Ease of use

•  Speed of development

•  Encourages scripting, rather than one-off analysis

•  Permanent record

•  Repeatability

•  If you want fastest possible performance

•  Highly parallel code

•  Need low-level control

•  Designed to be easy to learn and use – clear syntax

•  Well documented

•  Powerful, flexible, fully-featured programming language

•  ‘Batteries included’

•  Comprehensive scientific tools

•  Fast

•  Interpreter, introspection

•  Runs everywhere

•  Completely free

•  You already have it

•  Less stress

•  Get more science done

•  Widely used and growing popularity

•  Throughout academia and industry •  NASA, AstraZeneca, Google, Industrial Light & Magic, Philips,… •  Web services, engineering, science, air traffic control, quantitative finance,

games, education, data management, …

•  Python programmers in demand

•  Easy introduction to general programming concepts

Why not?

•  Existing code for your project in another language, but still…

http://python.org

>>> 2+2 4 >>> # This is a comment ... 2+2 4 >>> 2+2.0 # and a comment on the same line as code 4.0 >>> (50-5*6)/4 5 >>> width = 20 # assignment, no type declaration >>> height = 5*9 >>> width * height 900 >>> x = y = z = 0 # zero x, y and z >>> y 0 >>> n Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'n' is not defined

2+2 # This is a comment 2+2 2+2.0 # and a comment on the same line as code (50-5*6)/4 width = 20 # assignment, no type declaration height = 5*9 width * height x = y = z = 0 # zero x, y and z y

•  Can write in a text editor and copy and paste into interpreter

•  Can save and execute from command line: $ python test.py

•  Can save and use interactively in future sessions (import)

>>> 'spam and eggs' 'spam and eggs' >>> 'doesn\'t' "doesn't" >>> "doesn't" "doesn't" >>> '"Yes," he said.' '"Yes," he said.’ >>> hello = ‘Greetings!’ >>> hello ‘Greetings!’ >>> print(hello) Greetings! >>> print(hello + ‘ How do you do?’) Greetings! How do you do? >>> print(hello, ‘How do you do?’) Greetings! How do you do? >>> howdo = ’How do you do?’ >>> print(hello+’ ‘+howdo) Greetings! How do you do?

>>> 10 + 3 13 >>> 10 - 3 7 >>> 10 * 3 30 >>> 10 / 3 3 >>> 10 // 3 3 >>> 10 % 3 1 >>> 10**3 1000 >>> 10 + 3 * 5 # *,/ then +,- 25 >>> (10 + 3) * 5 65 >>> -1**2 # -(1**2) -1

>>> 10.0 + 3.0 13.0 >>> 10.0 - 3.0 7.0 >>> 10.0 * 3 30.0 >>> 10.0 / 3 3.3333333333333335 >>> 10.0 // 3 3.0 >>> 10.0 % 3.0 1.0 >>> 10.0**3 1000.0

>>> 4.2 + 3.14 7.3399999999999999 >>> 4.2 * 3.14 13.188000000000001

Augmented assignment: >>> a = 20 >>> a += 8 >>> a 28 >>> a /= 8.0 >>> a 3.5

Functions: >>> abs(-5.2) 5.2 >>> sqrt(25) 5.0

Comparisons: >>> 5 * 2 == 4 + 6 True >>> 0.12 * 2 == 0.1 + 0.14 False >>> a = 0.12 * 2; b = 0.1 + 0.14 >>> eps = 0.0001 >>> a - eps < b < a + eps True

Lists: >>> a = [1, 2, 4, 8, 16] # list of ints >>> c = [4, ‘candles’, 4.0, ‘handles’] # can mix types >>> c[1] ‘candles’ >>> c[2] = ‘knife’ >>> c[-1] # negative indices count from end ‘handles’ >>> c[1:3] # slicing [‘candles’, ‘knife’] >>> c[2:] # omitting defaults to start or end [‘knife’, ‘handles’] >>> c[0:4:2] # variable stride (could just write c[::2]) [4, ‘knife’] >>> a + c # concatenate [1, 2, 4, 8, 16, 4, ‘candles’, ‘knife’, ‘handles’] >>> len(a) 5

Tuples: >>> q = (1, 2, 4, 8, 16) # tuple of ints >>> r = (4, ‘candles’, 4.0, ‘handles’) # can mix types >>> s = (‘lonely’,) # singleton >>> t = () # empty >>> r[1] ‘candles’ >>> r[2] = ‘knife’ # cannot change tuples Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment

>>> u = 3, 2, 1 # parentheses not necessary

>>> v, w = ‘this’, ‘that >>> v ‘this’ >>> w ‘that’

Dictionaries: >>> a = {‘eyecolour’: ‘blue’, ‘height’: 152.0, 42: ‘the answer‘} >>> a[‘age’] = 28 >>> a {42: 'the answer', 'age': 28, 'eyecolour': 'blue', 'height': 152.0}

>>> del(a[‘height’]) >>> a {42: 'the answer', 'age': 28, 'eyecolour': 'blue'}

>>> b = {} >>> b[‘hello’] = ‘Hi!’

>>> a.keys() [42, 'age’, 'eyecolour’] >>> a.values() ['the answer’, 28, 'blue’]

>>> a = 4; b = 3 >>> if a > b: ... result = ‘bigger’ ... c = a - b ... >>> print(result, c) bigger 1

>>> a = 1; b = 3 >>> if a > b: ... result = ‘bigger’ ... elif a == b: ... result = ‘same’ ... else: # i.e. a < b ... result = ‘smaller’ ... >>> print(result) smaller

>>> if a < b: print ‘ok’ ok

Comparison operators:

== != > < >= <= is is not in not in

Boolean operators:

and or not

•  Indentation is important! •  be consistent •  use four spaces •  do not use tabs

>>> if ‘Steven’ in [‘Bob’, ‘Amy’, ‘Steven’, ‘Fred’]: ... print ‘Here!’ ... Here!

>>> if ‘Carol’ not in [‘Bob’, ‘Amy’, ‘Steven’, ‘Fred’]: ... print ‘Away!’ ... Away!

>>> test = a == b >>> if test: print ‘Equal’ ‘Equal’

>>> a = b = 0 >>> while a < 10: ... a += 3 ... print(a) ... 3 6 9 12

>>> while True: ... b += 3 ... if b >= 10: break ... print(b) 3 6 9

>>> for i in [2, 5, 3]: ... print(i**2) 4 25 9

>>> for j in range(5): print(j) 0 1 2 3 4

>>> range(3, 10, 2) [3,5,7,9]

>>> d = {‘this’: 2, ‘that’: 7} >>> for k, v in d.items(): ... print(‘%s is %i’%(k, v)) this is 2 that is 7

>>> def my_func(x, y=0.0, z=1.0): ... a = x + y ... b = a * z ... return b ... >>> my_func(1.0, 3.0, 2.0) 8.0 >>> my_func(1.0, 3.0) 4.0 >>> my_func(1.0, y=3.0) 4.0 >>> my_func(5.0) 5.0 >>> my_func(2.0, z=3.0) 6.0 >>> my_func(x=2.0, z=3.0) 6.0

>>> a = [2, 5, 3, 6, 5] >>> a.sort() >>> print(a) [2, 3, 5, 5, 6] >>> a.count(3) 1 >>> a.count(5) 2 >>> a.reverse() >>> print(a) [6, 5, 5, 3, 2]

>>> d = {‘black’: 100, ‘grey’: 50, ‘white’: 0} >>> d.values() [0, 50, 100]

>>> s = ‘-‘.join((‘2009’, ‘07’, ‘07’)) >>> print(s) 2009-07-07

•  A neat way of creating lists (and arrays) without writing a loop

my_fav_num = [3, 17, 22, 46, 71, 8]

even_squared = [] for n in my_fav_num: if n%2 == 0: even_squared.append(n**2)

# in one line: even_better = [n**2 for n in my_fav_num if n%2 == 0]

# both produce [484, 2116, 64]

freshfruit = [' banana', ' loganberry ', 'passion fruit ']

stripped = [weapon.strip() for weapon in freshfruit]

print(stripped)

['banana', 'loganberry', 'passion fruit']

>>> class MyClass: ... def __init__(self, x, y=1.0) ... self.my_x = x ... self.my_y = y ... ... def product(): ... return x*y ... >>> m = MyClass(2, 4) >>> m.product() 8 >>> p = MyClass(5) >>> p.product() 5 >>> m.product() 8

>>> import math >>> math.cos(math.pi) -1.0

>>> math.cos(pi) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'pi' is not defined

>>> from math import cos, pi >>> cos(pi) -1.0

>>> from math import *

•  Modules can contain any code

•  Classes, functions, definitions, immediately executed code

•  Can be imported in own namespace, or into the global namespace

>>> # My totally wicked function >>> def my_func(x, y=0.0, z=1.0): ... ”””This does some stuff. ... For example: >>> my_func(1.0, 3.0, 2.0) 8.0 Yep, it’s that good! ””” ... a = x + y ... b = a * z ... return b ...

•  Comments before function, class, etc. are used to generate help

•  “Docstrings” •  preferred way of documenting code •  can contain examples, which are automatically turned into tests! •  See doctest module

>>> fname = ‘myfile.dat’

>>> f = file(fname) >>> lines = f.readlines() >>> f.close()

>>> f = file(fname) >>> firstline = f.readline() >>> secondline = f.readline()

>>> f = file(fname) >>> for l in f: >>> print l.split()[1] >>> f.close()

>>> outfname = ‘myoutput’ >>> outf = file(outfname, ‘w’) # second argument denotes writable >>> outf.write(‘My very own file\n’) >>> outf.close()

>>> sigma = 6.76/2.354 >>> print(‘sigma is %5.3f metres’%sigma) sigma is 2.872 metres >>> d = {‘bob’: 1.87, ‘fred’: 1.768} >>> for name, height in d.items(): ... print(‘%s is %.2f metres tall’%(name.capitalize(), height)) ... Bob is 1.87 metres tall Fred is 1.77 metres tall

>>> nsweets = range(100) >>> calories = [i * 2.345 for i in nsweets] >>> fout = file(‘sweetinfo.txt’, ‘w’) >>> for i in range(nsweets): ... fout.write(‘%5i %8.3f\n’%(nsweets[i], calories[i])) ... >>> fout.close()

•  Also template system

>>> help(math)

>>> help(math.cos)

>>> a = [1, 2, 3] >>> help(a)

•  Powerful help tools

•  Every object, function, module, … can be inspected

•  New 3.x branch is intentionally backwards incompatible

•  Various improvements, removal of obsolete code, but annoying!

•  print a ➔ print(a)

•  1/3 == 1//3 == 0 ➔ 1/3 == 1.0/3.0 == 1.333…, 1//3 == 0

•  … various others …

•  Version 2.6 contains most backward compatible changes, and can warn of usage (-3 switch) which would be an error in 3.x

•  2to3 (semi-)automated code translator

•  Many useful modules not compatible with 3.x yet

•  Use 2.6.x for now…

•  Compromise between: •  producing results quickly and •  easy reusability and adaptation of code •  code that can be quickly understood by others

•  Comment clearly

•  Use functions

•  Use modules

•  Consider ‘refactoring’ before code gets too messy

•  Science, not software development

http://www.python.org/doc/

(note version numbers)

Part 2: Scientific programming in Python

An introduction to scientific programming with

Course webpage: http://stevenbamford.com/teaching/python_february_2010/

An introduction to scientific programming with

•  Extra features required:

•  fast, multidimensional arrays

•  libraries of reliable, tested scientific functions

•  plotting tools

•  Lists ok for storing small amounts of one-dimensional data

•  But, can’t use directly with arithmetical operators (+, -, *, /, …)

•  Need efficient arrays with arithmetic and better multidimensional tools

•  Numpy

•  Similar to lists, but much more capable, except fixed size

>>> a = [1,3,5,7,9] >>> print(a[2:4]) [5, 7] >>> b = [[1, 3, 5, 7, 9], [2, 4, 6, 8, 10]] >>> print(b[0]) [1, 3, 5, 7, 9] >>> print(b[1][2:4]) [6, 8]

>>> import numpy

>>> import numpy >>> l = [[1, 2, 3], [3, 6, 9], [2, 4, 6]] # create a list >>> a = numpy.array(l) # convert a list to an array >>> print(a) [[1 2 3] [3 6 9] [2 4 6]] >>> a.shape (3, 3) >>> print(a.dtype) # get type of an array int32 >>> print(a[0]) # this is just like a list of lists [1 2 3] >>> print(a[1, 2]) # arrays can be given comma separated indices 9 >>> print(a[1, 1:3]) # and slices [6 9] >>> print(a[:,1]) [2 6 4]

>>> a[1, 2] = 7 >>> print(a) [[1 2 3] [3 6 7] [2 4 6]] >>> a[:, 0] = [0, 9, 8] >>> print(a) [[0 2 3] [9 6 7] [8 4 6]]

>>> b = numpy.zeros(5) >>> print(b) [ 0. 0. 0. 0. 0.] >>> b.dtype dtype(‘float64’) >>> n = 1000 >>> my_int_array = numpy.zeros(n, dtype=numpy.int) >>> my_int_array.dtype dtype(‘int32’)

>>> c = numpy.ones(4) >>> print(c) [ 1. 1. 1. 1. ]

>>> d = numpy.arange(5) # just like range() >>> print(d) [0 1 2 3 4]

>>> d[1] = 9.7 >>> print(d) # arrays keep their type even if elements changed [0 9 2 3 4] >>> print(d*0.4) # operations create a new array, with new type [ 0. 3.6 0.8 1.2 1.6]

>>> d = numpy.arange(5, dtype=numpy.float) >>> print(d) [ 0. 1. 2. 3. 4.]

>>> numpy.arange(3, 7, 0.5) # arbitrary start, stop and step array([ 3. , 3.5, 4. , 4.5, 5. , 5.5, 6. , 6.5])

>>> a = numpy.arange(4.0) >>> b = a * 23.4 >>> c = b/(a+1) >>> c += 10 >>> print c [ 10. 21.7 25.6 27.55]

>>> arr = numpy.arange(100, 200) >>> select = [5, 25, 50, 75, -5] >>> print(arr[select]) # can use integer lists as indices [105, 125, 150, 175, 195]

>>> arr = numpy.arange(10, 20 ) >>> div_by_3 = arr%3 == 0 # comparison produces boolean array >>> print(div_by_3) [ False False True False False True False False True False] >>> print(arr[div_by_3]) # can use boolean lists as indices [12 15 18]

>>> arr.sum() 45 >>> arr.mean() 4.5 >>> arr.std() 2.8722813232690143 >>> arr.max() 9 >>> arr.min() 0 >>> div_by_3.all() False >>> div_by_3.any() True >>> div_by_3.sum() 3 >>> div_by_3.nonzero() (array([2, 5, 8]),)

>>> arr = numpy.array([4.5, 2.3, 6.7, 1.2, 1.8, 5.5]) >>> arr.sort() # acts on array itself >>> print(arr) [ 1.2 1.8 2.3 4.5 5.5 6.7]

>>> x = numpy.array([4.5, 2.3, 6.7, 1.2, 1.8, 5.5]) >>> y = numpy.array([1.5, 2.3, 4.7, 6.2, 7.8, 8.5]) >>> numpy.sort(x) array([ 1.2, 1.8, 2.3, 4.5, 5.5, 6.7]) >>> print(x) [ 4.5 2.3 6.7 1.2 1.8 5.5] >>> s = x.argsort() >>> s array([3, 4, 1, 0, 5, 2]) >>> x[s] array([ 1.2, 1.8, 2.3, 4.5, 5.5, 6.7]) >>> y[s] array([ 6.2, 7.8, 2.3, 1.5, 8.5, 4.7])

•  Most array methods have equivalent functions

•  Ufuncs provide many element-by-element math, trig., etc. operations •  e.g., add(x1, x2), absolute(x), log10(x), sin(x), logical_and(x1, x2)

•  See http://numpy.scipy.org

>>> arr.sum() 45 >>> numpy.sum(arr) 45

•  Array operations are implemented in C or Fortran

•  Optimised algorithms - i.e. fast!

•  Python loops (i.e. for i in a:…) are much slower

•  Prefer array operations over loops, especially when speed important

•  Also produces shorter code, often more readable

•  User friendly, but powerful, plotting capabilites for python

•  http://matplotlib.sourceforge.net/

•  Once installed, to use type:

•  Settings can be customised by editing ~/.matplotlib/matplotlibrc •  default font, colours, layout, etc.

•  Helpful website •  many examples

>>> import pylab

•  Suite of numerical and scientific tools for Python

•  http://scipy.org/

Example modules:

•  Optimisation / minimisation •  http://docs.scipy.org/doc/scipy/reference/optimize.html

•  Integration •  http://docs.scipy.org/doc/scipy/reference/integrate.html

•  Interpolation •  http://docs.scipy.org/doc/scipy/reference/interpolate.html

•  Statistics •  http://docs.scipy.org/doc/scipy/reference/stats.html

•  Image processing •  http://docs.scipy.org/doc/scipy/reference/ndimage.html

•  Python wrappers of GNU Scientific Library functions

•  PyGSL: http://pygsl.sourceforge.net/

•  GSL: http://www.gnu.org/software/gsl/

•  Incomplete documentation for Python functions, but almost all of GSL is wrapped, so refer to GSL documentation.

•  Most functionality implemented in SciPy •  Try SciPy first, if you can’t find what you need try PyGSL •  More comprehensive and sometimes more tested, but less ‘Pythonic’ •  e.g. Monte Carlo integration

•  http://rpy.sourceforge.net/

•  Wraps R – a statistics analysis language •  many advanced stats capabilities but quite specialised

•  http://www.sagemath.org/

•  Python-based mathematics software •  replacement for Maple, Mathematica

•  Planetary nebulae positions and velocities in edge-on galaxy NGC 891

•  Need to: •  read in data •  make plot of positions, colour coded

by velocity •  plot velocity histogram •  fit projected rotation curve to full

dataset

Part 3: Python for observers

An introduction to scientific programming with

•  http://web.mac.com/npirzkal/Scisoft

•  Quick way to install many of the most used data reduction packages: •  IRAF, PyRAF, MIDAS, Python and Python extensions and many more…

•  http://www.stsci.edu/resources/software_hardware/pyraf/stsci_python

•  Astronomy software provided by Space Telescope Science Institute

•  PyFITS

•  PyRAF

•  MultiDrizzle

•  Numdisplay

•  STScI also provide STSDAS and TABLES packages for IRAF

•  http://www.stsci.edu/resources/software_hardware/pyfits

•  Read, write and manipulate all aspects of FITS files •  extensions •  headers •  images •  tables

•  Low-level interface for details

•  High-level functions for quick and easy use

>>> import pyfits >>> imgname = ‘data/2MASS_NGC_0891_K.fits’ >>> img = pyfits.getdata(imgname) >>> img array([[ 0. , 0. , 0. , ..., -999.00860596, -999.00860596, -999.00860596], [-999.00860596, -999.00860596, -999.00860596, ..., -999.00860596, -999.00860596, -999.00860596], [-999.00860596, -999.00860596, -999.00860596, ..., -999.00860596, -999.00860596, -999.00860596], ..., [-999.00860596, -999.00860596, -999.00860596, ..., -999.00860596, -999.00860596, -999.00860596], [-999.00860596, -999.00860596, -999.00860596, ..., -999.00860596, -999.00860596, -999.00860596], [-999.00860596, -999.00860596, -999.00860596, ..., -999.00860596, -999.00860596, -999.00860596]], dtype=float32) >>> img.mean() -8.6610549999999993 >>> img[img > -99].mean() 0.83546290095423026 >>> numpy.median(img) 0.078269213438034058

>>> x = 348; y = 97 >>> delta = 5 >>> print img[y-delta:y+delta+1, ... x-delta:x+delta+1].astype(numpy.int) [[ 1 1 1 1 1 0 0 0 1 0 -2] [ 2 2 4 6 7 7 4 3 1 0 -1] [ 1 4 11 24 40 40 21 7 2 0 0] [ 1 6 23 62 110 107 50 13 2 0 0] [ 2 7 33 91 158 148 68 15 3 0 0] [ 3 7 27 74 123 115 53 12 2 0 0] [ 2 4 12 32 54 51 24 5 1 0 0] [ 1 1 2 7 12 12 5 0 0 0 0] [ 0 0 0 1 2 2 1 0 0 1 0] [ 0 0 0 1 0 0 0 0 0 0 0] [ -1 0 1 0 0 0 0 0 0 0 0]]

•  row = y = first index

•  column = x = second index

•  numbering runs as normal (e.g. in ds9) BUT zero indexed! x

y

>>> tblname = ‘data/N891PNdata.fits’ >>> d = pyfits.getdata(tblname) >>> d.names ('x0', 'y0', 'rah', 'ram', 'ras', 'decd', 'decm', 'decs', 'wvl', 'vel',

'vhel', 'dvel', 'dvel2', 'xL', 'yL', 'xR', 'yR', 'ID', 'radeg', 'decdeg', 'x', 'y')

>>> d.x0 array([ 928.7199707 , 532.61999512, 968.14001465, 519.38000488,… 1838.18994141, 1888.26000977, 1516.2199707 ], dtype=float32)

>>> d.field(‘x0’) # case-insensitive array([ 928.7199707 , 532.61999512, 968.14001465, 519.38000488,… 1838.18994141, 1888.26000977, 1516.2199707 ], dtype=float32)

>>> select = d.x0 < 200 >>> dsel = d[select] # can select rows all together >>> print dsel.x0 [ 183.05000305 165.55000305 138.47999573 158.02999878 140.96000671 192.58000183 157.02999878 160.1499939 161.1000061 136.58999634 175.19000244]

>>> h = pyfits.getheader(imgname) >>> print h SIMPLE = T BITPIX = -32 NAXIS = 2 NAXIS1 = 1000 NAXIS2 = 1200 BLOCKED = T / TAPE MAY BE BLOCKED IN MULTIPLES OF 2880 EXTEND = T / TAPE MAY HAVE STANDARD FITS EXTENSIONS BSCALE = 1. BZERO = 0. ORIGIN = '2MASS ' / 2MASS Survey Camera CTYPE1 = 'RA---SIN' CTYPE2 = 'DEC--SIN' CRPIX1 = 500.5 CRPIX2 = 600.5 CRVAL1 = 35.63922882 CRVAL2 = 42.34915161 CDELT1 = -0.0002777777845 CDELT2 = 0.0002777777845 CROTA2 = 0. EQUINOX = 2000. KMAGZP = 20.07760048 / V3 Photometric zero point calibration COMMENTC= 'CAL updated by T.H. Jarrett, IPAC/Caltech' SIGMA = 1.059334397 / Background Residual RMS noise (dn) COMMENT1= '2MASS mosaic image'

COMMENT2= 'created by T.H. Jarrett, IPAC/Caltech’ >>> h[‘KMAGZP’] 20.077600480000001 # Use h.items() to iterate through all header entries

>>> newimg = sqrt((sky+img)/gain + rd_noise**2) * gain >>> newimg[(sky+img) < 0.0] = 1e10

>>> hdr = h.copy() # copy header from original image >>> hdr.add_comment(‘Calculated noise image’)

>>> filename = ‘sigma.fits’

>>> pyfits.writeto(filename, newimg, hdr) # create new file

>>> pyfits.append(imgname, newimg, hdr) # add a new FITS extension

>>> pyfits.update(filename, newimg, hdr, ext) # update a file

# specifying a header is optional, # if omitted automatically adds minimum header

>>> import pyfits >>> import numpy as np

>>> # create data >>> a1 = numpy.array(['NGC1001', 'NGC1002', 'NGC1003']) >>> a2 = numpy.array([11.1, 12.3, 15.2])

>>> # make list of pyfits Columns >>> cols = [] >>> cols.append(pyfits.Column(name='target', format='20A',

array=a1)) >>> cols.append(pyfits.Column(name='V_mag', format='E', array=a2))

>>> # create HDU and write to file >>> tbhdu=pyfits.new_table(cols) >>> tbhdu.writeto(’table.fits’)

# these examples are for a simple FITS file containing just one # table or image but with a couple more steps can create a file # with any combination of extensions (see the PyFITS manual online)

>>> f = pyfits.open(tblname) >>> f.info() Filename: data/N891PNdata.fits No. Name Type Cards Dimensions Format 0 PRIMARY PrimaryHDU 4 () uint8 1 BinTableHDU 52 223R x 22C [E, E, E, E, E,

E, E, E, E, E, E, E, E, E, E, E, E, E, E, E, E, E]

>>> table = f[1] # data extension number 1 (can also use names)

>>> d = f[1].data # data, same as returned by pyfits.getdata() >>> h = f[1].header # header, same as returned by pyfits.getheader

()

>>> # make any changes >>> f.writeto(othertblname) # writes (with changes) to a new file

>>> f = pyfits.open(tblname, mode=‘update’) # to change same file >>> # make any changes >>> f.flush() # writes changes back to file >>> f.close() # writes changes and closes file

•  http://www.stsci.edu/resources/software_hardware/pyraf

•  Command line to replace cl, allows most normal IRAF commands and Python at same prompt

•  Can use IRAF tasks in Python scripts instead of having to create awkward cl scripts (or worse SPP)

•  Command and filename completion

•  Edit line and access history easily (like ecl or bash)

•  Use just as friendlier cl prompt or use Python whenever you want

•  Transfer data between IRAF tasks and Python

•  Use brackets for tasks when you want it to behave like Python

--> imstat 2MASS_NGC_0891_K.fits # IMAGE NPIX MEAN STDDEV MIN MAX 2MASS_NGC_0891_K.fits 1200000 -8.661 99.44 -1001. 7207.

--> fname = "data/2MASS_NGC_0891_K.fits" --> imstat fname # IMAGE NPIX MEAN STDDEV MIN MAX Error reading image fname ...

--> imstat(fname) # IMAGE NPIX MEAN STDDEV MIN MAX data/2MASS_NGC_0891_K.fits 1200000 -8.661 99.44 -1001. 7207.

•  Many IRAF tasks create output images or tables on disk, or just print to screen, so don’t need to pass information back (see later for this)

stsdas # note can’t unload packages improject(sky_file_2D, sky_file_1D, projaxis=2, average='no') imcalc(sky_file_1D, sky_file_1D, 'im1*%f'%apwidthratio) # calculate effective gain and ron due to combined images gain = 1.91; ron = 5.41 gain_eff = gain * ncombine ron_eff = ron * sqrt(ncombine) # sig = sci + sky imcalc('%s,%s'%(sci_file, sky_file_1D), sig_file, 'im1+im2') # sig = sqrt(sig * gain + ron**2) / gain equation = 'sqrt(max(im1,0.0)/%(g)8.5f + %(r2)8.5f/%(g2)8.5f)' equation = equation%{'g': gain_eff, 'r2': ron_eff**2, 'g2': gain_eff**2} imcalc(sig_file, sig_file, equation)

•  Use IRAF tasks in Python scripts

•  Just import iraf object from pyraf module

from pyraf import iraf from glob import glob

images = glob(‘*sci.fits’)

for img in images: iraf.imstat(img) newimg = img.replace(‘sci’, ‘sig’) iraf.imcalc(img, newimg, ’sqrt(im1)')

•  Can specify ‘default’ task parameters in neat fashion, instead of having to include on every call of a task

iraf.imstat.nclip = 3 iraf.imstat.lsigma = 5 iraf.imstat.usigma = 5

# now every time I use imstat it uses sigma clipping iraf.imstat(im1) iraf.imstat(im2) iraf.imstat(im3)

# can revert to task defaults by unlearning iraf.unlearn(‘imstat’) # note task name is a string

•  Useful to make shortcuts

# shortcut for a long task name crrej = iraf.stsdas.hst_calib.wfpc.crrej crrej.mask = ‘mymask.fits’ crrej.sigma = 5

crrej(in1, out1) crrej(in2, out2) crrej(in3, out3)

•  IRAF outputs lots of useful data to screen - how to get at it?

•  Some tasks need user input - would be nice to automate these

•  PyRAF defines extra parameters Stdout and Stdin for all tasks •  Stdout can be a filename, file object, or 1 (one) to direct to a list •  Stdin can be a filename, file object, or string

for img in images: # get a list with a string for each line of output output = iraf.imstat(img, format=‘no’,fields=‘midpt’,Stdout=1) # output can then be parsed as necessary to get at info med = float(output[0]) newimg = img.replace(‘sci’, ‘sub’) iraf.imcalc(img, newimg, ’im1 - %f’%med)

The End

An introduction to scientific programming with