Computational Biology

Röbbe Wünschiers

Computational Biology

A Practical Introduction to BioDataProcessing and Analysis with Linux,MySQL, and R

Second Edition

123

Röbbe WünschiersBiotechnology/Computational BiologyUniversity of Applied SciencesMittweidaGermany

ISBN 978-3-642-34748-1 ISBN 978-3-642-34749-8 (eBook)DOI 10.1007/978-3-642-34749-8Springer Heidelberg New York Dordrecht London

Library of Congress Control Number: 2012954526

� Springer-Verlag Berlin Heidelberg 2004, 2013This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part ofthe material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformation storage and retrieval, electronic adaptation, computer software, or by similar or dissimilarmethodology now known or hereafter developed. Exempted from this legal reservation are briefexcerpts in connection with reviews or scholarly analysis or material supplied specifically for thepurpose of being entered and executed on a computer system, for exclusive use by the purchaser of thework. Duplication of this publication or parts thereof is permitted only under the provisions ofthe Copyright Law of the Publisher’s location, in its current version, and permission for use must alwaysbe obtained from Springer. Permissions for use may be obtained through RightsLink at the CopyrightClearance Center. Violations are liable to prosecution under the respective Copyright Law.The use of general descriptive names, registered names, trademarks, service marks, etc. in thispublication does not imply, even in the absence of a specific statement, that such names are exemptfrom the relevant protective laws and regulations and therefore free for general use.While the advice and information in this book are believed to be true and accurate at the date ofpublication, neither the authors nor the editors nor the publisher can accept any legal responsibility forany errors or omissions that may be made. The publisher makes no warranty, express or implied, withrespect to the material contained herein.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

Dedicated to …… Károly Nagy, who gave me my firstcomputer, a Casio PB-100,… the Open-Source Community for providingfantastic software, and… my offsprings, with the hope that they mayfind appropriate filters to manage life’sinformation avalanche.

Foreword by an Experimental Biologist

The Data Culture

The preface for the first edition of this book was called ‘‘A shift in culture’’, sincethe increasing transition of the work profile of a molecular biologist from thebench to the computer became already more than apparent at that time. Eight yearshave passed since then and the shift has become even more radical than anyonecould have anticipated. Genomics has continued its explosive development, withsequencing costs within genome projects having gone down by a factor of 10,000in the past 10 years. The generation of huge masses of data, even within anaverage Ph.D. project, is quickly becoming routine. Benchwork may becomereduced to the mere extraction of DNA and RNA, followed by sending thematerial to sequencing centers. Data are then often returned on hard disks, sinceeven the fastest online connections are too slow to transport them. Working withthese data is now the challenge of our days, computational skills are the top keyqualification for the molecular biologist.

Bioinformatics has kept up with this challenge and the small helpful ‘‘progies’’of the past have transformed to increasingly sophisticated software packages thatcan deal with many of the initial standard problems of data analysis, such asquality control, mapping, and basic statistics. But handling the data remains thetask of each individual scientist and being able to work with them in a Unixenvironment is crucial. This is why this book devotes several chapters to the basictask of mere data processing. The goal of each genomics experiment is of course tomake sense out of the data and this requires connecting them with known infor-mation stored in databases. Skills in working with relational databases, such asMySQL, understanding their syntax, and being able to do some simple program-ming within them, is therefore equally indispensable.

But there is also another big problem hidden in the transition from a largelyqualitative science, as molecular biology was, to a data-driven quantitative one—the understanding of statistics. An eye-catching outlier may very easily be simply aconsequence of random fluctuation in a sea of data. If your chances to win a

vii

jackpot in a lottery are 1 in 50,000,000, you would consider this an extremelyunlikely event. But if 50,000,000 players take part in the lottery every week, it is amere rule of statistics that one of them will take the jackpot almost every week.Sounds trivial, but there have indeed been a number of genomics papers that fellinto exactly this trap by reporting a seemingly very novel insight, which wasnothing else than statistical fluctuation. To guard you against this trap, statisticalanalysis and a full understanding of what its results tell you is a must. Hence, thebook now includes also a new chapter on the use of R, a powerful statisticalpackage for data analysis and graphics that allows running many controls at everystep of data analysis—and can provide also the matching graphical output. I amconvinced that this book should be the required reading for every molecularbiologist. It will of course be particularly helpful for those dealing with genomicsdata, but even if genomics is currently not on your experimental agenda, handlinglarge datasets and doing proper statistics is a basic qualification that cannot beunderestimated in our discipline today.

Plön, Germany, August 2012 Prof. Dr. Diethard TautzDirector at the Max-Planck Institute

for Evolutionary Biology

viii Foreword by an Experimental Biologist

Foreword by a Computer Scientistand Father of AWK

A Must Read for any Computational Scientist

This book is a must read for any scientist interested in computational biology. Allexperimental scientists today must know how to analyze data and manipulateinformation by themselves. Because of cost and time, they cannot rely on hiredhelp to do the analyses and manipulations for them. In a few hundred pages, thisbook teaches the reader to do useful data analysis and processing in the Internetage using the versatile and powerful tools available for all Unix/Linuxenvironments.

The reader need not know anything about programming or Unix/Linux.Professor Dr. Wünschiers begins from scratch explaining the hardware/softwarearchitecture of a computer system starting from the hardware level and then goingthrough the operating system kernel and the shell to the languages and packagesavailable at the applications level. He even provides a succinct history of thedevelopment of the history behind the various versions of Unix and Linux systems.

The author provides a collection of concise chapters explaining how to getstarted on a Unix/Linux system, how to create and work with files, how to writeprograms in data-processing scripting languages such as awk, and how to down-load useful application packages. He introduces shell programming by showing thereader how to create simple but powerful shell scripts using sed and other shellcommands. The author teaches the reader how to write real programs to do usefuldata-processing tasks. As one of the inventors of awk, I am delighted he haschosen our programming language as his primary vehicle. Since awk is a simple,easy-to-learn, and an easy-to-use data-processing language, he provides a com-prehensive description of awk through a graded sequence of awk programsdesigned to solve illustrative problems. By the end of the section on awk, thereader should be familiar with most of the language and be able to solve usefulproblems in computational biology by writing his or her own awk programs.

The book also includes sections showing how to use resources available on thenet and downloadable application packages such as the relational database system

ix

MySQL, the statistics suite R, and the similarity-sequence detection BLAST tosolve representative problems in contemporary computational biology.

There are many things I like about this book. First, the material on Unix/Linuxis presented in a no-nonsense manner that would be familiar and appealing to anyUnix/Linux programmer. It is clear the author has internalized the powerful Unix/Linux building-block approach to problem solving. Second, the book is written ina lively and engaging style. It is not a turgid user manual. Finally, throughout thebook the author admonishes the reader to write programs continuously as he or shereads the material. This cannot be overemphasized—it is well known that the onlyway to learn how to program effectively is by writing and running programs.

If you want to become a computational biologist proficient in solving realproblems in the Unix/Linux environment by yourself, then this book is a mustread.

New York, USA, September 2012 Alfred V. AhoLawrence Gussman Professor

Department of Computer ScienceColumbia University

x Foreword by a Computer Scientist and Father of AWK

Preface to the Second Edition

AWKology at Its Best

This year was full of innovative achievements in the field of computationalbiology and bioinformatics. I just like to mention two personal highlights: (a) Thepublication of a whole-cell computational model of the bacterium Mycoplasmagenitalium that allows prediction of phenotype from genotype (Karr et al 2012)and (b) the coordinated publication of major results from the internationalENCODE (Encyclopedia of DNA Elements) project as a set of 30 papers acrossthree different journals that are digitally cross-linked as so-called threads. Eachthread consists of theme-specific paragraphs, figures, and tables from across thesepapers. Both research projects handle a huge amount of complex data. But at thevery basis there are a number of tabulator-delimited text files that needed to befiltered, rearranged, reformatted, statistically analyzed, or transferred to relationaldatabases for improved data handling and number crunching.

This is precisely what this book is about. My aim is to place you in a betterposition to handle and analyze data—lots of data. It arose from my own needsand experiences to do so. Once you learn how to play with your datasets, these inturn may change your mindset (as Hans Rosling once put it). The core is dataprocessing and visualization. Thus, this book is about data piping, not pipetting.

The Book’s Title. The title of this book is Computational Biology. Some wouldargue that its content is bioinformatics. Why is that? I am a biologist and I com-plement my experiments with computational methods. To me, computationalbiology is the complement to experimental biology. During my professionalcareer in industries I was head of projects which aimed at different things like genediscovery, data integration and visualization, and statistical analyses. I wasemployed as a bioinformatics manager. But was it bioinformatics that I was doing?Or was it computational biology? Or something else?

xi

There certainly is a difference between computational biology and bioinformatics.However, it heavily depends on whom you are asking (see Sect. 1.3 on page 6). I oftenhear that bioinformatics is about the development of software tools while computa-tional biology deals with mathematical modeling. A leading journal, PLOS Compu-tational Biology, states that it publishes work that furthers our understanding of livingsystems at all scales through the application of computational methods. Computa-tional methods in turn involve information processing—and this is what this book isabout. Anyway, both terms are frequently used synonymously and the buzz wordclearly is bioinformatics—I am glad that you still found this book.)

New Chapters, Extended Concept and a Dinosaur. What has changed since thepublication of the first edition in 2004? A lot! Next-generation sequencing is notthe next generation any more—it is presence. This means that there is a lot moredata available (Strasser 2012). High throughput methods became the standard inmolecular analysis and provide even more data. More data means that there aremore and higher possibilities to find correlations between datasets. But it alsoimplies that there are more datasets that have to be processed.

This new edition grew out of my experience of working with biological data inboth academia and industries. I saw the need to add chapters on databases(MySQL) and statistical data analysis & visualization (R). From my courses oncomputational biology I learned about the importance of having a tangible prob-lem to solve. This motivates to move on in the command line and likewisedemonstrates its power. Therefore, I chose to add worked examples.

Since 2004, Linux has become much more comfortable—not only to use, butalso to install. Gaining access to a USB-stick was a pain in the neck back then:mount -t vfat /dev/hde1 /home/Freddy/USB/. Nowadays, everybodycan install a free virtual machine and run almost any operating system on anyoperating system—in parallel. I take advantage of these developments by showinghow to set up Ubuntu Linux in a VirtualBox.

My dinosaur is AWK. Though there is almost no further development, it is stilljust amazing to see what one can do. I met several experimentalists who are intocomputational biology that apply AWK. Why? One of its three developers, AlfredAho, recently said: If I had to choose a word to describe our centering forces inlanguage design, I’d say Kernighan emphasized ease of learning; Weinberger,soundness of implementation; and I, utility. I think AWK has all three of theseproperties (Biancuzzi and Warden 2009). That describes it well. Students withoutany programming experience usually pickup data processing in the command linewith AWK within some days. I love it.

xii Preface to the Second Edition

Acknowledgments. I wish to thank all colleagues and students who have read,commented upon, and corrected various chapters of this book. This second editionbenefited a lot from the waking eyes of the students attending my computationalbiology course, especially Sebastian Gustmann and Robin Schwarzer at CologneUniversity, Germany.

Quedlinburg, Germany, September 2012 Röbbe Wünschiers

References

Karr et al (2012) A whole-cell computational model predicts phenotype fromgenotype. Cell 150:389. simtk.org.Strasser BJ (2012) Data-driven sciences: from wonder cabinets to electronicdatabases. Stud Hist Philos Biolo Biomed Sci 43:85.Biancuzzi F, Warden S (eds) (2009) Masterminds of programming. O’Reilly,Sebastopol, p 104.

Preface to the Second Edition xiii

Preface to the First Edition

Welcome on Board!

With this book I would like to invite you, the scientist, to a journey throughterminals and program codes. You are welcome to put aside your pipette, cultureflask, or rubber boots for a while, make yourself comfortable in front of a computer(do not forget your favourite hot alcohol-free drink), and learn some unixing andprogramming. Why? Because we are living in the information age and there is ahuge amount of biological knowledge and databases out there. They containinformation about almost everything: genes and genomes, rRNAs, enzymes,protein structures, DNA-microarray experiments, single organisms, ecologicaldata, the tree of life, and endless more. Furthermore, nowadays many researchapparatuses are connected to computers. Thus, you have electronic access to yourdata. However, in order to cope with all this information you need some tools. Thisbook will provide you with the skills to use these tools and to develop your owntools, i.e., it will introduce Unix and its derivatives (Linux, Mac OS X, CygWin,etc.) and programming (shell programming, awk, perl). These tools will make youindependent of the way in which other people make you process your data—in theform of application software. What you want is open functionality. You want todecide how to process (e.g., analyze, format, save, correlate) data and you want itnow—not waiting for the lab programmer to treat your request; and you know itbest—you understand your data and your demands. This is what open functionalitystands for, and both Linux and programming languages can provide it to you.

I started programming on a Casio PB-100 hand-held built in 1983. It can store10 small Basic programs. The accompanying book was entitled ‘‘Learn as you go’’and, indeed, in my opinion this is the best way to learn programming. My firstcontact to Unix was triggered by the need to copy data files from a Unix-drivenBruker EPR-Spectrometer onto a floppy disk. The real challenge started when I

xv

tried to import the files to a data-plotting program on the PC. While the firstproblem could be solved by finding the right page in a Unix manual, the latterrequired programming skills—Q-Basic at that time. This problem was minorcompared to the trouble one encounters today. A common problem is to feed oneprogram with the output of another program: you might have to change lines tocolumns, commas to dots, tabulators to semicolons, uppercase to lowercase, DNAto RNA, FASTA to GenBank format, and so forth. Then there is that huge amountof information out there on the Web, which you might need to bring into shape foryour own analysis.

You and This Book. This book is written for the total beginner. You need noteven know what a computer is, though you should have access to one and find thepower switch. The book is the result of (a) the way I learned to work with Unix, itsderivatives, and its numerous tools and (b) a lecture which I started at the Institutefor Genetics at the University of Cologne/Germany. Most programming examplesare taken from biology; however, you need not be a biologist. Except for two orthree examples, no biological knowledge is necessary. I have tried to illustratealmost everything practically with so-called terminals and examples. You shouldrun these examples. Each chapter closes with some exercises. Brief solutions canbe found at the end of the book.

Why Linux? This book is not limited to Linux! All examples are valid for Unix orany Unix derivative like Mac OS X, Knoppix or the free Windows-based CygWinpackage, too. I chose Linux because it is open source software: you need not investmoney except for the book itself. Furthermore, Linux provides all the great toolsUnix provides. With Linux (as with all other Unix derivatives) you are close to yourdata. Via the command line you have immediate access to your files and can useeither publicly available or your own designed tools to process these. With the aidof pipes you can construct your own data-processing pipeline. It is great.

Why awk and perl? awk is a great language for both learning programming andtreating large text-based data files (contrary to binary files). For 99 % you willwork with text-based files, be it data tables, genomes, or species lists. Apart frombeing simple to learn and having a clear syntax, awk provides you with thepossibility to construct your own commands. Thus, the language can grow withyou as you grow with the language. I know bioinformatic professionals entirelyfocusing on awk.perl is much more powerful but also more unclear in its syntax(or flexible, to put it positively), but, since awk was one basis for developingperl, it is only a small step to go once you have learned awk – but a giant leap foryour possibilities. You should take this step. By the way, both awk and perl runon all common operating systems.

xvi Preface to the First Edition

Acknowledgments. Special thanks to Kristina Auerswald, Till Bayer, BenediktBosbach, and Chris Voolstra for proofreading, and all the other students forencouraging me to bring these lines together.

Hürth, Germany, January 2004 Röbbe Wünschiers

Preface to the First Edition xvii

Contents

Part I Whetting Your Appetite

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1 A Very Short History of Bioinformatics

and Computational Biology . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Contemporary Biology . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Computational Biology or Bioinformatics? . . . . . . . . . . . . . 61.4 Computers in Biology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.5 The Future: Digital Lab Benches

and Designing Organisms . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Content of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1 The Main Chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.2 Shell Programming . . . . . . . . . . . . . . . . . . . . . . . 122.1.3 Sed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1.4 AWK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1.5 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1.6 MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.1.7 R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.1.8 Worked Examples . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2 Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3 Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.4 Additional Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Part II Computer and Operating Systems

3 Unix/Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.1 What is a Computer? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 Some History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

xix

3.2.1 Versions of Unix. . . . . . . . . . . . . . . . . . . . . . . . . 233.2.2 The Rise of Linux. . . . . . . . . . . . . . . . . . . . . . . . 243.2.3 Why a Penguin? . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.4 Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . 26

3.3 X-Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3.1 How Does it Work? . . . . . . . . . . . . . . . . . . . . . . 27

3.4 Linux Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.5 What is the Difference Between Unix and Linux? . . . . . . . . 293.6 What is the Difference Between Linux and Windows? . . . . . 293.7 What is the Difference Between Linux and Mac OS X? . . . . 293.8 One Computer, Two Operating Systems . . . . . . . . . . . . . . . 30

3.8.1 VMware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.8.2 CygWin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.8.3 Wine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.8.4 Others . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.9 Knoppix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.9.1 Vigyaan Knoppix for Biosciences . . . . . . . . . . . . . 32

3.10 Software Running Under Linux . . . . . . . . . . . . . . . . . . . . . 333.10.1 Bioscience Software for Linux . . . . . . . . . . . . . . . 333.10.2 Office Software and Co. . . . . . . . . . . . . . . . . . . . 343.10.3 Graphical Desktops . . . . . . . . . . . . . . . . . . . . . . . 353.10.4 Others . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Part III Working with Linux

4 The First Touch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.1 Just Before the First Touch . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1.1 Running Linux from USB or CDROM . . . . . . . . . 394.1.2 Running Linux as a Virtual Machine. . . . . . . . . . . 40

4.2 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.2.1 Working with CygWin or Mac OS X . . . . . . . . . . 444.2.2 Working Directly on a Linux Computer… . . . . . . . 444.2.3 Working on a Remote Linux Computer . . . . . . . . . 44

4.3 Using Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.3.1 History and Autocompletion. . . . . . . . . . . . . . . . . 474.3.2 Syntax of Commands . . . . . . . . . . . . . . . . . . . . . 474.3.3 Editing the Command Line . . . . . . . . . . . . . . . . . 484.3.4 Change Password . . . . . . . . . . . . . . . . . . . . . . . . 494.3.5 Help: I’m Stuck . . . . . . . . . . . . . . . . . . . . . . . . . 494.3.6 How to Find out More. . . . . . . . . . . . . . . . . . . . . 49

4.4 Logout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

xx Contents

5 Working with Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.1 Browsing Files and Directories. . . . . . . . . . . . . . . . . . . . . . 535.2 Moving, Copying and Renaming Files and Directories . . . . . 565.3 File Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.4 Special Files: . and .. . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.5 Protecting Files and Directories . . . . . . . . . . . . . . . . . . . . . 59

5.5.1 Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.5.2 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.5.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.5.4 Changing File Attributes . . . . . . . . . . . . . . . . . . . 605.5.5 Extended File Attributes . . . . . . . . . . . . . . . . . . . 62

5.6 File Archives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.7 File Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.8 Searching for Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.8.1 Selecting Files for Backups . . . . . . . . . . . . . . . . . 685.9 Display Disk Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

6 Remote Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716.1 Downloading Data from the Web . . . . . . . . . . . . . . . . . . . . 71

6.1.1 wget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716.1.2 curl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.2 Secure Copy: scp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726.3 Secure Shell: ssh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.4 Backups with rsync . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.5 ssh, scp & rsync without Password. . . . . . . . . . . . . . . . 74

7 Playing with Text and Data Files . . . . . . . . . . . . . . . . . . . . . . . . 777.1 Viewing and Analyzing Files . . . . . . . . . . . . . . . . . . . . . . . 78

7.1.1 A Quick Start: cat . . . . . . . . . . . . . . . . . . . . . . . 787.1.2 Text Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797.1.3 Extract Unique Lines. . . . . . . . . . . . . . . . . . . . . . 817.1.4 Viewing File Beginning or End . . . . . . . . . . . . . . 817.1.5 Scrolling Through Files . . . . . . . . . . . . . . . . . . . . 817.1.6 Character, Word, and Line Counting . . . . . . . . . . . 827.1.7 Splitting Files into Pieces. . . . . . . . . . . . . . . . . . . 827.1.8 Cut and Paste Columns . . . . . . . . . . . . . . . . . . . . 837.1.9 Finding Text: grep . . . . . . . . . . . . . . . . . . . . . . 847.1.10 Text File Comparisons. . . . . . . . . . . . . . . . . . . . . 85

7.2 Editing Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 867.2.1 The Sparse Editor Pico . . . . . . . . . . . . . . . . . . . . 877.2.2 The Rich Editor Vim. . . . . . . . . . . . . . . . . . . . . . 887.2.3 Installing Vim. . . . . . . . . . . . . . . . . . . . . . . . . . . 897.2.4 Immediate Takeoff . . . . . . . . . . . . . . . . . . . . . . . 897.2.5 Starting Vim. . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Contents xxi

7.2.6 Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 907.2.7 Moving the Cursor . . . . . . . . . . . . . . . . . . . . . . . 917.2.8 Doing Corrections . . . . . . . . . . . . . . . . . . . . . . . . 927.2.9 Save and Quit. . . . . . . . . . . . . . . . . . . . . . . . . . . 937.2.10 Copy and Paste. . . . . . . . . . . . . . . . . . . . . . . . . . 937.2.11 Search and Replace . . . . . . . . . . . . . . . . . . . . . . . 94

7.3 Text File Conversion (Unix $ DOS) . . . . . . . . . . . . . . . . . 957.3.1 Batch Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

8 Using the Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978.1 What is the Shell? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978.2 Different Shells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 988.3 Setting the Default Shell . . . . . . . . . . . . . . . . . . . . . . . . . . 998.4 Useful Shortcuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.5 Redirections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018.6 Pipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1038.7 Lists of Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1048.8 Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1048.9 Pimping the Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1058.10 Batch Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068.11 Scheduling Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078.12 Wildcards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088.13 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

8.13.1 Checking Processes . . . . . . . . . . . . . . . . . . . . . . . 1108.13.2 Realtime Overview . . . . . . . . . . . . . . . . . . . . . . . 1118.13.3 Background Processes . . . . . . . . . . . . . . . . . . . . . 1118.13.4 Killing Processes. . . . . . . . . . . . . . . . . . . . . . . . . 113

9 Installing BLAST and ClustalW . . . . . . . . . . . . . . . . . . . . . . . . . 1159.1 Downloading the Programs via FTP . . . . . . . . . . . . . . . . . . 115

9.1.1 Downloading BLAST . . . . . . . . . . . . . . . . . . . . . 1169.1.2 Downloading ClustalW . . . . . . . . . . . . . . . . . . . . 117

9.2 Installing BLAST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1189.3 Running BLAST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1199.4 Installing ClustalW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1209.5 Running ClustalW. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1219.6 A Wrapper for ClustalW . . . . . . . . . . . . . . . . . . . . . . . . . . 122

10 Shell Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12510.1 Script Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12510.2 Modifying the Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12710.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

xxii Contents

10.4 Input and Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13010.4.1 echo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13010.4.2 Here Documents: \\. . . . . . . . . . . . . . . . . . . . . . 13110.4.3 read and line . . . . . . . . . . . . . . . . . . . . . . . . . 13210.4.4 Script Parameters . . . . . . . . . . . . . . . . . . . . . . . . 133

10.5 Substitutions and Expansions . . . . . . . . . . . . . . . . . . . . . . . 13410.5.1 Variable Substitution . . . . . . . . . . . . . . . . . . . . . . 13410.5.2 Command Expansion. . . . . . . . . . . . . . . . . . . . . . 136

10.6 Quoting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13610.6.1 Escape Character . . . . . . . . . . . . . . . . . . . . . . . . 13710.6.2 Single Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . 13710.6.3 Double Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . 137

10.7 Decisions: Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . 13810.7.1 if...then...elif...else...fi. . . . . . . . 13810.7.2 test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13910.7.3 while...do...done . . . . . . . . . . . . . . . . . . . 14110.7.4 until...do...done . . . . . . . . . . . . . . . . . . . 14310.7.5 for...in...do...done . . . . . . . . . . . . . . . . 14410.7.6 case...in...esac . . . . . . . . . . . . . . . . . . . . 14510.7.7 select...in...do . . . . . . . . . . . . . . . . . . . . 146

10.8 Desktop Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14710.8.1 MacOSX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14710.8.2 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14810.8.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

10.9 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14810.9.1 bash -xv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14810.9.2 trap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

10.10 Remote Control Interactive Programs . . . . . . . . . . . . . . . . . 15010.11 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

10.11.1 Check for DNA as File Content . . . . . . . . . . . . . . 15210.11.2 Time Signal . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15310.11.3 Select Files to Archive . . . . . . . . . . . . . . . . . . . . 15310.11.4 Remove Spaces. . . . . . . . . . . . . . . . . . . . . . . . . . 155

11 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.1 Regular Expression and Neurons . . . . . . . . . . . . . . . . . . . . 15711.2 Get Started … . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15811.3 Using Regular Expressions. . . . . . . . . . . . . . . . . . . . . . . . . 16011.4 Search Pattern and Examples . . . . . . . . . . . . . . . . . . . . . . . 161

11.4.1 Single-Character Meta Characters . . . . . . . . . . . . . 16211.4.2 Quantifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16411.4.3 Grouping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16511.4.4 Anchors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16611.4.5 Escape Sequences . . . . . . . . . . . . . . . . . . . . . . . . 167

Contents xxiii

11.4.6 Alternation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16711.4.7 Back References . . . . . . . . . . . . . . . . . . . . . . . . . 16811.4.8 Character Classes . . . . . . . . . . . . . . . . . . . . . . . . 16911.4.9 Priorities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16911.4.10 egrep Options . . . . . . . . . . . . . . . . . . . . . . . . . 170

11.5 RE Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17011.6 Regular Expression Training . . . . . . . . . . . . . . . . . . . . . . . 172

11.6.1 Regular Expressions and vim. . . . . . . . . . . . . . . . 17211.7 Regular Expression in Genome Research. . . . . . . . . . . . . . . 172

12 Sed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17512.1 When to Use Sed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17612.2 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17612.3 How Sed Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

12.3.1 Pattern Space . . . . . . . . . . . . . . . . . . . . . . . . . . . 17812.3.2 Hold Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

12.4 Sed Syntax. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17912.4.1 Addresses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18012.4.2 Sed and Regular Expressions . . . . . . . . . . . . . . . . 181

12.5 Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18112.5.1 Substitutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18212.5.2 Transliterations . . . . . . . . . . . . . . . . . . . . . . . . . . 18412.5.3 Deletions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18512.5.4 Insertions and Changes . . . . . . . . . . . . . . . . . . . . 18612.5.5 Sed Script Files . . . . . . . . . . . . . . . . . . . . . . . . . 18812.5.6 Printing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18912.5.7 Reading and Writing Files . . . . . . . . . . . . . . . . . . 18912.5.8 Advanced Sed. . . . . . . . . . . . . . . . . . . . . . . . . . . 190

12.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19012.6.1 Gene Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19112.6.2 File Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19112.6.3 Reversing Line Order . . . . . . . . . . . . . . . . . . . . . 192

Part IV Programming

13 AWK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19713.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19813.2 AWK’s Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19913.3 Example File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20013.4 Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

13.4.1 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . 20113.4.2 Pattern-Matching Expressions . . . . . . . . . . . . . . . . 20213.4.3 Relational Character Expressions . . . . . . . . . . . . . 203

xxiv Contents

13.4.4 Relational Number Expressions . . . . . . . . . . . . . . 20513.4.5 Mixing and Conversion of Numbers

and Characters . . . . . . . . . . . . . . . . . . . . . . . . . . 20613.4.6 Ranges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20613.4.7 BEGIN and END . . . . . . . . . . . . . . . . . . . . . . . . . 207

13.5 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20813.5.1 Assignment Operators . . . . . . . . . . . . . . . . . . . . . 20813.5.2 Increment and Decrement . . . . . . . . . . . . . . . . . . 20913.5.3 Predefined Variables . . . . . . . . . . . . . . . . . . . . . . 21013.5.4 Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21513.5.5 Shell Versus AWK Variables . . . . . . . . . . . . . . . . 219

13.6 Scripts and Executables . . . . . . . . . . . . . . . . . . . . . . . . . . . 21913.7 Decisions: Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . 220

13.7.1 if...else... . . . . . . . . . . . . . . . . . . . . . . . . 22013.7.2 while... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22113.7.3 do...while... . . . . . . . . . . . . . . . . . . . . . . . 22213.7.4 for... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22313.7.5 Leaving Loops . . . . . . . . . . . . . . . . . . . . . . . . . . 224

13.8 Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22513.8.1 Printing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22513.8.2 Numerical Calculations . . . . . . . . . . . . . . . . . . . . 22813.8.3 String Manipulation. . . . . . . . . . . . . . . . . . . . . . . 22913.8.4 System Commands . . . . . . . . . . . . . . . . . . . . . . . 23413.8.5 User-Defined Functions . . . . . . . . . . . . . . . . . . . . 234

13.9 Input with getline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23813.9.1 Reading from a File . . . . . . . . . . . . . . . . . . . . . . 23913.9.2 Reading from the Shell . . . . . . . . . . . . . . . . . . . . 241

13.10 Produce Nice Output with . . . . . . . . . . . . . . . . . . . . . 24213.11 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

13.11.1 Sort an Array by Indices . . . . . . . . . . . . . . . . . . . 24313.11.2 Sum up Atom Positions . . . . . . . . . . . . . . . . . . . . 24413.11.3 Convert FASTA $ Table Formats . . . . . . . . . . . . 24413.11.4 Mutate DNA Sequences. . . . . . . . . . . . . . . . . . . . 24513.11.5 Translate DNA to Protein . . . . . . . . . . . . . . . . . . 24613.11.6 Calculate Atomic Composition of Proteins . . . . . . . 24713.11.7 Dynamic Programming: Levenshtein Distance

of Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

14 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25514.1 Intention of This Chapter. . . . . . . . . . . . . . . . . . . . . . . . . . 25514.2 Running Perl Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25614.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

14.3.1 Scalars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25714.3.2 Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Contents xxv

14.3.3 Hashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26214.3.4 Built-in Variables . . . . . . . . . . . . . . . . . . . . . . . . 266

14.4 Decisions: Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . 26614.4.1 if...elsif...else . . . . . . . . . . . . . . . . . . . 26714.4.2 unless, die and warn. . . . . . . . . . . . . . . . . . . 26714.4.3 while... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26814.4.4 do...while... . . . . . . . . . . . . . . . . . . . . . . . 26814.4.5 until... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26814.4.6 do...until... . . . . . . . . . . . . . . . . . . . . . . . 26814.4.7 for... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26914.4.8 foreach... . . . . . . . . . . . . . . . . . . . . . . . . . . 26914.4.9 Controlling Loops . . . . . . . . . . . . . . . . . . . . . . . . 269

14.5 Data Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27114.5.1 Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . 27114.5.2 Internal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27214.5.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

14.6 Data Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27414.6.1 print, printf and sprintf . . . . . . . . . . . . . 27414.6.2 Here Documents:\\ . . . . . . . . . . . . . . . . . . . . . . 27414.6.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

14.7 Hash Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27614.8 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

14.8.1 Special Escape Sequences . . . . . . . . . . . . . . . . . . 27714.8.2 Matching: m/.../ . . . . . . . . . . . . . . . . . . . . . . . 278

14.9 String Manipulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27914.9.1 Substitute: s/.../.../ . . . . . . . . . . . . . . . . . . 27914.9.2 Transliterate: tr/.../.../ . . . . . . . . . . . . . . . 27914.9.3 Common Commands . . . . . . . . . . . . . . . . . . . . . . 280

14.10 Calculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28114.11 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28214.12 Packages and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . 28314.13 Bioperl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28514.14 You Want More? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28614.15 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286

14.15.1 Reverse Complement DNA . . . . . . . . . . . . . . . . . 28614.15.2 Calculate GC Content . . . . . . . . . . . . . . . . . . . . . 28714.15.3 Restriction Enzyme Digestion. . . . . . . . . . . . . . . . 288

15 Other Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . . 291

xxvi Contents

Part V Advanced Data Analysis

16 Relational Databases with MySQL . . . . . . . . . . . . . . . . . . . . . . . 29516.1 What is MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295

16.1.1 Relational Databases . . . . . . . . . . . . . . . . . . . . . . 29616.2 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

16.2.1 Get in Touch with the MySQL Server . . . . . . . . . . 29716.2.2 Starting and Stopping the Server. . . . . . . . . . . . . . 29816.2.3 Setting a Root Password . . . . . . . . . . . . . . . . . . . 29916.2.4 Set Up an User Account and Database . . . . . . . . . 301

16.3 Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30316.3.1 Remote Access . . . . . . . . . . . . . . . . . . . . . . . . . . 30316.3.2 MySQL Syntax. . . . . . . . . . . . . . . . . . . . . . . . . . 30316.3.3 Creating Tables. . . . . . . . . . . . . . . . . . . . . . . . . . 30416.3.4 Filling and Editing Tables . . . . . . . . . . . . . . . . . . 30616.3.5 Querying Tables . . . . . . . . . . . . . . . . . . . . . . . . . 30916.3.6 Joining Tables . . . . . . . . . . . . . . . . . . . . . . . . . . 31216.3.7 Access from the Shell . . . . . . . . . . . . . . . . . . . . . 313

16.4 Backups and Transfers . . . . . . . . . . . . . . . . . . . . . . . . . . . 31416.5 How to Move on?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

17 The Statistics Suite R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31717.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

17.1.1 Getting Help. . . . . . . . . . . . . . . . . . . . . . . . . . . . 32017.2 Reading and Writing Data Files . . . . . . . . . . . . . . . . . . . . . 32017.3 Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

17.3.1 Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32217.3.2 Arrays and Matrices . . . . . . . . . . . . . . . . . . . . . . 32217.3.3 Lists and Data Frames . . . . . . . . . . . . . . . . . . . . . 324

17.4 Programming Structures. . . . . . . . . . . . . . . . . . . . . . . . . . . 32517.5 Data Exploration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

17.5.1 Saving Graphic . . . . . . . . . . . . . . . . . . . . . . . . . . 33017.5.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33217.5.3 t-Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33617.5.4 Chi-Square Test . . . . . . . . . . . . . . . . . . . . . . . . . 337

17.6 R Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33817.7 Extensions: Installing and Using Packages . . . . . . . . . . . . . . 339

17.7.1 Connect to MySQL . . . . . . . . . . . . . . . . . . . . . . . 33917.7.2 Life-Science Packages . . . . . . . . . . . . . . . . . . . . . 341

Contents xxvii

Part VI Worked Examples

18 Genomic Analysis of the Pathogenicity Factorsfrom E. coli Strain O157:H7 and EHEC StrainO104:H4 . . . . . . . 34518.1 The Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34518.2 Bioinformatic Tools and Resources. . . . . . . . . . . . . . . . . . . 346

18.2.1 BLAST? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34718.2.2 Genome Databases . . . . . . . . . . . . . . . . . . . . . . . 347

18.3 Detailed Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34818.3.1 Downloading and Installing BLAST? . . . . . . . . . . 34918.3.2 Downloading the Proteomes . . . . . . . . . . . . . . . . . 35118.3.3 Comparing the Genomes . . . . . . . . . . . . . . . . . . . 35318.3.4 Processing the BLAST? Result File . . . . . . . . . . . 35718.3.5 Playing with the E-Value . . . . . . . . . . . . . . . . . . . 36018.3.6 Organize Results with MySQL . . . . . . . . . . . . . . . 36118.3.7 Extract Unique ORFs . . . . . . . . . . . . . . . . . . . . . 36618.3.8 Visualize and Analyze Results with R . . . . . . . . . . 367

19 Limits of BLAST and Homology Modeling . . . . . . . . . . . . . . . . . 37519.1 The Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37519.2 Bioinformatic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376

19.2.1 genomicBLAST . . . . . . . . . . . . . . . . . . . . . . . . . 37619.2.2 ClustalW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37619.2.3 Jpred . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37719.2.4 SWISS-MODEL . . . . . . . . . . . . . . . . . . . . . . . . . 37819.2.5 Jmol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378

19.3 Detailed Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37819.3.1 Download E. coli HybD Sequences . . . . . . . . . . . . 37819.3.2 Cyanobacterial BLAST I . . . . . . . . . . . . . . . . . . . 37919.3.3 Cyanobacterial BLAST II . . . . . . . . . . . . . . . . . . 38019.3.4 Compute Alignment and Tree Diagram . . . . . . . . . 38219.3.5 Secondary Structure Examination . . . . . . . . . . . . . 38419.3.6 Tertiary Structure Alignment

(Homology Modeling) . . . . . . . . . . . . . . . . . . . . . 38619.3.7 Download and Visualize E. coli HybD

Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38719.3.8 View and Compare Cyanobacterial Structures . . . . 389

20 Virtual Sequencing of pUC18c . . . . . . . . . . . . . . . . . . . . . . . . . . 39320.1 The Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39320.2 Bioinformatic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393

20.2.1 Sequencer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39320.2.2 Assembler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39320.2.3 Dotter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39420.2.4 GenBank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394

xxviii Contents

20.3 Detailed Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39420.3.1 Download the pUC18c Sequence . . . . . . . . . . . . . 39520.3.2 Download and Setup the Virtual Sequencer . . . . . . 39520.3.3 Download and Install the TIGR Assembler . . . . . . 39620.3.4 Download and Setup Dotter . . . . . . . . . . . . . . . . . 39720.3.5 Virtual Sequencing . . . . . . . . . . . . . . . . . . . . . . . 39920.3.6 Sequence Assembly. . . . . . . . . . . . . . . . . . . . . . . 40220.3.7 Testing Different Parameters . . . . . . . . . . . . . . . . 40320.3.8 Visualizing Results with R . . . . . . . . . . . . . . . . . . 406

21 Querying for Potential Redox-Regulated Enzymes . . . . . . . . . . . . 40921.1 The Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40921.2 Bioinformatic Tools and Resources. . . . . . . . . . . . . . . . . . . 409

21.2.1 RCSB PDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40921.2.2 Jmol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41021.2.3 Surface Racer . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

21.3 Detailed Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41121.3.1 Download Crystal Structure Files . . . . . . . . . . . . . 41121.3.2 Analyze Structures Visually . . . . . . . . . . . . . . . . . 41221.3.3 Analyze Structures Computationally . . . . . . . . . . . 41321.3.4 Expand the Analysis to Many Proteins . . . . . . . . . 41721.3.5 Automatically Test for Solvent Accessibility . . . . . 419

Appendix A: Supplementary Information . . . . . . . . . . . . . . . . . . . . . . 425

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433

Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443

Contents xxix