COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus...

44
COLOSSUS AND THE BREAKING OF THE WARTIME "FISH" CODES Donald Michie Professor Emeritus of Machine Intelligence, University of Edinburgh, UK, and Adjunct Professor of Computer Science and Engineering University of New South Wales, Australia [Not for distribution: To appear in Cryptologia] Introduction On 29 th September 2000, the 505-page "General Report on Tunny" was released to the Public Records Office by the British Government Communications Headquarters (GCHQ), Cheltenham. The document does not give the identities of its authors. They were I.J. Good, D. Michie and G.A. Timms. The Report details how the "Colossus" high-speed electronic computers were used to break the German wartime enciphering by Lorenz machines ofthe traffic between Hitler's Berlin HQ and the various Army Groups in occupied Europe and North Africa, including collateral links such as Vienna- Athens. The first manual break in December 1941 of a 4000-characters message intercepted on one of these links was achieved by Brigadier John Tiltman at the secret code-breaking centre at Bletchley. The entire structure and intrinsic logic ofthe Lorenz machines was then reconstructed by William Tutte from the sample of 4000 characters ofpure key obtained by Tiltman. By the late spring of 1942 Major (later Colonel) Tester formed a section able to exploit Tutte 's feat so as in favourable cases to recover by hand both the wheel-patterns that defined the current intrinsic logic for a given month on a given link, and also the start positions, or "settings", ofthe 12 cipher wheels used to encipher each message. I joined the "Testery" soon after its creation.

Transcript of COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus...

Page 1: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

COLOSSUS AND THE BREAKING

OF THE WARTIME "FISH" CODES

Donald Michie

Professor Emeritus ofMachine Intelligence,University of Edinburgh, UK, and

Adjunct Professor ofComputer Science and EngineeringUniversity ofNew South Wales, Australia

[Not for distribution: To appear in Cryptologia]

Introduction

On 29th September 2000, the 505-page "GeneralReport on Tunny" was releasedto the Public Records Office by the British Government CommunicationsHeadquarters (GCHQ), Cheltenham. The document does not give the identitiesof its authors. They were I.J. Good, D. Michie and G.A. Timms.

The Report details how the "Colossus" high-speed electronic computers wereused to break the German wartime enciphering by Lorenz machines ofthetraffic between Hitler's Berlin HQ and the various Army Groups in occupiedEurope and North Africa, including collateral links such as Vienna-Athens. Thefirst manual break in December 1941 of a 4000-characters message interceptedon one of these links was achieved by Brigadier John Tiltman at the secretcode-breaking centre at Bletchley. The entire structure and intrinsic logic oftheLorenz machines was thenreconstructed by William Tutte from the sample of4000 characters ofpure key obtained by Tiltman. By the late spring of 1942Major (later Colonel) Tester formed a section able to exploit Tutte's feat so as infavourable cases to recover by hand both the wheel-patterns that definedthecurrent intrinsic logic for a given month on a given link, and also the startpositions, or "settings", ofthe 12 cipher wheels used to encipher each message.I joined the "Testery" soon after its creation.

Page 2: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

In December 1942 the mathematician M.H.A. Newman was given the job ofdeveloping and testing his bold idea for mechanizing the discovery ofcipher-wheel settings ofLorenz-generated messages. He at once commissioned anelectronic machine, the "HeathRobinson", ofwhich the main component wasput together by thePost Office at Dollis Hill, and the output printer togetherwith interface by TRE at Malvern.

Early in 1943 he formed his "Newmanry" section with one cryptographer(myself), two engineers and 16 Wrens. The prototype prefabricated parts ofthethepilot Heath Robinson were delivered soon afterwards, on-site assembly andtesting began, and by April 1943 the machine was operational. Wheel-settingtrials were started, but ancillary use ofthe machine was needed for amassingstatistics on German military plain text before operational use could start inearnest. Meanwhile Newman had already followed up with conceptualspecifications for the Colossus. He selected the Dollis HillPost Office researchengineer T.H. Flowers, originally recommended to him by Alan Turing, toconvert his requirements into detailed implementations. As soon as initial trialsusing the somewhat temperamental pilot Heath Robinson had indicatedfeasibilityofmethod, he was able to commission construction ofthe first oftheColossus machines.

The Colossus designpossessed radically new properties. Once the data-tapecontaining the intercepted cipher text of a message had been scanned in via thehigh-speed optical tape reader, the whole business of storing the data and ofapplying to it trial-and-error combinations of settings of simulated Lorenz-machine wheel-patterns was performed internally.In addition, hand-switchesallowed the user to apply specified boolean constraints to the machine's searchofthe stored data for statistical regularities. Constraints could be varied inthelight of intermediate results read from the on-line typewriter. Colossus 1 wasassembled over Christmas 1943 and was operational in January 1944,when itsucceeded against its first test against a real encipheredmessage tape. Operationon a serious scale began in February.

Starting with monthly changes, by the closing stages ofthe war the Germanswere changing the intrinsic logic, that is the wheelpatterns, ofthe Lorenzcipher every day on every link. The German machines at each end of a linkacted both in encipher-and-send mode and in receive-and-decipher modes.Provided that the operators at each end ensured that the same patterns and thesame settings ofthe 12 wheels were used for a given transmission, the plain-

Page 3: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

text message input on punched paper tape at one end was output as plain-text atthe other end, having made the journeyover the airwaves as cipher-text.

To read the intercepted messages sent on a given link on a given day it wasnecessary both to break the day's new logic (represented as patterns set up onthe Lorenz machine's 12 wheels) and then, for each ofthe day's messagesintercepted on that link, to discover the starting positions to which each ofthe12 wheels had been set for thatparticular message-transmission. We named theintercepted traffic on the different links, Berlin-Paris, Berlin-Rome, Berlin-Tunis, etc. after different species offish. The first ofthese to be broken wascalled Tunny, and it became a common practice to use "Tunny" as a genericlabel. This explains the title borne by the Good-Michie-Timmsreport.

By the end ofthe war the closely interlaced operations ofColonel Tester'shand-cryptanalysis section (the "Testery") and ofMax Newman's machinesection (the "Newmanry"), were breaking about 25 new sets ofwheel patternsweekly and the complete settings ofabout 150 transmissions. On grounds oftherelative cost-effectiveness ofanticipated further effort, a rather larger number ofintercepts were abandoned with some settings still not found. Under the co-ordinated control of 22 cryptographers and 273 Wrens on a 3-shift regime, nineColossus machines, each weighingover a ton, were working continuouslyround the clock, with one still under completion. Peak monthly decryptionwasin excess of 600 messages per month.

To convey a broad appreciation ofthe combinatorial magnitudes ofthe twotasks, of finding the patterns and finding the settings: the number ofpossiblesets of 12 wheelpatterns was ofthe order of 10120. The number ofpossiblesettings ofthe 12 wheels was (41 x3 l x29x 26 x 23) x (43 x47x 5 1 x53 x59) x (61 x 37), i.e. approximately 1020. The use ofbrackets above designatesthe three kinds ofwheels, namely five chi-wheels corresponding to the fiveteleprinter channels of5-hole punched paper tape, five psi wheels, and two mvwheels, or "motor" wheels. The latter were of a different character from theothers and exercised a control function. Their task was to govern step by stepwhether the psi wheels were to move on en bloc, in concert with the chi wheels,or whether they should stand still for the current step.

It is worth noting at this point that this further elaboration oftheLorenzmachine design had the object offurther complicating the task ofwould-becodebreakers. In actuality it had the opposite effect, introducing a subtleelement ofregularity. Ifthe motor wheels had been omitted from the German

Page 4: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

designs, it is overwhelminglyprobable that the Fish codes would never havebeen broken. Pure key, ofwhich Brigadier Tiltman's initial 1941 break provideda sample of some 4000 successive characters, could readily have been soconstructed as to destroy any practical possibility ofreverse engineeringtorecover the logic that generated it. Admittedly, with just the 10 chi-wheels plusthe 10 psi-wheels, the above combinatorics getknocked down to about 1090 and17.10 . But given reasonably prudent restrictions on the properties ofthe wheel

patterns themselves, thesereductions still guarantee a sufficient semblance ofrandomness in the generated chi + psi stream ofcharacters. But as it was, theextra and gratuitous touch ofhaving all psi wheels intermittently stutter insynchrony allowed entry of a serious and systematic departure fromrandomness, as will later be illustrated.

The 1500-valve Colossus 1 became operational in February 1944 with the jobof substantively mechanizing the finding ofwheel settings. Note that this couldonlybe done for the tiny subset of all traffic for which the wheelpatterns forthe given link and day had already been broken by laborious, slow and chancyhand processes.

In April 1944, as briefly described by I.J.Good (1994), I made a technicalproposal which was simple enough to allow the two ofus to validate itscorrectness in a couple ofhours' experimentation. Using the machine in waysfor which it had not been designed, we demonstrated the feasibility ofgoingbeyond the automated finding ofwheel settings, to machine-aided breaking ofthe wheel-patterns themselves ~ the "intrinsic logic" referred to earlier.Technical aspects ofthis part ofthe story will form a main topic ofmychapterin B. J. Copeland's forthcoming book.

A "crash programme" was immediately authorized at War Cabinet level,targeted on having a working Colossus ofthe new design in time for D-day on6th June. This was achieved, with a few days to spare, by a team ofPost Officeresearch engineers ledby T.H. Flowers' lieutenant A.W.M. Coombs. The taskwas achieved by equippingthe 2500-valve Colossus 2, already underconstruction since January, with a "special attachment". The design,engineering and patching-in ofthis attachment was led by a team member,Harry Fensom.

The remaining eight new-design Colossi, each further enhanced over itspredecessor, joinedthe others in rapid sucession over the ensuing nine months.

Page 5: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Colossus 10 was under completion at the end ofhostilities, by which time morethan 63 million characters ofhigh grade German messages had been decrypted.

What follows is based on an invited talk delivered on 24th June 2000 at ameeting entitled "History of Cryptography"held by the British Society for theHistory ofMathematics.

The Newmanry Report

By good fortune I was able to announce at that meeting my receipt a few daysearlier ofthe following memorandum from the Government CommunicationsHeadquarters (GCHQ):

MEMORANDUM

To: PROFESSOR DONALD MICHIE

From: ANN THOMPSON, GCHQ

Date: 21 June 2000

RELEASE OF 'NEWMANRY

This confirms that the two volumes of 'NEWMANRY' are to bereleased to the PRO. Weexpect them to be available for public scrutiny in the next few months.

I reproduce below the Table ofContents. The document as a whole is inpreparation for publication on the Web by Dr Whit Diffie, and in hard copy byMIT Press. Costs ofconversion ofthe original typescript into suitable formshave been covered by grants to Dr J.V.Field as editor-in-chieffrom the RoyalSociety ofLondon and from theRoyal Statistical Society, with activefacilitation from the Public Records Office.

Page 6: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

GENERAL REPORT ON TUNNY

With Emphasis on Statistical Methods

TABLE OF CONTENTS

PartO

01

Part 1 INTRODUCTION

1112131415

Preface

German TunnyCryptographic AspectsMachinesOrganisationSome Historical Notes

Part 2 METHODS OF SOLUTION

21 S<22 St23 M24 R25 C26 W27 C:28 L;

Part 3 ORGANISATION

31 M32 M33 X34 R35 T;36 C37 M38 W39 Lj

Some Probability techniquesStatistical FoundationsMachine SettingRectanglingChi-breaking(from Cipher)Wheel-breaking (from Key)CribsLanguage Methods

Mr Newman's SectionMajor Tester's SectionKnockholtRegistration and CirculationTape-making and CheckingChi-breaking and CribsMachine SettingWheel-breaking (from Key)Language Methods

Page 7: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

94 QEP ResearchMechanical Flags95

Part 4 EARLY METHODS AND HISTORY

41424344

Part 5 MACHINES

Part 7 REFERENCE

Part 9 APPENDICES

919293

The First BreakEarly Hand MethodsTestery Methods 1942-4Hand Statistical Methods

General IntroductionDevelopment of Robinson and ColossusColossusRobinsonSpecialised Counting MethodsCopying MachinesSimple MachinesPhotographs

Raw Materials and Productionwith Plans of Tunny Links

Glossary and IndexNotationBibliographyChronology

Conclusions

5202MotorRectanglesThrasher

Page 8: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

The technical facts concerning the breaking ofthe Fish codes, starting withTiltman's initial break in December 1941, can be put together in moderate detailfrom a variety of sources, some listed at the end of this paper. The release oftheGeneral Report on Tunny now allows the level of detail as regards machineprocedures (Newman's section) to be substantially extended and refined.

The Testery report

Still to reach the public domain is a technical report on the work ofColonelTester's section. On cessation ofhostilities, I was assigned to prepare the latterdocument, and completed it in a month or two before joiningGood on the"Newmanry" report. Although the Testery report remains classified, I wasrecently enabled by special arrangement ofthe Director to visit GCHQ atCheltenham and to refresh my memory of it. I also received guidance fromGCHQ's ChiefMathematician as to which details in it cannot yet be disclosed.

A good deal ofknowledge ofthese hand procedures is directly inferrable fromsuch sources as those previously referenced, includingthe "Newmanry" Report.The full "Testery" Report amplifies this knowledge.

TheTestery's changing roles

Before the establishment ofthe Newmanry, the finding ofall wheel settings,and ofvirtually all wheel patterns, was done by hand in the Testery on samplesofpure key. On rare occasions, such as that which Tiltman had seized upon,samples ofkey could be reconstructed from careless mistakes made by theGerman operators ("depths", see later). On evenrarer occasions a free presentofpure key was made by the even more careless error ofomitting properly toinsert the plain-text message tape into the Lorenz machine's tape reader whenstarting the transmission. A message consisting ofall blanks could result, or ifthe operator was inadvertently leaning on some arbitrary part ofthe keyboard,then ofcontinual repetition ofthe same character.

After the successful launch ofthe Newmanry, but before the new machinemethods were developed to full capability, the Testery retained soleresponsibility for breaking wheelpatterns . But settings ofthe chi wheels werenow done at speed in the Newmanry from the cipher-text intercept without needofkey. Machine methods for additionallybreaking chi patterns weresubsequently developed and made operational by an extension ofColossusdesign, the earlier mentioned "special attachment".

Page 9: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

It would be a mistake to thinkthat the Testery 'srole somehow lost its criticalitywhen it proved possible by use ofthe Newmanry's Colossus machines to breakboth chi-wheel patterns and settings at rates unthinkable by hand methods. Inactuality this development created a new and even more critical role for theTestery.

The Newmanry stage ofthe operation simply yielded a first intermediateproduct, in the form ofthe original intercepted message with the chi-wheelcomponent ofthe total key stripped out. These partially solved texts from whichthe effects ofthe five chi wheels had been removed, were called "de-chi's". Afurther chain ofcryptanalytic successes in the Testery had then to take placebefore an end-product could finally be mounted on the Testery' selectromechanical "Tunnymachine" and translated into plain text. Each furtherstep in the chain required separate elucidation from de-chi text (not key),whether in the form ofpattern breaking, or ofthe much commoner task offinding the start-settings (a) ofthe five psi-wheels and (b) ofthe two motor-wheels.

On arrival from the Newmanry ofthe swellingflood of de-chi's, components(a) and (b) ofthe original obscuring key, thus remained to be manually brokenand mechanically stripped out to finally yield plain text. This task was possible,and to the war's end only possible, through thetechniques, experience andingenuity ofthe Testery's complement ofcryptanalysts and ATS operators.

Post-war loss of log books

As was also necessary with the "Newmanry"report, declassificationrequiresthat a decision be taken in concert by GCHQ and the US National SecurityAgency. Eventual declassification ofthe "Testery report" can hardly be indoubt. In a differentcategory is the irreparable loss ofno fewer than 500 hand-written log books, including six exercise-books numbered R0 to R5, in whichwe recorded and initialled research ideas and points of discussion as they arose.Existence of an additional book labelled R4l is also mentioned in the Report.The GCHQ archivists have made painstaking searches at myrequest withoutresult. It should be borne in mind that BP's successor organization was twicephysically uprooted after the war en route to its permanenthome atCheltenham. In suchrepeated house-cleaningit would not be too surprising ifthe much-thumbed, ragged and informal appearance ofthe documents causedthem at some stage to be taken for junk, and thrown out.

Page 10: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Topics to be covered

The audience at my June 2000 talk covered so broad a spectrum ofbackgroundsthat I concentrated solely on expounding some basics and telling a story.

Colossus

In expansion ofthe same plan, I start with a brief overview ofthe Colossusseries ofmachines, thenrelate some extensions oftheir later scope, and finallycomment on their wartime impact on military intelligence.

Depths

Three years before Colossus, theFish story started with the breaking by hand ofwhat was called a "depth". I give an explanation ofthe phenomenon and ofitsexploitation in the Testery to recover complete lengths ofpure key.

Vernam system

Treatment of depths involves a look at some cryptological basics, with specialreference to certain properties ofthe Vernam teleprinter code on which the logicofthe German Lorenz ciphers were based. These are exhibited, including an all-important property wherebythe "differencing" of a string of teleprint codeproduces what we called its "delta" form, in which each successive repetition ofany character in the undifferencedtext is automatically flagged in the delta textas a "/".

Turingery

Once William Tutte had disinterred the overal logical structure common to allFish ciphers, Turingery could be applied, whenever pure key was available, toany sufficient long sample ofpure key. With this methodrecovery was possibleboth for the particular wheel-patterns employed to encipher all traffictransmitted over a particular link for the duration of that pattern set, and todiscover the wheel-settingsofthose ofthe messages intercepted on that linkthat were components of a depth. A brief description ofthe method will begiven. But powerful though it was, its domain of applicabilitywas limited tocomparatively rare gifts offortune when German operators strayed from theirprescribed path.

Page 11: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Newman's proposal

Newman's proposal for a mechanized attempt on wheel-setting comes next. Hisidea was to recover the wheel settings from non-depthcipher-text messages.These were the the overwhelmingmajority ofall intercepts. Clearly large-scalestatistical processing, impossible by hand methods, offered the only chance.

Demonstration of feasibilty of machine wheel-setting

Demonstration of his concept in the newly formed Newmanry was obtained byNewman's complement oftwo cryptanalysts. I had been recruited from theTestery and was shortly joinedby I.J. Good, from the Enigma Naval Section.Our capabilities-analysis and use ofthe custom-built "Heath Robinson",forerunner ofColossus, enabled Newman to commission the Colossusprogramme and to recruit a new influx ofcryptanalysts. There followed a risingcurve ofcollective innovation both cryptanalytic and organizational. Leadingroles were played by David Rees, Shaun Wylie among others in effecting aheadlong transformation ofthereading ofthe German military traffic,eventually to a factory production-line scale as earlier outlined.

Selective anecdotalrecap

Finally I return to the beginning andre-traverse selected topics ofthe story,expanding them anecdotally from a personal perspective. Arising from this, Ihave added material concerning an impact on "Fish" ofa mathematical methodof great generality devised by A.M. Turing for his work on the German navalcode Enigma, but later employed also in the Newmanry. The widespread notionthat his war-time cryptanalytic contribution was limited to Enigma falls farshort ofthe facts. The true authorship ofwhat we knew at the time as "Turing'ssequential Bayes rule" is here disclosed for the first time.

Incompleteness of coverage

From the topics covered, there are several conspicuous omissions. Nodescription is given here ofthe "rectangling"procedure with which thebreaking ofchi wheel patterns was kicked off, and no technical account ofhowthis and the subsequent steps ofbreaking the chi wheel patterns was turned intoa Colossus-aided procedure by special extension of Colossus capabilities. Norare the Testery's manual procedures described by which the psi-wheel and

Page 12: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

motor-wheel patterns and settings were obtained from the "de-chi's" whicheventually streamed over from the Newmanry in bulk. In what is already alengthy treatment, it did not seem reasonable to plunge into such depth anddetail. Much ofthe material is alreadywell expounded by Frank Carter (1996,1997) in technical reports available from the Bletchley Park Trust. Whatever ofinterest that is not to be found there willbe treated inmy chapter in J.B.Copeland's forthcoming book.

Diktat versus interlock

The paper concludes by considering a seemingly baffling question. How couldit be that, with the means of an operationally unbreakable code at theircommand, so efficient an opponent could yet hand us on a plate the "depths"that permitted the initial breaks?

The major cause is identified as a tropism towards placing embargos on humansrather than engineeringthem into devices. The tropism is not peculiar to anyparticular nation or epoch. Bureaucracies everywhere prefer immediatelyapplicable measures to those requiring additional planning, particularly ifthelatter may entail budgetary outlays or new administrative arrangements. It isconcluded that an active arms-race, rather than mutually assuredimpenetrability,will be thecontinuingpattern ofcryptology into the foreseeablefuture.

Colossus, the first high-speed electronic computer

In the BBC television programme "Station X", subsequentlypublished in bookform, Michael Smithremarked:

Colossus was the first practical application of a large-scaleprogram-controlled computer and as such the forerunner ofthe post-war digital computer.

Note that it was not a stored-program machine, and hence not a general-purposecomputer. In this sense its logic was more primitive than Charles Babbage'snineteenth-centurydesigns for his uncompleted Analytical Engine. The world'sfirst general-purpose digitalcomputer became operational in June 1948, whenthe Manchester "Baby" ran its first non-trivial program. Kilburn and Williamswho led the team had been appointed by Max Newman at the end ofthe war,

Page 13: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

using money from the Royal Society specifically for the purpose of developingsuch a machine.

In the light ofNewman's wartime part in conceptual design and practical use,the forerunnerrole ofthe Colossus machines was rooted not only in theirlogical affinities to the post-war digital computer but also in the qualitities andconsequences of a great mathematician and an extraordinary man, my wartimeboss from early 1943 until the summer of 1945,Max Newman.

Ten years earlier, in 1935, as a young Cambridge undergraduate Alan Turinghad attended Newman's lectures on the logical foundations ofmathematics. Theexperience led him directly to formulate in a now famous paper a weird-seeming mathematical construction known today as the Universal TuringMachine (UTM). We no longer see it as weird. It formed the theoretical basefrom which not only Newman at Manchester, but also Womersley at the UK'sNational Physical Laboratory and yon Neumann in the USA, launched theirimmediate post-war machine-building initiatives. Today's multiplying varietiesofcomputing machine, from the smallest hand-helds to the largestsupercomputers, are still formally describable as engineering approximations toa single invariant abstract specification, the UTM of 1936. As earlier stated, theColossus machines were not general-purpose, hence not UTM's. The Colossus1 , however, marked a small step inthat direction, and the later Colossi marked afurther step in programmability.

The Colossus 2, 3, ... 9 crash programme

Colossus 1 had facilities for plug-board programming. By manipulation ofconnections and hand-switches, up to 100 boolean functions of selectedchannels of a running tape could be simultaneously evaluated on the fly. Hard-wired branching was also possible, for example to effect conditional printing,conditional, that is, on current values of intermediate computed results. Asearlier mentioned, I.J. Good (1994) helped me to test a proposal for anunorthodox use ofthe machine which conferred an extension of functionalityfar beyond that of Colossus 1 . The resulting engineeringcrash programmeleading to the Colossus 2, 3, ... 9 machines nudged the design further inthedirection of "programmability" in the modern sense. In a taped interview withChristopher Evans (undated) in the mid- 1970s Newman speaks in guardedlanguage (some elements ofwartime secrecy were still in force) about

Page 14: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

new problems for, I suppose, reasons perhaps I shouldn'tmention, but which had to be dealt with by doing somethingnew with the Colossus itself and it was a great tribute toFlowers' design of this thing that he ... made it a bit moregeneral than we had asked him to [in Colossus 1] in such away that when we had these new problems we often foundthat it could be done on the machine without anymodification of it, just as it is. This involved a lot ofwork byvarious people. By Good and Michie particularly and all ofus ... but it was very satisfactory to find that this machinecould do this and this is perhaps ... one ofthe things whichmakes it justifiable to say that it is really at least aforerunner and perhaps the first germ of a computingmachine; a general purpose computing machine.

In partial corroboration ofthis, as a "fun" project after the German surrender onMay Bth 1945 Geoffrey Timms programmed one ofthe later Colossi to multiplywhole numbers. In practice, only very elementary multiplications could bedone; otherwise the machine cycle interrupted the process before the calculationhad completed. None the less, Timms thus demonstrated the in-principleapplicability ofthe later Colossi to problem domains far beyond the original.

Spanning

Among enhancements to later machines a design extension was originated byI.J. Good called "spanning". It enabled the user to screen off selected segmentsofthe data tape from the machine's current inspection, e.g. if suspicion aroseduring processing that sections ofa transmitted message might be offset withrespect to other sections. This could occur as a result of losses duringinterception of individual characters, or of interpolations of spurious characters.Spanning allowed statistical computations to be repeatedly performed ondifferent selected subsets of a given intercepted message which might besuspected ofhavingbeen corrupted in one or another of these ways. Practicalusefulness of spanning for exploratory analysis was very great.

Comparison with ENIAC

The Colossi, then, were special-purpose in practice, but inprinciple not entirely.They were probablyroughly comparable in functionality to the US post-warelectronic computer ENIAC, operational in 1946. As sheerperformance

Page 15: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

greyhounds, however, there is no comparison. The later Colossi, for example,could read 5-channel paper tape at the astonishingrate of 25000 characters persecond.

Depths

A depthconsisted of two transmissions in succession using the same twelvesettings ofthe wheels that generatedthe key, both transmissions beingencipherments ofone andthe same plain-languagetext. One would think in thatcase that the two transmissions would send one and the same character stringsas cipher text. This wouldbe so ifthe Lorenz machine operator re-started bypulling theplaintext tape back to precisely the same starting point in themachine's tape-reader preparatory to the second transmission. But a morenatural way wouldbe to pull the tape back a few characters into the length ofblank tape which ordinarily precedes the start ofthe plain-text message. In thisway the operator assures himself that the initial characters ofthe plain-textcannot be lost from the transmission. Unfortunately the result may besomething like (writing "b" for each step ofblank tape through the teleprinter'stape reader):

PLAIN-TEXT 1: bbbbbb9 S PRUCHNUMMER9 .PLAIN-TEXT2: b b b 9 S P RUC HNUMME R 9

The properties ofthe Vernam teleprintcharacter-code used by the Lorenzmachines entail that when any character is added on top of another, theircombination obeys the following simple laws:

For any teleprint character X,

(1) X+ X = "blank", usually symbolized as "/".

For any substitution of teleprint characters saitisfying

(2) X+Y = Z, the following necessarily hold:

(3) Y+Z = X and (4) X+Z = V,

Now suppose that:

Page 16: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Xrepresents a message's plain-text,V represents the stream ofcharacters constitutingkey,Z represents the cipher-text,

Then (2) says: plain-text plus key gives cipher-text, (encipherment)(3) says: key plus cipher- text gives plain-text, (decipherment)(4) says: plain-text plus cipher-text gives key (as whenkey is

extracted from the successful breaking of a depth)

Operations (2) and (3) correspond to text-transformations at the sending andreceiving end respectively of a Fish link. Notice that the above relations entailthat in effect precisely the same machine is used by an operator, whether he isin sending or in receiving mode. For sender and receiver to cause their twomachines to operate as "the same machine", they need the 12 wheel patterns ofthe two machines to be the same, and the 12 wheels to be set to the samestarting positions ("settings"). What must not happen, ifcomplete security is tobe preserved, is the transmission of two different messages on differentoccasions (the earlier two "spruchnummer" transmissioins count as twodifferent messages) but using "the same machine". Let us see what happens,making the appropriate substitutions for X, V and Z:

first transmission: plain-text1 + keyl = cipher-text 1

second transmission: plain-text2+ keyl = cipher-text2.

Now add the two cipher-texts:

Theresult is:

The 32 characters ofthe teleprint alphabet includes one, symbolized as "/", thatstands for blank tape. Adding it to anything gives the same thing backunchanged. It can be inferred from (1) earlier, and is indeed the case that:

for any X , X plus Xalways gives blank. So in the above case

plain-textl + plain-text2+ keyl + keyl.

keyl + keyl gives blank tape,

leaving justplain-textl + plain-text2.

Page 17: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

When, as in the "spruchnummer" case, plain-textl and plaintext2are no morethan the same text at different off-sets from the start, as in

PLAIN-TEXT 1: I I I I I I 9 S PRUCH NUMMER9 .PLAIN-TEXT2: / / / 9SPRUCHNUMME R 9

then we add the two plain-texts and call the resulting text a "depth". Here /, theofficial symbol for blank, has been used in place ofthe earlier "b". Just as amatter of interest, character-by-character addition ofthe two streams gives:

///9SPC4OVJD3TI PO

This character-string is myreconstruction from information in Sale's note (seereferences) ofthe first 17 characters ofthe historic depth of 30 August 1941broken by John Tiltman. Since it was at that time habitual for operators oftheTunny link to start a message by giving its message number, the first thing thatTiltman would undoubtedly have done wouldbe to extrapolate as follows:

///9SPC4OVJD3TIPOI I I I I I 9 SPRUCHNUMMER9 and add, - yielding:///9SPRUCHNUMMER9. . .in satisfactory confirmation ofhis guess. How one proceeds further to extendsuch a break is illustrated in theconcluding anecdotal recap ofthis paper.

Vernam teleprinter logic

For encrypting text represented as 1 's and o's, "Boolean addition" as we usedthe term at BP expresses the equations:

o+o= 0, o+l = 1, I+o = 1, I+l = 0 where "1" is read as true, and "0" as false.

Example: take the five-channel teleprinter alphabet, in which the last two itemsofthe character- stream below mean "space" and "blank" respectively. Tobridge the gap between symbols and physical paper tape, note that "1"designates a hole punched in the tape and "0" designates a no-hole. So thecharacter symbolized as "9" is encoded by a single hole punched in channel 3and cause the teleprinter thatreads that tape to advance one character without

Page 18: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

printing which is why it is the typographical equivalent of"space". Thecharacter symbolized as / is encoded by the absence ofholes from all channels.Its action when read by the teleprinter is nil. Unlike "space", "blank" cannotform a legal character in a plain language teleprint message. There are twoother such characters, symbolized as "3" and "4". Here is the completerepertoire. It is clear, or can quickly be seen as clear, that the number ofpossible 5-channel combinations of O's and 1 's is 2 x 2 x 2 x 2 x 2, otherwiseknown as 25 , or 32.

ABCDEFGHIJKLMNOPQRSTUVWXYZ+B439 /

1101110001100 0001010101 1 1 11100001010001011110 0011100110 0 0 01110000010010110101 1011010110 1 1 00100100111011001101 1100100011 1 0 01101000100001100011 01110010 11 1 1 1110000

The first non-alphabet character above is in my memory symbolized as "+",although the Newmanryreport has it symbolized as "5". This stirs an echo. ButI shall here use "+" throughout, pending resolution ofthe divergence of usage.

To cause aFULL STOP to be printed at the receiving end, the operator has firstto hit "+", for "shift to upper case", then "M" for "full stop", then "8" for "backto lowercase", then "9" for "space-bar", making the character sequence +MB9.Teleprinter keyboards, unlike those of typewriters, lack means for effectingsuch shifts as single keypresses.

Now check: "+" + "M" = "U", "M" + "8" = "A", "8" + "9" = "+"Differencing

Conversion of+MB9 into UA+ results from adding each character in themessage text to its successor. The process was called differencing anddifferenced text was known as delta text. Thus.

To make sure that the case-shifts have "taken", the operator has represented theFULL STOP by doing "++MBB9" for safety. In a five-channel tape representation++MBB9 looks like this:

++M88 9 ?? + + M 8 8 9

? / U A / + ?

Page 19: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

+ + M 8 8 91 1 0 1 1 0 As earlier sketched,110 110 ifwe "difference" the0 0 1111 string of characters,111110 using the boolean111110 additionrules, we get:

/UA / +0 110 10 110 10 10 0 0000 0 1000 0 1

To give some idea ofhow non-randomness can be unmasked by the simpleoperation of differencing, try adding either the top two or the bottom twochannels of this delta text. In both cases the answer is:

000 0 0

In this particular case it so happens that the additions chance to have the sameeffect using the undifferenced text. So to show how just any abundance ofrepetitions gets picked up, I will invent an imaginary Australian tribal word or

Addition of channels 2 and 4 is observed to give the sequence:

name: JJbjoonnawoolloo.

800 NNA W O O JL LOO _?1 0 0 0 0 110 0 0 0 0 0 00 0 0 0 0 110 0 1 10 0 0 What happens if we difference0 0 0 110 0 0 0 0 0 0 0 1 the string ofcharacters?1 1 1 110 0 11 0 0 11 01 1 1 0 0 0 111 J_ 11 1 I0

We get:E / H /KTJ / R / R/ / M10 0 0 10 10 0 0 00 0 00 0 0 0 10 10 10 10 0 00 0 1 0 10 0 0 0 0 00 0 10 0 0 0 10 10 10 10 0 10 0 1 0 0 10 0 0 0 00 0 1

Page 20: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

0 000000000 0 01 with 12 out of 13 "o's where randomexpectation is 6.5. This near-perfection arises from the fact, unintended when Iconcocted the example, namely that addition of these same channels in theundifferencedtext yields:

11111111111110

Whether or not this "delta (2+4)" tendency holds up in large samples ofplainlanguage, even as a modest but exploitable statistical bulge over the randomexpectation of50:50 for o's and 1's, wouldremain for investigationby thecryptanalysts, ~ supposing that they wished to read Fish-encoded messages inthe imagined tribal language. My purpose here is to illustrate that large-scalemachine analysis ofthe statistical properties ofplain language, by cryptanalyststrained in statistics rather than in linguistics or inthe language itself, is anessential first step in any attempt to break a new code for a new traffic.Empiricism is all. This theme came to the surface in an unexpectedlysituationthat arose duringthe birth ofthe Newmanry, as will later be told.

Further principles ofVernam-based ciphers

To encipher is to turn plain-text into cipher text. In the following, by the symbol" + " we always denote "boolean addition", i.e.

Recall that for enciphering, the relevant equation is:

PLAIN-TEXT +KEY-TEXT = CIPHER-TEXT.

To decipher is to turn cipher text into plain text. For this the relevant equationis:

CIPHER-TEXT + KEY-TEXT =PLAIN-TEXT

In a useful analogy, this operation ofboolean addition is known in propositionallogic as the "exclusive or" meaning "EITHER the first OR the second BUTNOT both". Equivalently, ifthe two summands are the same (i.e. both 0 orboth 1) then the answer is 0; otherwise the answer is 1, as we have seen in theearlier examples.

o+o= 0, o+l = 1, I+o = 1, I+l = 0.

Page 21: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

As an example ofencipheringa teleprint message by adding KEY to PLAIN, letus add a five-channel teleprint message to some key:

T H E 9 FOX9ARRIVED9/001010101000000000001111010110110001000011100110110001100000

0 110 0 MESSAGE10 0 0 010 0 1010 10 010 0 0 0

The top character string is plain-text. The next is key, and its two-way functioncan be seen by trying the effect of adding thekey to thebottom sequence,which is cipher text. Theresult is:

THE9FOX9ARRIVED9/

which gets us back to where we started. This is the distinctive attribute oftheadditive method for encipheringteleprinter messages invented in 1918 byGilbert Vernam in America, adopted by the Lorenz company as the basis oftheir cipher machine.

Observing that in an ideal world containing trulyrandom key:

MESSAGE (plain-text) + RANDOMI (key-text) = RANDOM2(enciphered message),

w M V1 0 1 1 1 1 1 0 1 1 0 1 0 0 0 0 1 KE1 1 0 1 0 0 0 1 0 0 0 1 0 1 1 0 11 0 0 0 1 0 1 0 1 0 0 0 1 1 1 1 11 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 00 0 0 0 1 0 0 1 0 1 0 1 1 1 1 1 1

w R1 0 0 1 0 1 0 0 0 1 0 1 0 1 1 0 1 CIPHER1 1 0 1 0 0 0 1 1 1 1 0 1 1 1 0 11 1 0 1 0 0 0 1 1 0 0 1 0 1 1 0 11 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 01 1 0 0 1 1 1 1 0 1 0 1 0 1 1 1 1

Page 22: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

and that

RANDOM2+ RANDOMI = MESSAGE,

why not use a random string as key?

Problem: how does the receiver get hold ofthe key used by the sender so as todo the reverse transformation trick andrecover the original plain text? Thepossible solutions are:

1 Both ends have the same "one-time pad";

2. Thereceiving end re-manufactures the key on the fly,by usingthe same recipe as the sender.

The solution chosen by the German high command was along the lines ofthesecond solution. Acknowledgingthat by definitionthere is no recipe forgenerating truly random text, employ the best pseudo-random generating recipethat money could buy, and have it hardwired into the circuitry ofthe requiredenciphering/decipheringmachines. A machine should be sufficiently small thatan army group headquarters on the move could take it along,with its operators,in an armoured car. The task was given in the mid 1930's to the Lorenzcompany.

Random and pseudo-random

As stated, the definition ofrandomness entails that no recipe can exist formanufacturing it. The challenge for the Germans then was to develop a recipefor generating a "pseudo-random" sequence according to such a complexrecipethat reverse-engineering it from arbitrarily large intercepted samples ofciphertext would be infeasible inpractical terms. Putting it colloquially:

NON-RANDOM + RANDOM = RANDOM

NON-RANDOM + ALMOST-RANDOM =ALMOST-RANDOM

The Schluessel-zusatz 40 machine was developed and built by the Lorenzcompany to the requirements ofthe German highcommand. But the statisticalproperties ofNON-RANDOM, in this case German military language text, werevery far from RANDOM, and the properties of ALMOST-RANDOM, in this casethe Lorenz-generated key, were sufficientlyfar from RANDOM that to qualifythe resulting cipher-text as ALMOST-RANDOM is perhaps too generous. Indeed,

Page 23: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

thanks to the counter-productive complication introduced with themotor wheels(the cause ofthe "synchronized stuttering" ofthe psi wheels) the machine-generated key proved to be "more PSEUDO than RANDOM". This bon mot isfrom Tony Sale. It was his heroic post-warrebuild in the early 1990's oftheessentials ofColossus 1 that paved the way for rescuing the greater part oftheBletchleyPark site as a publicly accessible museum under the curatorship oftheBletchley Park Trust.

What was Turingery?

Given a length ofkey the first step is to difference it and to work with delta key.Then it becomes possible directly to exploit the following facts concerningrawundifferenced key.

CHI-STREAM = CHARACTER-STRING GENERATED BY ALLOWINGTHE CHI WHEELS, ONCE SET, EACH TO STEP ROUND ITS OWNPROPER CYCLE

of41, 3 1, 29, 26, and 23, repeating the stream only after every 41 x3l x29x26 x 23 steps. But note that when each channel is isolated for study, we canobserve that channel 1 ofthe chi-stream repeats every 41 steps, channel 2 every3 1 steps, and so on. It is therefore a trivial matter, given a length ofchi-streamexceeding 41 in length to recover the patterns and settings ofthe five chiwheels. In the case that we have a similar length of delta chi-stream, exactly thesame reasoning applies.

PSI-STREAM - CHARACTER-STRING GENERATED BY ALLOWINGTHE PSI WHEELS, ONCE SET, EACH TO STEPROUND ITS OWNPROPER CYCLE WITH IRREGULARLYPLACED PAUSESSIMULTANEOUSLY AFFECTING ALL FIVE WHEELS

The placement ofthe pauses is determined by the occurrence ofo's in aMOTOR STREAM of o's and 1's in whichthe 1 's predominate. Its preciseconstruction by the action ofthe two motor wheels will not concern us.

A fragment ofextended psi stream might look like this:

KEY - CHI-STREAM + PSI-STREAM

Page 24: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

HRMMSAAACPS JJVE

Note that the five psi wheels generating the "extended" stream either all changetogether or all remain the same from one position to the next. The delta streamlooks like this:

VP/ BI / /FOWC/Y8000 10001010110 0 10 0 0 0 11110 0 10 0 10 0 110 0 10 0 0 110 1110 10 0 0 0 110

0 1 10 0 10 1 10 0 10 1 1

Ifwe had a sufficient length ofpsi-stream, we could perform the same recoverytrick as before illustrated for the chi-stream, except that in the case ofundifferencedpsi stream we must first delete all the character-repetitions, andin the case ofdelta psi-stream we must delete all the /'s before proceeding. Ofcourse occasionally a repetition or a / respectively may occur by random chanceand not through the action ofthe motor stream. In that case checks will fail andthe cryptanalyst must back offand reconstruct what has happened and exactlywhere. This he must do by processes of successive conjecture and refutationbefore continuing and completing the job.

But what we now have is neither delta chi-stream nor delta psi-stream but theirsum, i.e. delta key. Next, therefore, we need to apply some way offirstextracting from this the delta chi. What follows is overly condensed andtechnical, but is included for the benefit of sufficiently interested professionalcryptologists. Otherwise the reader is invited to skip to the nextparagraph.

Channel by channel we do the following. Assume a value for a given bit in thedelta chi wheel, either oor 1, and repeat this assumption at the appropriate chiperiod throughout the differencedkey. From the agreement or disagreement ofthese assumptions with the values found at these periods, inferences are madeas to whetherthe true value or its reverse occurs at that point, and by correlating

0 0 0 0 1 1 1 1 0 0 1 1 1 0 10 1 0 0 0 1 1 1 1 1 0 1 1 1 01 0 1 1 1 0 0 0 1 1 1 0 0 1 00 1 1 1 0 0 0 0 1 0 0 1 1 1 01 0 1 1 0 0 0 0 0 1 0 0 0 1 0

Page 25: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

the results of these inferences the delta chi patterns (and from these the chipatterns, and from these the psi and motor wheel patterns) are obtained.

Newman's proposals

The limitation on Turingery and on almost everything else done in the Testerywas the requirement to have key-text to work on, obtainable only from a smalland diminishing number of intercepted depths. Newman based his proposed afrontal assault on the cipher text itself, hoping for sufficient occurrence ofnon-random features ofthe enciphered plain text.

A small example ofregularity in the plain text was given above, using thekeystroke sequence that implements the full stop on a teleprinter. Operationally,it was not actually a "small" example. Military German was crammed withabbreviations, marked by full stops, or by other punctuation signs whose key-stroke profiles showed similar statisticalregularities. These were found to bemost marked in plain text when the first and second delta channels werecombined by addition. By cancelling out to zeros this provided frequent"windows" for viewing snatches ofunderlyingkey. As for the deltakey itself, acomponent of it (deltapsi) was also sprinkled with blanks resulting fromcharacter-repetitions inthe non-delta psi-stream. This provided a low butexploitable frequency ofpeep-holes through which to catch glimpses of delta-chi.

In exploitation of this behaviour, delta-ing the key-text in order to recover deltachi wheels was howTutte got in in the first place. As we have seen, interceptionofrare depths made it possible to recover and work on pure key. The Testeryused deltakey for "Turingery", both to break wheel patterns in the first place,and then, less laboriously,the wheel settings ofeach individual messagesubsequently intercepted during theperiod (initiallya month) for which thatparticular set ofwheel patterns was operative for the given link. But as thelaxity of German operators' practice dwindled, so didthe already sparseavailability ofpure key, - for lack ofnew depths.

Max Newman's proposal thus extended to intercepted cipher text some oftheprinciples that were applicable by hand only to pure key. The idea was aimed atthe statistical tendency for delta plain text, and particularly booleancombinations of selected channels ofplain text, to have more zeros than 1 's andhence to represent a pale and fuzzyreflection ofunderlyingdelta key. Perhapsdelta cipher would be just sufficiently like deltakey, that although hand

Page 26: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

exploitation of this correlation was out ofthe question, fast machine processingmight bring it within striking distance. Newman's insights led first to the"HeathRobinson" electronic machines and then to the Colossi.

The plan was thus to operate on such a scale of sample-size and speed that thestatistical properties ofmilitary German plain text contained a sufficientfrequency ofcharacter repetitions to combine with the strongly above-randomfrequency ofrepetitions in the stream ofpsi-wheel characters. Then, after delta-ing both character-streams a weakly above-random frequency ofthe all-zeroscharacter:

00000

would serve as an intermittent "window" through which the delta-chi at thispoint could be glimpsed. A logical consequence would be that ifthecryptanalysts were in possession ofthe chi wheel patterns, but not oftheirsettings (starting points), then hypotheticallyan exhaustive search through allthe 41 x3 l x29x26 x 23 possible settings, on each trial setting, adding thedelta chi stream to the deltaplain message stream, would leave just delta-psi +delta-plain. This might reveal itselfby the aforementioned faint statisticalexcess or "bulge" in the number ofall-zero characters. Because oftheastronomical combinatorics of such a search, Newman followed an earliersuggestion ofTutte's that logical combinations of, say, only two selectedchannels out be processed at any one time, with an indication that combiningthe first two channels as delta (I+2) might be the place to start. This notionwillbe made clearer as we go along.

Enter Heath Robinson, engineers, Wrens and Good

Funds for Newman's proposal were granted. Tworooms, comprising Hut 11,were provided. The first machine ("Heath Robinson") was conceptuallyspecified, and then designed and built off-site. A few electronic engineers andsome operators from the Wrens, plus one cryptographer (myself, transferredfrom the Testery) were assembled. A floorful ofpre-fabricated parts wasdelivered and put under assembly-and-test by the engineers, and the Newmanrysprang into energetic if disjointed activity. I was one day sitting at my table in

Page 27: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

solitary wonderment when I saw in the further corner a smallish figureseemingly frozen in meditative yet enquiringreverie. He slowly approachedwith a deliberative step, right arm and hand in semi-extension. I rose and waitedfor the hand to come into range. Atthat point he stopped, and gazedcomposedly upon my bafflement. After what seemed a long while he made anannouncement in tones of quiet precision: "I am Good!"

In April 1987 an international symposium on the Foundations and PhilosophyofProbability and Statistics was held to honour Good's 70th birthday. Inprefacing my own contribution I remarked that my lifetime experience,(continued today), had confirmed that he was indeed Good then, and has beengetting better ever since.

Proof ofconcept: the Heath Robinson

Exhaustive search ofthecombinations ofall five chi-wheel settings at once wasofcourse not remotely possible even for electronic machines. Therefore thestrategy thatNewman had proposed was, as mentioned above, to findcombinations ofchannels in German plain-text messages that were soproductive of statistical regularities that the rest could initiallybe disregardedwithout loss in the size ofthe statistical excesses over chance. In the event, thesystematic studies that I helped Jack Good then to conduct, using the firstelectronic machine (the "Heath Robinson") as a statistician's slave, confirmedNewman's suspicion that adequate relative excesses ("proportional bulges" inour terminology) could be got even after disregarding channels 3, 4 and 5,leaving only 31 x 41 = 1271 combinations ofpossible settings ofthe chi-1 andchi-2 wheels to be tried. Once these two chi wheels were set, matters becamemore problematical. Tackling channels 4 and 5 in like manner sometimes founda marginallysufficient bulge, sometimes not. So more sophisticated statisticalproperties ofplain text had to be pressed into service. But first such propertieshad to be prospected for and discovered, perhaps properties that could be forcedinto the open by exploiting more complex relations between channels.

Advance statistical reconnaissance

The hope, therefore, was that by amassing such statistics, we might eventuallyoperate on raw intercepts as input, withouthaving to rely on depths. Depthswere dwindling and were to become vanishinglyrare. The heaven-sent gift tothe Testery ofpure key obtainable from depths can be appreciated when onerealizes that key can be seen as key + plain-text in the special case that the

Page 28: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

plain-text consists ofthe message: 000000000000000000000000000000,etc.Clearly, here thecharacter repetition frequency is 100%. In other words deltaplain-text + delta key gives all zeroes, just as it would ifthe plain-text messagewere 11111111111111111, say, or GGGGGGGGGGGGGGGGGGGGGGG,etc. Sometimes, by the way, it was, - at least for long stretches, even intheabsense of a depth. My best guess was that the enciphering operator had eitherfallen asleep in midkey-stroke or had carelessly leaned with his elbow on thekeyboard, thus keeping some key depressed. In that case the correspondingcharacter would simply repeat in every machine cycle and be transmitted.

Ifhowever the plain text takes the form of a message in military German, thenonly when this plain text message has a repeated character can this act as a littlewindow of "blank" in the delta cipher-text, through which a character oftheunderlyingdeltakey can be seen. Ofcourse, if the operator "leans on thekeyboard" for any length oftime, the delta text over the correponding segmentwill have a correspondingly extended peephole

Size and frequency of "peep-holes"

How frequent, then, were the peep-holes? Ifdelta plain-text gave 100% o's onall five channels then the delta plain-text would be fully transparent, as with theconjectural sleeping or leaning operator. But these free gifts, as also depths,were getting vanishingly rare. What ifdelta plain-text were generally to give50% zeroes on some given channel, but consistently more than 50 % whencertain channels were combined in some way, such as by addition, the otherchannels being temporarily ignored? The idea then arises to enhance the peep-holes by imposing a logical restriction on some other, third, delta channel, orthe sum of two others, gaining inreturn a more than compensating increase ofthe relative size ofthe effect, — contingent on how well the logical constraintswere designed to fit the actual statistical properties ofplain text.

Let us return to the earlier example, namely the delta of ++MBB9 which is

/UA / +0 110 10 110 10 10 0 0000 0 1000 0 1

Page 29: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

As earlier noted, it so happens that justas the sum ofthe first two channelsgives all zeroes, so also does the sum ofchannels 4 and 5. On channels 1 and 2this regularity was usuallyenough, along with other channel 1 and 2 features ofdelta German plain text, to find by machine the settings ofchi-1 and chi-2. Thefeatures ofplain text on channels 4 and 5 were not always so favourable as toallow the same trick to be applied. Use ofthe plug-board to cause the machineonly to look at channels 4 and 5 at those points which satisfied some conditionon 1 and 2 (e.g. that delta of I+2 = 0) was found to increase the 4+5 bulge to adegree that sometimes more than compensated the loss in effective sample size.Other more sophisticated statistical interactions were required to cope with allchannels in all cases. First, however, far-reaching knowledge ofthe intricatestatistical characteristics ofplain text was required in order to discover whatprecisely these statistical interactions might be.

When the first Robinson became operational, Jack Good and I spent our dayshift in frontal assault, with Maxpacing around forpositive results to announce.Although he agreed in theory with our argument that pure reconnaissance oftheproblem should be the first use ofthe newlyoperational machine, the need forcredibility, with highranking military and others dropping in to see whatresultswere being got, pressed sorely on him. Once he had laid it down, MaxNewmanwas not someone that a person in his senses would continue to oppose. Fromnine to six each day Jack and I accordingly went through the motions.

Ghost shifts

But many evenings were spent in a clandestine ghost shift, with one or twovolunteer Wrens and an engineer. Our purpose was to use the new instrument togather massive delta plain-text statistics, including in particular the frequenciesof zeroes (i.e. repetitions in the original plain-text) in the boolean sum ofselected pairs ofchannels, conditional on what was happening on otherchannels.

I say above "conditional on what was happening ..." This was all-importantafter the first two chi wheels had been got, yieldingknowledge ofthe first twochannels of delta-psi + delta-plain. To get knowledge of further, and possiblymore recalcitrant, chi wheel settings, we needed somehowto sharpen thestatistical "bulges" characterizing other channels ofdelta plain text. On theHeath Robinson we could only screen out channels unwanted in a givenrun byconcocting tapes in whichthe unwanted channels were left all blank. In theColossus, this conditionalizing was later done at the flick of a set of hand-

Page 30: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

switches, together with plug-board programming for forming arbitrary booleancombinations of selected channels.

With the aid ofHeath Robinson and our volunteer assistants we systematicallyextracted from Testery decrypts batteries of general rules governing thestatistico-logical structure ofmilitary German, with and without delta-psistreams superadded. Owing to the action on the psi's ofthe motor stream, thislatter component was guaranteed to supply a stream ofextra zero's.

Armed with these tabulations, statistical summaries and empirical rules we werenow in a position to make frontal assaults in earnest. This yielded sufficientoperational success for MaxNewman to announce feasibility. I doubt ifhe everknew ofthe clandestine operation. Ifhe had, his forward-pressing propensity,and preference for focussing on the nextbig thing, would I believe have led himto smile and dismiss it from mind. The next big thing, ofcourse, was Colossus.

Personal prehistory

In late 1941 following my 18thbirthday a normal next phase would be twofurther terms at boarding school, with an option for scholarship holders toproceed to a shortened University degree course before joining up. But over thatChristmas my teenage imagination was fired by a tale from my fatherconcerning a mysterious establishment at Bedford. He had it on the authority ofthe then War Minister, Sir James Grigg, that as preparation for doing somethingunspecifiedbut romantic behind enemy lines there were opportunities to sign upfor a Japanese course starting in a couple ofmonths' time. I duly journeyedtoBedford and presented myselfat the address given.

Sorry, wrong info

Myrequest to enrol elicited from the Intelligence Corps officer who saw me asomewhat puzzled reply: "Who told you that we have a Japanese course now?Thatparticular exercise is planned for the Autumn."Noting my confusion headded: "But we have courses on code-breaking. There's a new intake juststarting. Would that interest you instead? I'll have someone find you a billetnearby. Make sure to be back here at 9 a.m. Monday."

In World War 2 one did not mess about. Returning to the London suburbs justlong enough to pack a suitcase, I was back and signed in to the School ofCodes and Ciphers, Official Secrets Act and all, on the Monday morning. With

Page 31: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

the rest ofthe new class I was soon held in thrall by our instructor, a certainCaptain Cheadle, and by the black arts ofcodes and ciphers.

With nothing to occupy my evenings, I arranged to have my own key to thebuilding and classroom. My habit became to return after hours to the texts andexercises. The resulting accelerated learning curve made my selectioninevitable when a Colonel Pritchard arrived from Bletchley. He was on amission to recruit for the new section that was being formed by Tester to followup Tiltman's and Tutte's successive coups. The hope was that breaking andreading Fish traffic could be placed on a regular basis. The Pritchard interviewlasted no more than a few minutes. I was to present myselfwithin 48 hours atthe entrance to Bletchley Park with a sealed letter.

After admission and a visit to the billeting office, I was parked in the MansionHouse. My first task was to memorise teleprinter code until I could fluentlysight-read punched paper tape. Pending completion ofthe Hut assigned toMajor Tester's new section I sat as an ugly duckling in a large room filled tocapacity by members ofthe Women's Auxiliary Air Force. What were theydoing? Who knows? New arrivals were imprinted with a draconian DON'TASKDON'T TELL principle inregard to anyone's immediate business buttheir own. I did, however, discover that those whose boy friends were on activeservice felt only contempt for an apparently fit young male in civilian attire.Some ofthem had lost boyfriends in the RAF, and many had boyfriends stillalive but in daily peril.

Charm of a second lieutenant

The experience did nothing to ease my sense of disorientation in the newsurroundings. Relief appeared in the person of a uniformed and exquisitelycharming Intelligence Corps officer, second lieutenantRoy Jenkins. My taskwas to bring him up to my own recently acquired sight-reading skills. Roy'spost-war career was to include Cabinet Minister and Chancellor ofOxfordUniversity. In my isolation, his company was rescue and balm. We departed toswell theranks ofTester's new section, in my case via a most curious diversion.

Forty men and a teenager

On reporting to Ralph Tester I was immediately dispatched to take charge of aroom like a small aircraft hangar. It was located at some distance from his newHut. Within it there sat at tables several dozenuniformed men who remain in

Page 32: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

my memory as being all ofthe rank of lance corporal. What I can attest beyonderror is that I quickly became convinced ofthe infeasibility ofthe operationwhich itwas now my jobto supervise.

As earlier explained, once the offset had been determined oftwo messagesknown to constitute a depth they were added so as to cancel out their commonkey. Theresulting text must logically then consist ofthe addition to itselfat thatoffset, or "stagger", of a Germanplain language message. Given the text ofsuch a depth, ifone guessed that some character-sequence, say"GESELLSCHAFT" was likely to appear somewhere in the plain text, then theexperiment could be tried ofadding that 12-character sequence (a "crib") to thedepth's first 12 characters, inspecting the result, then to characters 2-13, 3-14,... etc. in tedious progression through the text, a procedure known as"dragging". One stopped only if the result ofthe addition at any stage yielded,say, "SELLSCHAFT9U"(recall that the symbol "9" denotes character-space).One would at once conclude from such a local break that the offset was 2 andthat the plain text contained the sequence "GESELLSCHAFT9U". In the handsofa cryptanalyst the immediate next step would be to extend by two charactersthe "crib" that had been dragged, yielding,perhaps, "SELLSCHAFT9UNT"which would strongly suggest some further extension, say"GESELLSCHAFT9UNTER" which might possibly be rewarded by"SELLSCHAFT9UNTER9A" and so forth.

Misplaced task-decomposition

Someone had had thereasonable-seeming thought that the task could bedecomposed into a brute force (crib-draggingonly) component and a skilled(extending the breaks) component. Why not first throw brute force at it and thenpass the text on to the cryptanalysts with candidate breaks already found andflagged? A few dozen Intelligence Corps clerks could each be equipped with alist of cribs to be dragged, together withrule-sheets for theboolean addition ofteleprint characters and for the recognition ofcommon fragments ofmilitaryGerman. Let them do the dragging, marking all local breaks found or suspected.Marked-up texts could then be sent on to the Testery proper, to receive theattention ofcryptanalysts whose time would thusbe conserved by priordelegation ofthe drag-work.

It sounded good. Experience soon convinced me otherwise. But myconvictionhad to be validated in the eyes ofothers. My only course was to drive theproject along until its futility became evident, not to the band ofmassed lance-

Page 33: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

corporals, but to the authors ofthe originalproposal, whoever they were (this Inever knew).

The flaw lay inthe non-decomposability of a task once talent and much practicehas melded it into a fluent unity. The cognitive psychologists speak of"automatization". To the eye of an observer from Mars, delivery ofthe serve attennis might appear to be a sufficiently separate and stereotyped task to suggesta change ofrules. The expert player might be allowed to employ a brute-forceserver (crib-dragging posse of lance-corporals) who would on delivery ofhisservice instantly quit the court, leaving the tactically highly skilled tennisprofessional (cryptanalyst) to continue the rally.

Trade-offs can be debated for each separate athletic or intellectual skill, but canonly be quantified empirically, case by case. In tennis, as in depth-breaking,each opening move (the serve) flows smoothly and subliminally into the move-sequence (the rally) that follows. The gains from continuity ofthe single-agentscheme probably outweigh in tennis the sacrifice ofsheer serving speed. Thesame principles were eventually shown to dominate the depth-breaking case.

To the Testery

The dogged endeavours ofmy well-drilled force ofcrib-draggers in due coursegenerated sufficient documentation for me to report that the "human wave"assault was unlikely to contribute effectively and would best be disbanded.After this interlude, depressing for all concerned, I gained the long-sought shoreofthe Testery proper. I was turned over to a young graduate, now theinternationally distinguished mathematician P. J. Hilton, for instruction in theearlier mentioned methodknown as "Turingery".

Peter knew all the Testery hand-procedures backwards and forwards, andplayed a massive part in perfecting them. My first and vivid memory was that,although only a year or two older thanme, he smoked a pipe. My second was ofhis didactic strictures on my fetish oftidiness and aesthetics in paper-and-pencilwork. I should say "my then fetish". With efficiency and speed at anunimaginable premium, not to mention justified awe ofmy new mentor, I wascured for life!

Other vivid images ofmy first encounter with the Testery are first and foremostofMajor (later Colonel) Tester himself. I recall his mesmeric impact on femalespectators in the lunch break as he leapt, daemonic and glowing about thetennis

Page 34: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

court with an animality that I had only ever envisaged as radiating from thegreat god Pan. Yet the same man a year later, when I was already in theNewmanry, was ashen under his tan when he had to summon me (presumablyat Newman's request) to reprove my conduct. Why had I been canvassing thecryptographic staffofboth Newmanry and Testery for signatures to a petitionfor the administrative merging ofthe two sections? With the naivety of anineteen-year-old I was oblivious of such facts as that, even ifa Foreign Officesection and a War Office section could have been merged, one or other ofTester and Newman would have had to be dumped and that it would not havebeen Newman. An ingenious administrative compromise resulted. A fictional"Mr. X" appeared on Newman's books whose fake identity four selectedTestery staff assumed for periods in rotation, acting as a species of internalconsultant. This gave good technical liaison, previously absent.

Testerhad the sense ofpurpose combined with personal humilityof a leader. Atthe time ofRommel's retreat to Tunisia, we suddenlyfound that somemysterious change in the system had locked us out ofthe Berlin-Tunis channel.Agroup of us offered to go flat out round the clock. Ralph's cryptographicskills were really too unpractised to be ofmaterial help, as he and we knew. Buthe sat among us, bolt upright as was normal for him, unflagging as the hoursraced by. In the end the hours were not racing, and we young Turk's weredrooping and nodding. Ralph, focussed and refulgent as ever, saw this: "Youknow," he said tactfully, "it's easy for me. Most things go downhill with age.Stamina for some reason goes the other way. So you're no good at this sort ofthing until you're at least forty. Another coffee, anyone?"

During the glory days ofthe American space programme, whenthe mean age ofspace vehicle commanders seemed to be getting more and more venerable, Irecalled Tester's words.

Strange incident, best forgotten

The Testery 's machine operators were ATS girls ("Auxiliary TerritorialService" I think). One of them, Helen Pollard (now Currie, see References) inher reminiscence ofthe Testery speaks not only ofthe thrill of it all but brieflyhints at a romantic attachment. That attachment outlasted the war. If there is tobe a dedication ofthis memoir then let it be to her.

For all the attractions ofthe new life, orperhaps because of them, I could notdrop from my mind the initial "white feather" impact of that roomful ofWAAF

Page 35: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

girls. While on leave visiting my home in Weybridge, I learned from myfatherof questions from his peers at the St Georges Hill golf club about whathis sonwas doing for the war effort. Apart from knowing that I was not after alllearning Japanese, his mind was unavoidablyblank on what I was now up to. Itwas out ofthe question to give information of any kind to any person outsidethe wire beyond "sort ofclerical work" or the like. He asked me whether I hadever considered active service.

Back in BP I asked for an interview with Colonel Pritchard, and requested atransfer to the North African desert. I pointed out that I was already in theArmy, being recorded as having served for one day in theRoyal Fusiliersbefore relegation to the War Reserve. Pritchard let me finish. Then he said:"Who's been getting at you?" Taken offguard, I waffled. "No-one?" heenquired politely, and let his question hang in the air.

Eventually I blurted out that my father had mentioned such a possibility, buthad applied no pressure. Anyway, I maintained, that had nothingto do with mydecision. There was another uncomfortable pause. Then: "I have to instruct youto return to duty. You see, MrMichie, we have a war on our hands.Inconvenient, but unfortunately true. Unless you have further questions, you arefree to return at once to your Section." Pause. "And by the way, I do not expectyou to raise such matters again." Pause. "Either with me or with anyone else."Longer pause. "As for your father, I do not anticipate that he will raise themeither."

I returned to the Testery and I confess I felt relieved. I don't believe I gave itfurther thought. But many years later my mother told me that my father hadreceived a visit at his place ofwork in the City ofLondon from an armycolonel, who presented himselfas my superior officer. Did I know anythingabout it? I shook my head. For a decade or two after the war, to reveal anythingwhatsoever about BletchleyPark and its activities continued to be embargoedunder the Official Secrets Act. Inevitably its subjective restraints weakenedover long time. None the less 25 years passed before any mention ofBritish usein 1943-45 ofelectronic computers for a cryptanalytic purpose appeared in theopen literature (Good, 1970).

Chess, Turing and "thinking machines"

It was through needing to consult the originator ofTuringery on some point thatI first met Turing. We soon discovered a common interest in chess, and also the

Page 36: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

fact that we were both sufficientlypoor players to be able to give each other alevel game. At BP a person was either a chess-master, having been recruited forthat reason (similarly with winners ofnational crossword competitions) or hedid not count chess among his interests. We formed the habit ofmeeting once aweek for a game ofchess in a pub in Wolverton. On the pervasive need-to-know principle we never discussed his work in the Naval section. When I wasdemobilized I still knew nothing about Enigma, except possibly its name. Ourshared topics of interest were (a) the possible mechanization of chess-playingand (b) learning machines. These interests were inspired in me by him, andwere shared with Jack Good. They continued to occupy the three ofus inoccasional correspondence and meetings in the post-war years until Turing'sdeath.

Dragon and the automatized skills of expert depth-breakers

At a much later stage ofthe war, the role ofthe 40 lance-corporals wasmechanized inthe Testery by an electro-mechanical machine. Called theDragon, it was the brain-child ofAngus Macintosh. Some decades later I knewhim again as a fellow member ofthe Edinburgh professoriate. The Dragonrepaid its keep in a modest way. Occasionally one ofthe candidate breaksflagged for the cryptanalyst's attention proved useful. But particularly in thecommon case ofa stagger of 1, the cryptanalyst's eye spotted breaks throughentirely different and scarcely mechanizable means. Endless practice hadequipped us with relatively huge mental dictionaries of cribs, the latter in theform ofsubliminallycoded character- string cliches. Take the above case ofGESELLSCHAFT, and generalize it to form the class ofall words ending"SCHAFT". While scanning the textof a stagger- 1 depth, the cryptanalyst' s eyewould be instantly caught if "JGQCX" occurred, and continued, say, so as toform JGQCXJQW+. Addition to itself at a stagger of one goes:

SCHAFT ..SC HAFT

.J GQCX .By instinct, as it were, one would immediately feel like appending a space tothe upper ofthe two plain-text lines,

SCHAFT 9. SCHAFT

Page 37: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

But the experienced Testery cryptanalyst wouldn't even try the new charactersum 9 + T, get the answer H, and note that it doesn't fit into JGQCXJQW+. Hewould have already recognized another familiar sequence, namely QW+, whichwouldbe trigger in his brain the sequence "shift up, N, shift down, space", or+NB9, which is teleprintese for "comma". So . . .

Good! Now for some backwards extension, thecryptanalyst would say. Howabout 9WISSENSCHAFT? . . . 9GESELLSCHAFT? ... and so forth. If none ofone's stock cliches leapt to mind either at plain-text or delta level, then onemight ask the sole German-speakingcoworker in the Hut, Peter Ericsson. Hislinguistic gifts, matched with an even more fluent associative imaginationwouldkick in. If his first verbal suggestion didn't hit the mark, he'd be over toyour deskwithpencil and eraser. In minutes he wouldhave extended your smallbreak in both directions in a lightning succession oftrials, retractions andconsolidations.

The mental style ofanother notable young colleague, Peter Benenson, was analmost unnerving contrast. Sheerconcentrated doggedness was appliedsystematically, obsessively, hour after hour. That too, I noted, could garnermiracles, less swiftlybut perhaps more surely, sometimes reducing jobsthatothers had tossed onto the heap as intractable. After the war he went on tofound Amnesty, culminating in a colourful exit with psychiatric overtones fromthat admirable organization. I did not follow the details.

It was essentially by such generate-and-test cycles ofconjecture that BrigadierTiltmanhad got out the very first depth, intercepted on August 30, 1941 . In hiscase, the displacement, or "stagger", was three characters rather than one. Thisis more adverse in terms of guessing candidate extensions, but partlycompensated by a gain in the length of each individual successful extension.

Peter Ericsson and I shared digs for the last two years. He opened the world ofvisual arts to me, including everything, such as it is, that I know about film-making. He was also spellbindingon both thecomical and the crazy aspects ofurban cultures ofmany lands. After the war he and Lindsay Anderson, destined

SCHAFT + N 8 9 .. SCHAFT + N 8 9

. J G QCXJQW+ .

Page 38: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

to become one ofBritain's innovative film directors (remember "1f ..."?),launched an avant-garde film magazine called Sequence. They worked on thefloor of a virtually unfurnished London apartment. Occasionally I visited fromOxford and helped where I could.

Turing's sequential Bayesrule

A major statistical contribution which Turing had earlier developed for theEnigma workwas of such generality that it was quickly taken up as a staplesupport for certain critical operations of our own work on Fish. After the war"Turing's sequential Bayes rule" became the foundation stone of an entire newbranch of applied statistics. It was expounded with further extensions byI.J.Good in his 1950 book Probabilityand the Weighing ofEvidence. WhenJack Good joinedme shortly after my recruitment by Newman to found the newmachine section, he brought with him everything ofEnigma methods whichwere generalizable. Good came straight from close collaboration with Turing inC.H. O'D Alexander's Naval Enigma section, ofwhich he had been the originalnucleus and first Head.

A digression here will illustrate certain unique qualities ofTuring himselfandofthe organization in which history had embedded him.

Turing style and BP style -both unusual

TheNaval section had originallybeen founded to support Turing's greatEnigma breakthrough. Hence it was natural that he be asked to head it.Unfortunately his uncanny intellectual gifts were tightly interwoven with an atleast equally uncanny lack ofwhat are ordinarily called "social skills". Thepredictable result was administrative chaos. Rapid ad hoc extemporizationscame to the rescue from one ofhis brightest lieutenants, the one-time BritishChess Champion Hugh Alexander. He had made his pre-war living (there wasin those days no money in chess) as an experienced and fast-thinkingmanagerofthe John Lewis London department store. So a little job like quietlyandtactfullyreorganizing Turing's bewildered section was to him an interestingchallenge. In short order Alexander flowed into the defacto headship. Turingcontinued happily as dejure Head, no longer distracted by these matters.

One day Turing arrived at the gate ofthe Park late for work. On such occasionsone signed oneself into a book withruled columns and headings that included"Name of Head of Section". Turing unexpectedly wrote "Mr Alexander", and

Page 39: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

proceeded in to work. Nothing was said. But somewhere wheels turned silently.Records were updated. Alexander continued his miracles of inspired and oftenunorthodox deployment of human and material resources, but now as theofficial Head ofNaval section.

I had this from a third party, and never asked Turing about it. I think he wouldhave found my question uninteresting.

Post-war development

Jack Good was AlanTuring's close friend and colleague, and also mine. I haveadded to this article a short informal note on their Bayesian innovations inapplied statistics. Post-war secrecy at the time ofGood's publication, and formany years thereafter, constrained those who had worked at BP to eliminatefrom their writings any clue, however miniscule and indirect, that might lead topublic discovery ofthe very existence ofany such organization. Hence the trueauthorship ofthe new statistical methodology and all other conceivablytraceable links to its origin had to be concealed.

As far as I am aware, the present article constitutes the first disclosure on publicrecord that the originator ofwhat is today called sequential analysis (followingAbraham Wald's coinage ofthis term in his post-warpublication of a bookunderthat name) was in fact A.M. Turing. His development of a Bayes-derivedweights-of-evidence approach to rational belief-revision was more general andmore far-reaching than Wald's non-Bayes contribution.

Asked in the 1975-76 taped interview to name Turing's greatest intellectualcontribution during the war, Newman had this to say:

NEWMAN: The main ... the real contribution was thestatistical theory ofTuring which he didn't invent for us[the Newmanry] but for another problem which he wasconcerned with [Enigma] which afterwards became avery important advance in the statistical method.

EVANS: Could you say something about that?

NEWMAN: Well, it's not my field but it's ... somethingcalled sequential analysis.

Page 40: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Turing used the dual formulation ofprobabilityofthe Bayesian school, whichgives two distinct interpretations according as we wish to speak oftheprobability, under some hypothesis (e.g. that "this shuffled pack contains onlyone ofthered suits") ofthe outcome of an observation (e.g. "I drew a club"), orofthe probability ofthe hypothesis itself.

The first is characterized as a limiting frequency (of an event E) and is oftencalled "objective probability".

The second is characterized as a degree of rational belief (in a hypothesis H)and is often called "subjectiveprobability"

The wartime problem to be solved was ofthe following type:

A hypothesis initially judged to be true with probability P(H) is subjected tosequential repetitions of a test with a view to deciding either to accept or toreject it. For example we may wish to decide whether the above-statedhypothesis is true of a given shuffled pack.

We proceed to withdraw cards one at a time, note the colour, and thenreplace itat a randomly chosen place in the pack. We don't want to commit ourselves toaccepting H if it is going to turn out false. Equally we don't want to reject H ifit is going to turn out to be true. Each draw of a card costs us x dollars, so wealso don't want to go on sampling for too long.

Before Turing, all statistical methods for calculating risks of error were basedon fixed sample-size statistics. But can a stopping rule be devised that controlsthe risks of the two kinds of error to exactly the same extent as the best fixedsample-size procedure and at the cost of substantiallyfewer repetitions?

In practice, with Turing's sequential Bayes rule, as with Wald's subsequent non-Bayes sequential analysis, "substantially fewer" not uncommonly approachessavings of almost 50 per cent. Further, by transforming probabilities into logodds measure, where odds = P/(l - P), Turing achieved a massive simplificationwhich gave the whole system additive properties analogous to Shannon'sintroduction of the logarithmically derived "bit" as the cornerstone ofinformation theory. I have already mentioned Turing's additive unit, the "ban".His method has the advantage of making the identification with "weights ofevidence " in the ordinary (not explicitly defined) sense evident immediately (toparaphrase Church's 1937 comment on Turing in another context.

Page 41: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Turing went further. By introducing expected utilities as a refinement of thestopping criterion, he anticipated statistical decision theory. In using theexpected weight ofevidence as a "defaultutility"he out-ran even modern texts,in offering a simple solution to the "exploration versus exploitation" trade-off.

Using Good's term "plausibility" for the log odds measure of degrees ofrational belief, a final summary of some essentials may be expressed as below(H stands for "hypothesis" and E stands for event, "prior" means before theobserved event-outcome and "posterior" means after. The vertical bar means"given"):

Posterior-plausibility(H) after k observations = prior-plausibility(H) plus thecumulative weight of evidence derivable from the first k outcomes.

where the functional expression w (Ej) is defined as

log (tP(Ei|H)] / tP(Ei|not-H)]),

i.e. the logarithm ofthe probability ofthe ith observed outcome given that Histrue, divided by its probability given H false.

Note that unless successive events are independent, strictly each P(Ej|H) has tobe calculated in its sequential context, i.e. as P(Ej| Eb E2 , ... Ei and H) andlikewise for the not-H case. Here the comma can be read as "then". Methodswhich disregard this qualification are today known as "naive Bayes". Note thatthe numbering of the events corresponds to the order in which their outcomesare observed. Normally, but not necessarily, this correspond to the order inwhich they happen.

Outlinerecapitulation

Plausibility(H) is definedas logodds(H) where [odds = p/(l-p)]

H is a proposition expressing a hypothesis(e.g. that a given penny is double-headed)

Ek is a the outcome ofan observation ofthekth event

Prior-plausibility(H)+ w (EO + w (E2 ) + . . . w (Ek) = posterior-plausibility(H)

Page 42: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

(e.g. "it's heads" or "it's tails")

Weights w(E) are the incremental changes (positiveor negative) to H's plausibilityfrom observingtheEj's, using

The expression inround brackets is called the "log likelihood ratio".

Secure conduct: enforcement or interlock?

The Germanoperators were, not surprisingly, forbidden to retransmit a messagewithout changing the wheel settings. Someone doubtless had figured out thedanger of generating a depth through disregarding this precaution. Ananalogous peace-time interdict is supposed to deter drivers from taking vehiclesonto theroad without fastening seatbelts. Reflection suggests a safer expedient.Road vehicles could by law incorporate interlocks in their manufacture,enablingthe ignition to fire when and only when the driver's seat-belt isfastened.

As weknow, that is not the way that the minds ofcivil transport authoritieswork. Nor was it the way ofwar-time Germany's military authorities with theLorenz machine. Rather thancommission a re-design, they banned undesiredoperator behaviours. The curative effect proved patchy at best. By degrees thestable door was in the end closed. By then the horse had bolted. IfGermancryptanalysts had been given opportunity to vet the Lorenz for design flaws,then the British project to crack it could never have got off the ground.In Britain the code-makers were the Royal Corps ofSignals. The code-breakerswere the mixed-services signals intelligence organization built up at BP duringWorld War 2. Similarcompartmentalization prevailed, so some ofmycolleagues ascertained, both in America and in Russia. Interlock, filtering, etc.precautions would ofcourse at once occur to a codebreaker. In like manner, theonly dependable way ofprotecting corporate and governmental computernetworks today from thecriminal trespasses ofhackers is to hire from their topechelon on theprinciple of fighting fire with fire.

Generating depths was but the first and most extreme ofmany humanfoiblesthat introduced unwanted regularities into messages. The habit, for example, ofhitting shift-up and shift-downkeys twice in quick succession, justto make

w (EO = log ([P(Ei|H is true)] / [P(Ei|H is false)])

Page 43: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

sure, could have been rendered harmless electromechanically. A simple logicalfilter placed between the stream of characters from the keyboard and thewireless transmitter, could have done the trick. Examples could be multiplied.

At all events, one thingis sure. The day that the world's nations break thehermetic seal between code-makers and code-breakers will see the end ofthemilitary cryptology game as we know it.

At the end of the day

A common enterprise ofcomrades: many people ofboth sexes and ofeveryrank and degree have spoken or written of their WW-2 experiences, and oftheexhilaration of "Each for all and all for each". At BP, friendship and mutualenjoyment continued in the more than halfof each twenty-four hour cycle thatwas passed outside the wire, at dances with Wrens or ATS, sing-songs in thetransport coaches, group expeditions to cinemas and pubs, daytime discussion-walks in the countryside on coming off a night shift.

Far from being shouldered aside by the urgency ofwork-time preoccupations,in wartime Britain every kind ofcultural interest, educational activity andentertainment blossomed. 2 1st century leisure notions of dumbed downamusements, ofmindless hanging out, ganging up or freaking out, would haveseemed like bad news from anotherplanet. Work-place politics, turf wars andpetty spites lay in the future. Later, often much later, people ofmy generationcame upon them for the first time and made belated adjustment. I am not alonein the impression that our new world seems sometimes locked in joylesspursuitofthe transient, or ofthe downright unattainable. We and our juniorselboweach other under ever more unpredictable competitive conditions.Yet memory tells us thattoday's gathering ills do not necessarily springpredestinate from unalterable flaws. There are other modes of living andworking together. We know. We were there.

References, sources and further readings

Carter, Frank (1996) Codebreaking with the Colossus Computer,Report No. 1, TheBletchleyPark TrustReports\ MiltonKeynes: BP Trust.

Page 44: COLOSSUS ANDTHE - Stanford Universityhz950cv9440/hz950... · 2015-10-24 · Colossus 10wasundercompletionattheendofhostilities,bywhichtimemore than63millioncharactersofhighgradeGermanmessageshadbeendecrypted

Carter, Frank (1997) Codebreakingwith the Colossus Computer: Finding theK-Wheel Patterns by Frank Carter, Report No. 4 June 1997, in The BletchleyPark TrustReports, MiltonKeynes: BP Trust.

Copeland, B.J. (inpreparation) <Book title etc. to be supplied>

Currie, Helen (undated) An ATSgirl 's memories ofBletchleyPark at war - andafter. Apply to Tony Sale, c/o BP Trust, The Mansion, BletchleyPark,Bletchley, MiltonKeynes MK3 6EF.

Evans, Christopher, (undated: internal evidence indicates approximately 1976).Pioneers ofComputing number 15. Interview in an oral history ofcomputingcompiled with the support ofthe Science Museum in London and the NationalPhysical Laboratory at Teddington, UK.

Fensom, Harry (Nov. 1999)Mathematics of Codebreaking (Cryptanalysis) withColossus or What did Colossus Really Do?Apply to Mr Tim Burslem, Editor, RSS Newsletter, Honeysuckle Cottage,School Road, Waldringfield, Woodbridge IP 12 4QR.

Good, I.J. (1970) Some future social repercussions of computers. Intern. J.Environmental Studies, 1,67-79.

Good, I.J. (1994) Enigma and Fish, Chapter 19 in Codebreakers: the insidestory ofBletchley Park (eds. F.H. Hinsley and Alan Stripp), OUP paperback.

Good, I.J. (1980) Pioneering work in computers at Bletchley, Chapter 4 ofAHistory ofComputing in the Twentieth Century (eds. N. Metropolis, J. Howlettand Gian-Carlo Rota), Academic Press.

Randell, Brian (1980) The COLOSSUS, Chapter 5 in A History ofComputingin the Twentieth Century (eds. N. Metropolis, J. Howlett and Gian-CarloRota),Academic Press, 1980.

Sale, Tony (undated typescript),Lorenz and Colossus. Apply to Tony Sale, c/oBP Trust, The Mansion, BletchleyPark, Bletchley, MiltonKeynes MK3 6EF.

Tutte, William (undated), FISH and I. Apply to Professor William Tutte, 15Manderston Road, Newmarket Suffolk CBB ONS.