Conductor Heroweb.mit.edu/6.111/www/f2007/projects/ncheung... · • Conductor - User Interface -...

Conductor HeroNatalie Cheung, Yuta Kuboyama, Edgar Twigg

Introduction

Goal: Conduct an Orchestra based on Hand Movements

•Score transferred through a USB Port•User conducts the entire orchestra and smaller sections•Output orchestra music and graphical representation of the

orchestra on the screen

Overview

USBInput Score Data

Score Manager

Conductor

Mixer

enb enbpitch pitchpitch

dynamicsarticulationsection_sel

video_out

USB245M

note_data

beat_numbusynote_data(no beat)

video_input

enb

tempo

beat

stereo_left,stereo_right stereo_left, stereo_right

stereo_left, stereo_right

audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

• Score Management- Music data handling- Takes numeric user signals and score data to determine what instruments play when

USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

• Conductor- User Interface- Generates numeric signals from user movement


USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

• Conductor- User Interface- Generates numeric signals from user movement


• Sound Synthesis- Given on/off and pitch- Generates actual sound

Time

24

ticks

beatbeatbeat

• 128 ticks per beat• At 50 bpm a tick lasts for 9.4ms

- humans can detect differences of ~20-30ms• Uneven note durations (e.g. triplets) won’t sound choppy

NoteData Format

Start Beat Start Tick Instrument Pitch Enable

0-11 (0-4095) 12-18 (0-127) 19-25 (0-127) 26-32 (0-127) 33 (0-1)

• Each NoteData takes up one 36 bit wide SRAM location• Two NoteData events per note (on and off) = 72 bits per note• 500 Measures in 4/4, average of 15 notes per beat = 2.06Mb

Score Management

note_data note_data

note_data

USBInput

ZBT Score Data

address

19

34 34

address

19

USB 245M

reprogram

22 12

Score Manager

reset

reset

reset

pitch 7 enable

beat

tempo

10

Tick Counter

pitch 7 enable pitch 7 enable

reset7

beat_num

current_tick

note_data note_data

note_data

USBInput

ZBT Score Data

address

19

34 34

address

19

USB 245M

reprogram

22 12

Score Manager

reset

reset

reset

pitch 7 enable

beat

tempo

10

Tick Counter


reset7

beat_num

current_tick

puts USB data into

ZBT SRAM

note_data note_data

note_data

USBInput

ZBT Score Data

address

19

34 34

address

19

USB 245M

reprogram

22 12

Score Manager

reset

reset

reset

pitch 7 enable

beat

tempo

10

Tick Counter


reset7

beat_num

current_tick

puts USB data into

ZBT SRAM

When beat_num > current beatbusy = highoutput new note_data

note_data note_data

note_data

USBInput

ZBT Score Data

address

19

34 34

address

19

USB 245M

reprogram

22 12

Score Manager

reset

reset

reset

pitch 7 enable

beat

tempo

10

Tick Counter


reset7

beat_num

current_tick

puts USB data into

ZBT SRAM


If beat switches stateincrement beat_numwhile Score Data is busy

overwrite BRAM w/ new note_dataall data in BRAM now has same beat

If current_tick == tick in BRAMdo the note (turn on/off, set pitch, etc)

note_data note_data

note_data

USBInput

ZBT Score Data

address

19

34 34

address

19

USB 245M

reprogram

22 12

Score Manager

reset

reset

reset

pitch 7 enable

beat

tempo

10

Tick Counter


reset7

beat_num

current_tick

puts USB data into

ZBT SRAM


If beat switches stateincrement beat_numwhile Score Data is busy

overwrite BRAM w/ new note_dataall data in BRAM now has same beat

If current_tick == tick in BRAMdo the note (turn on/off, set pitch, etc)

counts clock cyclescompares to #cyc/tickif a tick has passed

increment tick counter

USB Input (Data Reading Timings)

- excerpt from FTDI UM245R USB-Parallel

FIFO Development Module Datasheet

Tick Counter

Tick Counter

reset

7tempo

10

• Counts the # of clock cycles that have elapsed since the last tick• Calculates the # of clock cycles that should elapse before the next

tick based on the tempo• Increments the current tick when count = calculated

tempo4

beatsmin

_128 tickmin

_ 160minsec

= tempo_ 815ticksec

1tempo

_158sectick

_27,000,000 clksec

= 50,625,000tempo

clktick

Conductor

Conductor Module

Video Decoder• Detect LED using thresholds• Reject other colors• Store data into ZBT

Conductor Module

ZBT• Each frame

- ZBT1 collects information from video_data- Conductor module reads from ZBT2

• Every frame- ZBT1 <= ZBT2- ZBT2 <= ZBT1

Conductor Module

Conductor• Determine if there are 2 LEDs (conduct

the whole orchestra)• Determine position of LED• If position of LED remains the same for a

certain number of frames (based on a counter), the next time the LED moves to a different position, the beat enable is high

• Tempo is based on the velocity of the LED

Sound Synthesis

USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

•Represents a section of instruments (strings, brass,etc), and produces audio signals for the instruments at specified pitches

•Consists of instrument parameters, signal processor, sample look-up table

•Updates the dynamics and articulation when the section selector matches

•Outputs a stereo signal representing the location in the orchestra

•

Instrument Manager

USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

•Simply adds the left channel and right channel signals separately

•Produces a 24-bit stereo signal from adding 16-bit stereo signals

Mixer

USBInput Score Data

Score Manager

Conductor

Mixer



video_out

USB245M

note_data


video_input

enb

tempo

beat



audio_out

reprogram

22

12

10

7 77

8 83

2x16 2x16 2x16

2x24Compressor

signal_sum

reset

reset

reset

ZBTnote_data

address address1919

34 34

2x16

clk2 clk2

clk2

clk2

clk2clk2

clk1

ZBT ZBT

write_enbread_enb

VideoDecoder

video_data

Instrument Manager

reset

•Balances the output and prevents clipping or getting too quiet

•Applies gain < 1 for signals that would clip

•Applies gain > 1 for signals that are too quiet

•Sends a 16-bit stereo signal to the DAC, then to the speakers

Compressor

Memory and Resources


16 16 16 16 16 16 16

Sound sample lookup table


16 16 16 16 16 16 16


16 16 16 16 16 16 16



~4000 samples per instrument ~4000 samples per instrument

16 16 16 16 16 16 16


16 16 16 16 16 16 16




~15 instruments

16 16 16 16 16 16 16


16 16 16 16 16 16 16




~15 instruments

16 16 16 16 16 16 16


16 16 16 16 16 16 16


0.96Mbits out of 2.5Mbits on one FPGA

Finale

Sun Mon Tues Wed Thurs Fri Sat

Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15


Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Detect LED ColorWrite USB Input ModuleSample Audio & Playback


Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch



Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Thanksgiving Break




Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Thanksgiving Break



Write Score ManagerCreate Conductor Module for whole orchestra

Playback audio with Dynamics and Articulation Parameters


Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Thanksgiving Break

Integration






Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Thanksgiving Break

Integration

Finishing Touches






Nov

Today

Dec

Timeline

30

2

25

18

17161514131211

9

19

26

3

10

20

27

4

11

21

28

5

12

22

29

6

13

23 24

7

14

1

8

15

Thanksgiving Break

Integration

Finishing Touches


Add orchstra sections and graphicsWrite MIDI to Bitfile ConverterWrite Mixer & Compressor Modules




Thank you...

Conductor Heroweb.mit.edu/6.111/www/f2007/projects/ncheung... · • Conductor - User Interface -...

Documents

Transcript of Conductor Heroweb.mit.edu/6.111/www/f2007/projects/ncheung... · • Conductor - User Interface -...