Synthetic Harmonies- An Approach to Musical Semiosis by Means of Cellular Automata
Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic...
Transcript of Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic...
![Page 1: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/1.jpg)
Automatic Construction of Synthetic Musical Instruments and Performers
Ning Hu
Ph.D. Thesis Proposal
Carnegie Mellon UniversityDecember 2004
![Page 2: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/2.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Thesis Committee• Roger B. Dannenberg, Chair• Michael S. Lewicki• Richard M. Stern• David Wessel (UC Berkeley)
![Page 3: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/3.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
![Page 4: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/4.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Introduction• Define thesis topic• Thesis statement• Start with modeling the trumpet for
classical music• Contributions of the thesis• Criteria for success
![Page 5: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/5.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis
![Page 6: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/6.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis• Automatic construction
![Page 7: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/7.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis• Automatic construction• Instrument model
![Page 8: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/8.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis• Automatic construction• Instrument model• Performance model
![Page 9: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/9.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis
• High quality• High performance• Capable of modeling different instruments • Capable of modeling different music styles
![Page 10: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/10.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Automatic construction
• Use machine learningmachine learning techniques• Learn from performance examples• Constructs instrument model and
performance model
![Page 11: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/11.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Instrument model
• Similar to traditional concept of synthesis• Input: control signals• Output: synthesized sound samples
![Page 12: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/12.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Performance model
• Generating appropriate control signals from music context is crucial
• Drives instrument model• Input: digital score (music notation)• Output: control signals
![Page 13: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/13.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Topic Definition• Title
Automatic Construction of Synthetic Musical Instruments and Performers
• Meaning• Framework that builds music synthesis• Automatic construction• Instrument model• Performance model
![Page 14: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/14.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Thesis Statement• To create a system framework that
can automatically create high-quality musical instrument synthesis by using machine-learning techniques to construct the instrument model and the performance model by learning from the performance examples (acoustic recordings and their corresponding scores).
![Page 15: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/15.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Start with Modeling…• Musical Instrument:
• Trumpet• Music Style:
• Classical Music
![Page 16: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/16.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Reasons for Modeling the Trumpet (1)• Most wind instrument synthesizers
do not sound realistic• Conflict between:
• Working mechanisms of wind instruments• Driven by continuous energy exerted by player• Continuous control drives sound production
• Basic structure of synthesizers• Mostly sampling-based• Based on single, isolated notes• Do not offer a wide range of control
![Page 17: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/17.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Reasons for Modeling the Trumpet (2)• Previous research
• By Dannenberg and Derenyi (1998)• Similar scheme• Produces convincing trumpet sound
![Page 18: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/18.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Reasons for Modeling Classical Music• Characteristics of classical music
• “Purer” playing style• More faithful to the score• Fewer articulation effects
• Characteristics of non-classical music• Significant inharmonic & transient sounds• Need to model noise with a residual model
![Page 19: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/19.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Contributions of the thesis• Use machine learning techniques • Automatically create high-quality
synthesis• The problem of control
• Problems of note-oriented synthesis• Problems of physical models• This approach simplifies the problem
![Page 20: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/20.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Criteria for success• Minimum requirement
• Design, implement & test basic framework• Being able to synthesize realistic trumpet
performance for classical music• Automated modeling process• Tested on one or two wind instruments.
• Extra tasks• Extend system framework
• Model different musical instruments• Model different music styles
• Future work
![Page 21: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/21.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
![Page 22: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/22.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
System Structure• Synthesis process• Pre-processing training data• Training process for instrument model• Training process for performance model
![Page 23: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/23.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Synthesis Process
Performance Model
Instrument Model
Control Signals
![Page 24: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/24.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Pre-processing training data
AutomaticAlignment &
Segmentation
Segmented Score
Segmented Audio
![Page 25: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/25.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Training the Performance ModelSegmented
Score
Control Signals
Segmented Audio
Control Signals
Error
Compare
Parameter Extraction
Performance Model
Correspond
![Page 26: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/26.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Training the Instrument Model
Spectra
Segmented Score
Control Signals
Segmented Audio
Parameter Extraction
Spectral Analysis
Correspond
Supervised Learning
Instrument Model
![Page 27: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/27.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
![Page 28: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/28.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Audio Alignment• Find recording score• Polyphonic audio alignment
• Extract feature sequences from …• Acoustic recording• Score
• Find optimal alignment• Dynamic programming (DP) or Hidden
Markov Model (HMM)• Satisfactory results
Correspondence
![Page 29: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/29.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Audio Segmentation (1)• Dannenberg, et. al., (1999) early work
• Define rules & thresholds• Use features: power; #peaks/period; #zero-
crossings/period• Not reliable and accurate enough
• Precise alignment = reliable segmentation• Require higher accuracy• Need further modification
![Page 30: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/30.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Audio Segmentation (2)•Kapanci & Pfeffer’s (2004) work
•Segmentation problem Classification problem•Hierarchical machine-learning framework•Detect soft onset: compare frames separated by increasingly longer distances
Classification Modules
![Page 31: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/31.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
√√
![Page 32: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/32.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Instrument Model
•Harmonic Model + Residual model
Instrument Model
Control Signal
![Page 33: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/33.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Harmonic Model
• Control signals Spectra• Spectra Wavetables• Wavetables Sound samples
Interpo-lation
Spectrum Generation
Waveform Construction
Table 2
Table 1
Phase
×
×
×+Amplitude Control
Frequency Control
![Page 34: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/34.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Control Signals Spectra• Spectral interpolation (Dannenberg, et. al., 1998)
• Memory-based approach (Wessel, et. al., 1998)
• Neural network (Wessel, et. al., 1998)
![Page 35: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/35.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Spectral interpolation (Dannenberg, et. al., 1998)
• Generate Spectral lookup table• Record a set of sounds• Obtain a spectrogram for each sound• Retain specific spectra at thresholds
Amplitude
Frequency SyntheticSpectra
2D Interpolated Spectral Lookup Actual
Spectra
![Page 36: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/36.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Memory-based approach (Wessel, et. al., 1998)
• Index spectra in a n dim. space• Interpolate among k nearest
neighbors• Special case
• Spectral interpolation technique• n=2 (frequency & amplitude)• k=4• Linear interpolation
![Page 37: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/37.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Neural network (Wessel, et. al., 1998)•A feed-forward neural network with multiples layers•Input: frequency, amplitude•Output: info of sinusoidal components•Back-propagation learning method•Advantage: very compact & generalize well
![Page 38: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/38.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Interpo-lation
Spectrum Generation
Waveform Construction
Table 2
Table 1
Phase
×
×
×+Amplitude Control
Frequency Control
Harmonic Model
•Control signals Spectra•Spectra Wavetables•Wavetables Sound samples
![Page 39: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/39.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Residual Model• Modeling attacks
• Use recorded attacks• Phase matching
• For other instruments
![Page 40: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/40.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
√√√√
![Page 41: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/41.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Performance Model
•Control signals•Amplitude envelope•Frequency envelope
•Error metrics•Envelope representation•Mapping scheme
Performance Model
Control Signal
![Page 42: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/42.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Amplitude Envelope
•A tongued note
•A slurred note
*Figures borrowed from (Dannenberg, et. al., 1998)
![Page 43: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/43.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Error Metrics
• Typical metric: RMS error• Recent work by (Horner et. al., 2004)
• Measure perceptual difference• Useful reference for error metrics
Acoustic and Synthetic Envelopes
![Page 44: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/44.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Envelope Representation•Envelope representation
•Characteristics of amplitude envelope•Specific duration•Specific shape•Specific properties of each part
![Page 45: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/45.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Ways to Represent Envelopes• Collection of general parameters
• Candidates: center of mass, global/local maximum/minimum, etc.
• Manual vs. automatic selection• Wavelets
• Hierarchical decomposing functions• Very powerful and popular
![Page 46: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/46.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Mapping Scheme• Non-linear regression
• Find music context actual envelopes• Examples:
• Neural network• Kalman filters Xk = AXk-1 + Wk-1
• Function approximation
• Pattern clustering• Classify envelopes into clusters• For each input data point:
• Use corresponding representative envelope• Stretch, scale & interpolate accordingly
• Considered as a form of case-based reasoning
![Page 47: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/47.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
√√√√√√
![Page 48: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/48.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Schedule (1)
Dec 2004 ~ May 2005 System Development
System integration and testingMay 2005
Design and implement performance modelMar 2005 ~ Apr 2005
Develop and compare instrument modelsFeb 2005
Incorporate SNDAN package to training data pre-processing stage
Jan 2005
Implement Audio Alignment & Segmentation module
Dec 2004 ~ Jan 2005
•Collect performance examples for initial experiments•Get familiar with SNDAN package•Read the original code by Dannenberg & Derenyi•Propose thesis topic
Oct 2004 ~ Nov 2004 (past) Thesis Proposal
![Page 49: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/49.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Schedule (2)
Sep ~ Nov 2005 Writing Thesis
Jun ~ Aug 2005 System Evaluation & Tuning
Thesis defenseNov 2005
Thesis write-up and revisionsSep 2005 ~ Nov 2005
System and model evaluation and fine tuningAug 2005Synthesize other types of music Jul 2005Model other instrumentsJun 2005
![Page 50: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/50.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Conclusion• Propose a scheme:
• automatically construct• high-quality instrument synthesis• by learning from performance examples
• Machine learning a crucial role. • Future work
• Modeling different instruments• Modeling different music styles• Make it work in real-time
![Page 51: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/51.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Roadmap• Introduction• System Structure• Main Modules
• Audio Alignment and Segmentation• Instrument Model• Performance Model
• Schedule• Conclusion
√√√√√√
![Page 52: Automatic Construction of Synthetic Musical …rbd/music/ninghu/thesis/ProposalSlides.pdfAutomatic Construction of Synthetic Musical Instruments and Performers NingHu Ph.D. Thesis](https://reader034.fdocuments.net/reader034/viewer/2022051802/5af5661f7f8b9a154c8fa1b9/html5/thumbnails/52.jpg)
December 2004 Ning Hu’s Ph.D. Thesis Proposal
Thank You• Acknowledgements
• Roger B. Dannenberg• Istvan Derenyi• James Beauchamp (SNDAN)
• ContactNing Hu ([email protected])