Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative...
Transcript of Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative...
![Page 1: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/1.jpg)
stand B-7
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music GenerationHao-Wen Dong and Yi-Hsuan YangResearch Center of IT Innovation, Academia Sinica
Gz
noise
True/Fake
Synth Pad
x1 or x1
x2 or x2
xM or xM
Drums
Piano
…
~
~
~
D
Generator DiscriminatorDrums
Piano
Guitar
Bass
Ensemble
Reed
Synth Lead
Synth Pad
Target outputs - Multi-track piano-rolls Convolutional Generative Adversarial Networks
CNNs + GANsmulti-trackpolyphonic
![Page 2: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/2.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
Drums Piano Guitar Bass Strings
![Page 3: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/3.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
bass linechords
drum patterns
Drums Piano Guitar Bass Strings
![Page 4: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/4.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
Gz
noise
D True/Fake
Synth Pad
x1 or x1
x2 or x2
xM or xM
Drums
Piano
…
~
~
~
Generator Discriminator
real-valued
![Page 5: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/5.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
Generator Refiner Discriminator
Gz
noise
x1
x2
xM
…
R
R
R
1
2
M
…
ˆ
ˆ
ˆ
D True/Fake
Synth Pad
x1 or x1
x2 or x2
xM or xM
Drums
Piano
…~
~
~
real-valued binary-valued
![Page 6: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/6.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
Generator Refiner Discriminator
Gz
noise
x1
x2
xM
…
R
R
R
1
2
M
…
ˆ
ˆ
ˆ
D True/Fake
Synth Pad
x1 or x1
x2 or x2
xM or xM
Drums
Piano
…
~
~
~
real-valued binary-valued
Generator’s outputs Real data
MuseGAN real-valued binary-valued
BinaryMuseGAN binary-valued binary-valued
![Page 7: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/7.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
Generator Refiner Discriminator
Gz
noise
x1
x2
xM
…
R
R
R
1
2
M
…
ˆ
ˆ
ˆ
D True/Fake
Synth Pad
x1 or x1
x2 or x2
xM or xM
Drums
Piano
…
~
~
~
Deterministic Binary Neurons (DBNs)𝐷𝐷𝐷𝐷𝐷𝐷 𝑥𝑥 = 𝑢𝑢 𝜎𝜎 𝑥𝑥 − 0.5
Stochastic Binary Neurons (SBNs)𝑆𝑆𝐷𝐷𝐷𝐷 𝑥𝑥 = 𝑢𝑢 𝜎𝜎 𝑥𝑥 − 𝑣𝑣 , 𝑣𝑣~𝑈𝑈 0, 1
![Page 8: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/8.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
shared/privatedesign
shared/private design
Generator Refiner Discriminator
onset/offset stream
chroma stream
![Page 9: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/9.jpg)
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang stand B-7
MuseGAN’s output
(real-valued)
hardthresholding
Bernoullisampling
many overly-fragmented notes
![Page 10: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/10.jpg)
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang stand B-7
BinaryMuseGAN(+DBNs)MuseGAN’s output
(real-valued)
hardthresholding
Bernoullisampling
many overly-fragmented notes
less overly-fragmented notes
![Page 11: Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and](https://reader030.fdocuments.net/reader030/viewer/2022040816/5e5dda5a7507f4489707636d/html5/thumbnails/11.jpg)
stand B-7Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and Yi-Hsuan Yang
MuseGAN BinaryMuseGAN
Come to learn more and listen to the demos!
(different training strategies)