Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2...

36
Boosted Decision Trees

Transcript of Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2...

Page 1: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosted Decision Trees

Page 2: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Previously on CSCI 4622

2

Ensemble Learning: Build a bunch of slightly different models. Predict by majority vote.

Bagging: Train a bunch of decision trees on bootstrapped resamples of training data.

Random Forests: Train a bunch of decision trees on bootstrapped resamples of training data. On each node, consider only a random subset of features to split on.

A single tree is very sensitive to training data. An average over many trees is less sensitive.

Bagging can sometimes create correlated trees, especially if there’s a handful of strong features

Random Forests helps to de-correlate trees by leaving strong features out in some splits

Page 3: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting Intuition

3

Boosting is another ensemble model, but with a slight twist

“How is it that a committee of blockheads can somehow arrive at highly reasoned decisions, despite the weak judgment of the individual members? How can the shaky separate views of a

panel of dolts be combined into a single opinion that is very likely to be correct?

Schapire and Freund, Boosting: Foundations and Algorithms”

"

I

Page 4: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting Intuition

4

Boosting is another ensemble model, but with a slight twist

“How is it that a committee of blockheads can somehow arrive at highly reasoned decisions, despite the weak judgment of the individual members? How can the shaky separate views of a

panel of dolts be combined into a single opinion that is very likely to be correct?

Schapire and Freund, Boosting: Foundations and Algorithms”

Idea: Build a sequence of dumb models

Modify the training data along the way to focus on difficult to classify examples

Predict based on weighted majority vote of all dumb models

o What do we mean by dumb? o How do we promote difficult examples? o Which models get more say in the vote?

Page 5: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Dumb Models

5

What do we mean by dumb?

Each model in our sequence will be a weak learner

A model is a weak learner if its training error is just under 50%

Most common weak learner is a Decision Stump – a Decision Tree with a single split

Other common weak learners include slightly deeper Decision Trees and perceptrons

h(x)<latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit>

err =1

m

mX

i=1

I(yi 6= h(xi)) =1

2� �

<latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit>

Page 6: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Promoting Difficult Examples

6

How do we promote difficult examples?

After each iteration:

o Increase importance of training examples we got wrong on previous iteration

o Decrease importance of training examples we got right on previous iteration

Each training example will carry around a weight

Weights are nonnegative normalized so that they behave like a probability distribution

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

mX

i=1

wi = 1, wi � 0<latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit>

{ C Ii, yD3I,

=

I = 1 , ... ,m

=

Page 7: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Weighted Majority Vote

7

Which models get more say in the weighted majority vote?

The models that performed better on the training data get more say in the vote

For our sequence of weak learners:

Boosted classifier defined by

The weight controls vote contribution of . Think of as measure of accuracy of

h1(x), h2(x), . . . , hK(x)<latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit>

H(x) = sign

"KX

k=1

↵khk(x)

#

<latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

↵k<latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit>

=

Page 8: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

The Plan

8

o Look at example of popular Boosting method

o Unpack it for intuition

o Come back later and show the math

Usual binary classification assumptions:

o Training set

o Feature vectors

o Labels are

{(xi, yi)}mi=1<latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit>

xi 2 Rp<latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit>

yi 2 {+1,�1}<latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit>

Page 9: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

9

1. Initialize training example weights to uniform distribution

2. For

3. Output H(x) = signhPK

k=1 ↵khk(x)i

<latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit>

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

k = 1, 2, . . . ,K :<latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit>

a) Fit classifier to training data with weights

b) Compute weighted error

c) Compute vote weight

d) Update training example weights

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

errk =Pm

i=1 wi·I(yi 6=hk(xi))Pmi=1 wi

<latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit>

↵k = 12 log((1� errk)/errk)

<latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit>

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

wi 1Zk

· wi exp [�↵kyihk(xi)]<latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit>

7

=( ADDING

MORE WEAK LEARNERS )

Page 10: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

10

1. Initialize training example weights to uniform distribution

Weights are initialized to uniform dist. At beginning, all training examples treated the same

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

Page 11: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

11

1. Initialize training example weights to uniform distribution

2a) Fit classifier to training data with weights

Decide Decision Stump split based on Impurity and Information gain as usual

Before: p was fraction of examples with positive label

Now: p is fraction of weights in pos class

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

I(D) = �p log2 p� (1� p) log2(1� p)<latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit>

p =

Pmi=1 wi · I(yi = +1)Pm

i=1 wi<latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit>

p =

Pmi=1 I(yi = +1)

m<latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit>

_

=

← pos pts=

0=←

Page 12: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

12

2b) Compute weighted errors:

o Still gives an error rate in [0, 1]

o Mistakes on highly weighted examples hurt more

o Mistakes on lowly weighted examples don’t hurt as much

errk =

Pmi=1 wi · I(yi 6= hk(xi))Pm

i=1 wi<latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit>

# gotis # wrong

=

Page 13: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

13

2c) Compute vote weight for the current weak learner

o Models with small weighted error get a larger vote

o Models with large weighted error get less vote (or, sometimes a negative vote!)

↵k = 12 log((1� errk)/errk)

<latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit>

= -

×

×

× ja= - 0,5X

×

X 1

Page 14: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

14

2d) Update training example weights

o If example is misclassified then weight goes up

o If example is classified correctly then weight goes down

o How big of a jump up/down depends on how much vote the current learner has

o is just a normalizing constant to make it a probability distribution

o Don’t really have to compute , just scale weights so they sum to 1

wi 1Zk

· wi exp [�↵kyihk(xi)]<latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit>

Zk<latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit>

Zk<latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit>

PREPlot .

L

- =a

- =

uol.net#Yane1strung #

Page 15: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Algorithm

15

3) Algorithm produces the final boosted classifier

o Sum up weighted votes from each model

o Apply sign function so that it produces predictions in {+1,�1}<latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit>

H(x) = signhPK

k=1 ↵khk(x)i

<latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit>

-

Page 16: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

16

Example: Suppose you have the following training data

Page 17: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

17

o First Decision Stump:

0°0

Page 18: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

18

o First Decision Stump:

err1 = 0.30↵1 = 0.42

<latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit>

=

=

=

Page 19: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

19

o First Decision Stump:

err1 = 0.30↵1 = 0.42

<latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit>

Page 20: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

20

o Second Decision Stump:

=

=

=

-

:-

Page 21: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

21

o Second Decision Stump:

err2 = 0.21↵2 = 0.65

<latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit>

.

-

÷

Page 22: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

22

o Second Decision Stump:

err2 = 0.21↵2 = 0.65

<latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit>

Page 23: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

23

o Third Decision Stump:

Page 24: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Adaboost Example

24

o Third Decision Stump:

err3 = 0.14↵3 = 0.92

<latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit>

Page 25: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Example: Boosted Classifier

25

0.42 + 0.65 + 0.92

=

" "a 1 1+1

-

'- -1

--1 L

42C1) t .65( 1) f. 921 D

×x

.yzf , ) + .

651 - 1) + r{2( D

= 7 0= -1.07+0-92=

- g.15<0⇒ +

Page 26: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Performance

26

Recall the standard experiment of measuring train and validation error vs model complexity

Once overfitting begins, validation error goes up

x

Page 27: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Performance

27

Boosting has a remarkable affect. Think of number of weak learners as model complexity

Even after training error becomes zero, validation error continues to improve!

tie

Page 28: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Performance

28

HOW CAN THIS BE?!

Page 29: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting as Margin Maximizing Algo

29

Adaboost actually induces a margin, and tries to maximize it

Despite the fact that we make the model more complex with each weak learner, the model actually becomes more confident with each boosting iteration

This confidence can be represented as a margin

Our learn classifier with K weak learners is

Define an intermediate classifier by where

H(x) = signh PK

k=1 ↵khk(x)i

<latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit>

H`(x) = signh P`

k=1 ↵khk(x)i

<latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit>

` K<latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit>

=

-

Page 30: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting as Margin Maximizing Algo

30

Note that the classifier only returns +1 or -1, which doesn’t give us any measure of confidence

Define the normalized voting coefficients as

Define the margin of a training example after boosting iterations as

H`(x) = signh P`

k=1 ↵khk(x)i

<latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit>

↵̂k =↵kPKj=1 ↵j

<latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit>

`<latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit>

margin`(x) =P`

k=1: hk(x)=y ↵̂k �P`

k=1: hk(x) 6=y ↵̂k<latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit>

=

=

g- ==

=

Page 31: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting as Margin Maximizing Algo

31

Handwritten Digit Recognition with Adaboost

Note that Adaboost achieves zero training error after roughly 250 iterations

Page 32: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Boosting as Margin Maximizing Algo

32

Handwritten Digit Recognition with Adaboost. Two easy and two hard training examples.

margin`(x) =P`

k=1: hk(x)=y ↵̂k �P`

k=1: hk(x) 6=y ↵̂k<latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit>

ownnunnmrnmmn

×

Page 33: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

Practical Advantages of Boosting

33

o It’s fast!

o Simple and easy to program (you’ll find out on Monday cuz we’re gonna do it!)

o No parameters to really tune, except K

o Flexible, can choose any weak learner

o Shift in mindset: Now can look for weak classifiers instead of strong classifiers

o Can be used in lots of settings: text learning, images, discrete features, continuous features)

Page 34: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

34

Page 35: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

35

Page 36: Boosted Decision Treesketelsen/files/courses/csci4622/slides/lesson37.pdfPreviously on CSCI 4622 2 Ensemble Learning: Build a bunch of slightly different models.Predict by majority

36