Maximum A Posteriori - MAP.pdfoFrequentist: play game infinitely many times and see how often...

Maximum A PosterioriChris Piech

CS109, Stanford University

Previously in CS109…

Estimators

Maximum Likelihood

Game of Estimators

Non spoiler alert: this didn’t happen in game of thrones

You observe n datapoints:

Maximum Likelihood Estimatorx(1), . . . , x(n)

<latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit>

X(1), . . . , X(n)<latexit sha1_base64="8ecriZO7NK5IzR4/GABdStbKcJw=">AAACAHicbVDLSgMxFL3js9bXqAsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkOSEcowG3/FjQtF3PoZ7vwb03YW2nogcHLOvTe5x485U9pxvq2V1bX1jc3CVnF7Z3dv3z44bKkokYQ2ScQj2fGxopwJ2tRMc9qJJcWhz2nbH99M/fYjlYpF4l5PYuqFeChYwAjWRurbx52HtOxWsirqDSKtqmh6F5Wsb5ecmjMDWiZuTkqQo9G3v8wAkoRUaMKxUl3XibWXYqkZ4TQr9hJFY0zGeEi7hgocUuWlswUydGaUAQoiaY7QaKb+7khxqNQk9E1liPVILXpT8T+vm+jgykuZiBNNBZk/FCQc6QhN00ADJinRfGIIJpKZvyIywhITbTIrmhDcxZWXSeu85jo19+6iVL/O4yjACZxCGVy4hDrcQgOaQCCDZ3iFN+vJerHerY956YqV9xzBH1ifPzGXlNY=</latexit><latexit sha1_base64="8ecriZO7NK5IzR4/GABdStbKcJw=">AAACAHicbVDLSgMxFL3js9bXqAsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkOSEcowG3/FjQtF3PoZ7vwb03YW2nogcHLOvTe5x485U9pxvq2V1bX1jc3CVnF7Z3dv3z44bKkokYQ2ScQj2fGxopwJ2tRMc9qJJcWhz2nbH99M/fYjlYpF4l5PYuqFeChYwAjWRurbx52HtOxWsirqDSKtqmh6F5Wsb5ecmjMDWiZuTkqQo9G3v8wAkoRUaMKxUl3XibWXYqkZ4TQr9hJFY0zGeEi7hgocUuWlswUydGaUAQoiaY7QaKb+7khxqNQk9E1liPVILXpT8T+vm+jgykuZiBNNBZk/FCQc6QhN00ADJinRfGIIJpKZvyIywhITbTIrmhDcxZWXSeu85jo19+6iVL/O4yjACZxCGVy4hDrcQgOaQCCDZ3iFN+vJerHerY956YqV9xzBH1ifPzGXlNY=</latexit><latexit sha1_base64="8ecriZO7NK5IzR4/GABdStbKcJw=">AAACAHicbVDLSgMxFL3js9bXqAsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkOSEcowG3/FjQtF3PoZ7vwb03YW2nogcHLOvTe5x485U9pxvq2V1bX1jc3CVnF7Z3dv3z44bKkokYQ2ScQj2fGxopwJ2tRMc9qJJcWhz2nbH99M/fYjlYpF4l5PYuqFeChYwAjWRurbx52HtOxWsirqDSKtqmh6F5Wsb5ecmjMDWiZuTkqQo9G3v8wAkoRUaMKxUl3XibWXYqkZ4TQr9hJFY0zGeEi7hgocUuWlswUydGaUAQoiaY7QaKb+7khxqNQk9E1liPVILXpT8T+vm+jgykuZiBNNBZk/FCQc6QhN00ADJinRfGIIJpKZvyIywhITbTIrmhDcxZWXSeu85jo19+6iVL/O4yjACZxCGVy4hDrcQgOaQCCDZ3iFN+vJerHerY956YqV9xzBH1ifPzGXlNY=</latexit><latexit sha1_base64="8ecriZO7NK5IzR4/GABdStbKcJw=">AAACAHicbVDLSgMxFL3js9bXqAsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkOSEcowG3/FjQtF3PoZ7vwb03YW2nogcHLOvTe5x485U9pxvq2V1bX1jc3CVnF7Z3dv3z44bKkokYQ2ScQj2fGxopwJ2tRMc9qJJcWhz2nbH99M/fYjlYpF4l5PYuqFeChYwAjWRurbx52HtOxWsirqDSKtqmh6F5Wsb5ecmjMDWiZuTkqQo9G3v8wAkoRUaMKxUl3XibWXYqkZ4TQr9hJFY0zGeEi7hgocUuWlswUydGaUAQoiaY7QaKb+7khxqNQk9E1liPVILXpT8T+vm+jgykuZiBNNBZk/FCQc6QhN00ADJinRfGIIJpKZvyIywhITbTIrmhDcxZWXSeu85jo19+6iVL/O4yjACZxCGVy4hDrcQgOaQCCDZ3iFN+vJerHerY956YqV9xzBH1ifPzGXlNY=</latexit>

X(i)<latexit sha1_base64="oebUgElUaJ17T+qKzaz7YakQLEQ=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoMQm3AnghYWARvLCOYDkjPsbfaSJXt7x+6cEI78CBsLRWz9PXb+GzfJFZr4YODx3gwz84JECoOu++0U1tY3NreK26Wd3b39g/LhUcvEqWa8yWIZ605ADZdC8SYKlLyTaE6jQPJ2ML6d+e0nro2I1QNOEu5HdKhEKBhFK7U7j1lVnE/75Ypbc+cgq8TLSQVyNPrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWqpoxI2fzc+dkjOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J47WdCJSlyxRaLwlQSjMnsdzIQmjOUE0so08LeStiIasrQJlSyIXjLL6+S1kXNc2ve/WWlfpPHUYQTOIUqeHAFdbiDBjSBwRie4RXenMR5cd6dj0VrwclnjuEPnM8fuSKPIg==</latexit><latexit sha1_base64="oebUgElUaJ17T+qKzaz7YakQLEQ=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoMQm3AnghYWARvLCOYDkjPsbfaSJXt7x+6cEI78CBsLRWz9PXb+GzfJFZr4YODx3gwz84JECoOu++0U1tY3NreK26Wd3b39g/LhUcvEqWa8yWIZ605ADZdC8SYKlLyTaE6jQPJ2ML6d+e0nro2I1QNOEu5HdKhEKBhFK7U7j1lVnE/75Ypbc+cgq8TLSQVyNPrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWqpoxI2fzc+dkjOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J47WdCJSlyxRaLwlQSjMnsdzIQmjOUE0so08LeStiIasrQJlSyIXjLL6+S1kXNc2ve/WWlfpPHUYQTOIUqeHAFdbiDBjSBwRie4RXenMR5cd6dj0VrwclnjuEPnM8fuSKPIg==</latexit><latexit sha1_base64="oebUgElUaJ17T+qKzaz7YakQLEQ=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoMQm3AnghYWARvLCOYDkjPsbfaSJXt7x+6cEI78CBsLRWz9PXb+GzfJFZr4YODx3gwz84JECoOu++0U1tY3NreK26Wd3b39g/LhUcvEqWa8yWIZ605ADZdC8SYKlLyTaE6jQPJ2ML6d+e0nro2I1QNOEu5HdKhEKBhFK7U7j1lVnE/75Ypbc+cgq8TLSQVyNPrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWqpoxI2fzc+dkjOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J47WdCJSlyxRaLwlQSjMnsdzIQmjOUE0so08LeStiIasrQJlSyIXjLL6+S1kXNc2ve/WWlfpPHUYQTOIUqeHAFdbiDBjSBwRie4RXenMR5cd6dj0VrwclnjuEPnM8fuSKPIg==</latexit><latexit sha1_base64="oebUgElUaJ17T+qKzaz7YakQLEQ=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoMQm3AnghYWARvLCOYDkjPsbfaSJXt7x+6cEI78CBsLRWz9PXb+GzfJFZr4YODx3gwz84JECoOu++0U1tY3NreK26Wd3b39g/LhUcvEqWa8yWIZ605ADZdC8SYKlLyTaE6jQPJ2ML6d+e0nro2I1QNOEu5HdKhEKBhFK7U7j1lVnE/75Ypbc+cgq8TLSQVyNPrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWqpoxI2fzc+dkjOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J47WdCJSlyxRaLwlQSjMnsdzIQmjOUE0so08LeStiIasrQJlSyIXjLL6+S1kXNc2ve/WWlfpPHUYQTOIUqeHAFdbiDBjSBwRie4RXenMR5cd6dj0VrwclnjuEPnM8fuSKPIg==</latexit>

f(X(i) = x(i)|✓)<latexit sha1_base64="JV06+CtdcBPBq8syHNKvX0Bw2uo=">AAACBnicbZDLSsNAFIYnXmu9RV2KMFiEdlMSEXShUHDjsoK9QBvLZDpph04mYeZELLErN76KGxeKuPUZ3Pk2TtsstPWHgY//nMOZ8/ux4Boc59taWFxaXlnNreXXNza3tu2d3bqOEkVZjUYiUk2faCa4ZDXgIFgzVoyEvmANf3A5rjfumNI8kjcwjJkXkp7kAacEjNWxD4Ji8zYt8tIIX+D7jB5wG/oMSKljF5yyMxGeBzeDAspU7dhf7W5Ek5BJoIJo3XKdGLyUKOBUsFG+nWgWEzogPdYyKEnItJdOzhjhI+N0cRAp8yTgift7IiWh1sPQN50hgb6erY3N/2qtBIIzL+UyToBJOl0UJAJDhMeZ4C5XjIIYGiBUcfNXTPtEEQomubwJwZ09eR7qx2XXKbvXJ4XKeRZHDu2jQ1RELjpFFXSFqqiGKHpEz+gVvVlP1ov1bn1MWxesbGYP/ZH1+QNUg5cZ</latexit><latexit sha1_base64="JV06+CtdcBPBq8syHNKvX0Bw2uo=">AAACBnicbZDLSsNAFIYnXmu9RV2KMFiEdlMSEXShUHDjsoK9QBvLZDpph04mYeZELLErN76KGxeKuPUZ3Pk2TtsstPWHgY//nMOZ8/ux4Boc59taWFxaXlnNreXXNza3tu2d3bqOEkVZjUYiUk2faCa4ZDXgIFgzVoyEvmANf3A5rjfumNI8kjcwjJkXkp7kAacEjNWxD4Ji8zYt8tIIX+D7jB5wG/oMSKljF5yyMxGeBzeDAspU7dhf7W5Ek5BJoIJo3XKdGLyUKOBUsFG+nWgWEzogPdYyKEnItJdOzhjhI+N0cRAp8yTgift7IiWh1sPQN50hgb6erY3N/2qtBIIzL+UyToBJOl0UJAJDhMeZ4C5XjIIYGiBUcfNXTPtEEQomubwJwZ09eR7qx2XXKbvXJ4XKeRZHDu2jQ1RELjpFFXSFqqiGKHpEz+gVvVlP1ov1bn1MWxesbGYP/ZH1+QNUg5cZ</latexit><latexit sha1_base64="JV06+CtdcBPBq8syHNKvX0Bw2uo=">AAACBnicbZDLSsNAFIYnXmu9RV2KMFiEdlMSEXShUHDjsoK9QBvLZDpph04mYeZELLErN76KGxeKuPUZ3Pk2TtsstPWHgY//nMOZ8/ux4Boc59taWFxaXlnNreXXNza3tu2d3bqOEkVZjUYiUk2faCa4ZDXgIFgzVoyEvmANf3A5rjfumNI8kjcwjJkXkp7kAacEjNWxD4Ji8zYt8tIIX+D7jB5wG/oMSKljF5yyMxGeBzeDAspU7dhf7W5Ek5BJoIJo3XKdGLyUKOBUsFG+nWgWEzogPdYyKEnItJdOzhjhI+N0cRAp8yTgift7IiWh1sPQN50hgb6erY3N/2qtBIIzL+UyToBJOl0UJAJDhMeZ4C5XjIIYGiBUcfNXTPtEEQomubwJwZ09eR7qx2XXKbvXJ4XKeRZHDu2jQ1RELjpFFXSFqqiGKHpEz+gVvVlP1ov1bn1MWxesbGYP/ZH1+QNUg5cZ</latexit><latexit sha1_base64="JV06+CtdcBPBq8syHNKvX0Bw2uo=">AAACBnicbZDLSsNAFIYnXmu9RV2KMFiEdlMSEXShUHDjsoK9QBvLZDpph04mYeZELLErN76KGxeKuPUZ3Pk2TtsstPWHgY//nMOZ8/ux4Boc59taWFxaXlnNreXXNza3tu2d3bqOEkVZjUYiUk2faCa4ZDXgIFgzVoyEvmANf3A5rjfumNI8kjcwjJkXkp7kAacEjNWxD4Ji8zYt8tIIX+D7jB5wG/oMSKljF5yyMxGeBzeDAspU7dhf7W5Ek5BJoIJo3XKdGLyUKOBUsFG+nWgWEzogPdYyKEnItJdOzhjhI+N0cRAp8yTgift7IiWh1sPQN50hgb6erY3N/2qtBIIzL+UyToBJOl0UJAJDhMeZ4C5XjIIYGiBUcfNXTPtEEQomubwJwZ09eR7qx2XXKbvXJ4XKeRZHDu2jQ1RELjpFFXSFqqiGKHpEz+gVvVlP1ov1bn1MWxesbGYP/ZH1+QNUg5cZ</latexit>

L(✓) =Y

i

f(X(i) = x(i)|✓)<latexit sha1_base64="Xueh2VOmh7cX8TKKOp/dsMC0u7M=">AAACGnicbVC7SgNBFJ31GeNr1dJmMAhJE3ZFMI0QsLGwiGAekF2X2clsMmT2wcxdMaz5Dht/xcZCETux8W+cJFto4oGBwznncucePxFcgWV9G0vLK6tr64WN4ubW9s6uubffUnEqKWvSWMSy4xPFBI9YEzgI1kkkI6EvWNsfXkz89h2TisfRDYwS5oakH/GAUwJa8kz7quzAgAGp4HPsJDLueRwH5c5tVuaVsdbuc/aA85xnlqyqNQVeJHZOSihHwzM/nV5M05BFQAVRqmtbCbgZkcCpYOOikyqWEDokfdbVNCIhU242PW2Mj7XSw0Es9YsAT9XfExkJlRqFvk6GBAZq3puI/3ndFIKam/EoSYFFdLYoSAWGGE96wj0uGQUx0oRQyfVfMR0QSSjoNou6BHv+5EXSOqnaVtW+Pi3Va3kdBXSIjlAZ2egM1dElaqAmougRPaNX9GY8GS/Gu/Exiy4Z+cwB+gPj6wcaUZ5i</latexit><latexit sha1_base64="Xueh2VOmh7cX8TKKOp/dsMC0u7M=">AAACGnicbVC7SgNBFJ31GeNr1dJmMAhJE3ZFMI0QsLGwiGAekF2X2clsMmT2wcxdMaz5Dht/xcZCETux8W+cJFto4oGBwznncucePxFcgWV9G0vLK6tr64WN4ubW9s6uubffUnEqKWvSWMSy4xPFBI9YEzgI1kkkI6EvWNsfXkz89h2TisfRDYwS5oakH/GAUwJa8kz7quzAgAGp4HPsJDLueRwH5c5tVuaVsdbuc/aA85xnlqyqNQVeJHZOSihHwzM/nV5M05BFQAVRqmtbCbgZkcCpYOOikyqWEDokfdbVNCIhU242PW2Mj7XSw0Es9YsAT9XfExkJlRqFvk6GBAZq3puI/3ndFIKam/EoSYFFdLYoSAWGGE96wj0uGQUx0oRQyfVfMR0QSSjoNou6BHv+5EXSOqnaVtW+Pi3Va3kdBXSIjlAZ2egM1dElaqAmougRPaNX9GY8GS/Gu/Exiy4Z+cwB+gPj6wcaUZ5i</latexit><latexit sha1_base64="Xueh2VOmh7cX8TKKOp/dsMC0u7M=">AAACGnicbVC7SgNBFJ31GeNr1dJmMAhJE3ZFMI0QsLGwiGAekF2X2clsMmT2wcxdMaz5Dht/xcZCETux8W+cJFto4oGBwznncucePxFcgWV9G0vLK6tr64WN4ubW9s6uubffUnEqKWvSWMSy4xPFBI9YEzgI1kkkI6EvWNsfXkz89h2TisfRDYwS5oakH/GAUwJa8kz7quzAgAGp4HPsJDLueRwH5c5tVuaVsdbuc/aA85xnlqyqNQVeJHZOSihHwzM/nV5M05BFQAVRqmtbCbgZkcCpYOOikyqWEDokfdbVNCIhU242PW2Mj7XSw0Es9YsAT9XfExkJlRqFvk6GBAZq3puI/3ndFIKam/EoSYFFdLYoSAWGGE96wj0uGQUx0oRQyfVfMR0QSSjoNou6BHv+5EXSOqnaVtW+Pi3Va3kdBXSIjlAZ2egM1dElaqAmougRPaNX9GY8GS/Gu/Exiy4Z+cwB+gPj6wcaUZ5i</latexit><latexit sha1_base64="Xueh2VOmh7cX8TKKOp/dsMC0u7M=">AAACGnicbVC7SgNBFJ31GeNr1dJmMAhJE3ZFMI0QsLGwiGAekF2X2clsMmT2wcxdMaz5Dht/xcZCETux8W+cJFto4oGBwznncucePxFcgWV9G0vLK6tr64WN4ubW9s6uubffUnEqKWvSWMSy4xPFBI9YEzgI1kkkI6EvWNsfXkz89h2TisfRDYwS5oakH/GAUwJa8kz7quzAgAGp4HPsJDLueRwH5c5tVuaVsdbuc/aA85xnlqyqNQVeJHZOSihHwzM/nV5M05BFQAVRqmtbCbgZkcCpYOOikyqWEDokfdbVNCIhU242PW2Mj7XSw0Es9YsAT9XfExkJlRqFvk6GBAZq3puI/3ndFIKam/EoSYFFdLYoSAWGGE96wj0uGQUx0oRQyfVfMR0QSSjoNou6BHv+5EXSOqnaVtW+Pi3Va3kdBXSIjlAZ2egM1dElaqAmougRPaNX9GY8GS/Gu/Exiy4Z+cwB+gPj6wcaUZ5i</latexit>

LL(✓) =X

i

log f(X(i) = x(i)|✓)<latexit sha1_base64="hDn6guzA9yU12FK9OkkzVwVW+fY=">AAACH3icbVC7SgNBFJ31GeMramkzGIRNE3ZFNI0QsLFIEcE8IBuX2clsMmT2wcxdMaz5Ext/xcZCEbHL3zhJttDEAwOHc87lzj1eLLgCy5oYK6tr6xubua389s7u3n7h4LCpokRS1qCRiGTbI4oJHrIGcBCsHUtGAk+wlje8nvqtByYVj8I7GMWsG5B+yH1OCWjJLVzUaqYDAwakhK+wo5LA5dgRUR/7Zvs+NXlprPXHjD3hLOsWilbZmgEvEzsjRZSh7ha+nV5Ek4CFQAVRqmNbMXRTIoFTwcZ5J1EsJnRI+qyjaUgCprrp7L4xPtVKD/uR1C8EPFN/T6QkUGoUeDoZEBioRW8q/ud1EvAr3ZSHcQIspPNFfiIwRHhaFu5xySiIkSaESq7/iumASEJBV5rXJdiLJy+T5lnZtsr27XmxWsnqyKFjdIJMZKNLVEU3qI4aiKJn9Ire0YfxYrwZn8bXPLpiZDNH6A+MyQ+u7aA+</latexit><latexit sha1_base64="hDn6guzA9yU12FK9OkkzVwVW+fY=">AAACH3icbVC7SgNBFJ31GeMramkzGIRNE3ZFNI0QsLFIEcE8IBuX2clsMmT2wcxdMaz5Ext/xcZCEbHL3zhJttDEAwOHc87lzj1eLLgCy5oYK6tr6xubua389s7u3n7h4LCpokRS1qCRiGTbI4oJHrIGcBCsHUtGAk+wlje8nvqtByYVj8I7GMWsG5B+yH1OCWjJLVzUaqYDAwakhK+wo5LA5dgRUR/7Zvs+NXlprPXHjD3hLOsWilbZmgEvEzsjRZSh7ha+nV5Ek4CFQAVRqmNbMXRTIoFTwcZ5J1EsJnRI+qyjaUgCprrp7L4xPtVKD/uR1C8EPFN/T6QkUGoUeDoZEBioRW8q/ud1EvAr3ZSHcQIspPNFfiIwRHhaFu5xySiIkSaESq7/iumASEJBV5rXJdiLJy+T5lnZtsr27XmxWsnqyKFjdIJMZKNLVEU3qI4aiKJn9Ire0YfxYrwZn8bXPLpiZDNH6A+MyQ+u7aA+</latexit><latexit sha1_base64="hDn6guzA9yU12FK9OkkzVwVW+fY=">AAACH3icbVC7SgNBFJ31GeMramkzGIRNE3ZFNI0QsLFIEcE8IBuX2clsMmT2wcxdMaz5Ext/xcZCEbHL3zhJttDEAwOHc87lzj1eLLgCy5oYK6tr6xubua389s7u3n7h4LCpokRS1qCRiGTbI4oJHrIGcBCsHUtGAk+wlje8nvqtByYVj8I7GMWsG5B+yH1OCWjJLVzUaqYDAwakhK+wo5LA5dgRUR/7Zvs+NXlprPXHjD3hLOsWilbZmgEvEzsjRZSh7ha+nV5Ek4CFQAVRqmNbMXRTIoFTwcZ5J1EsJnRI+qyjaUgCprrp7L4xPtVKD/uR1C8EPFN/T6QkUGoUeDoZEBioRW8q/ud1EvAr3ZSHcQIspPNFfiIwRHhaFu5xySiIkSaESq7/iumASEJBV5rXJdiLJy+T5lnZtsr27XmxWsnqyKFjdIJMZKNLVEU3qI4aiKJn9Ire0YfxYrwZn8bXPLpiZDNH6A+MyQ+u7aA+</latexit><latexit sha1_base64="hDn6guzA9yU12FK9OkkzVwVW+fY=">AAACH3icbVC7SgNBFJ31GeMramkzGIRNE3ZFNI0QsLFIEcE8IBuX2clsMmT2wcxdMaz5Ext/xcZCEbHL3zhJttDEAwOHc87lzj1eLLgCy5oYK6tr6xubua389s7u3n7h4LCpokRS1qCRiGTbI4oJHrIGcBCsHUtGAk+wlje8nvqtByYVj8I7GMWsG5B+yH1OCWjJLVzUaqYDAwakhK+wo5LA5dgRUR/7Zvs+NXlprPXHjD3hLOsWilbZmgEvEzsjRZSh7ha+nV5Ek4CFQAVRqmNbMXRTIoFTwcZ5J1EsJnRI+qyjaUgCprrp7L4xPtVKD/uR1C8EPFN/T6QkUGoUeDoZEBioRW8q/ud1EvAr3ZSHcQIspPNFfiIwRHhaFu5xySiIkSaESq7/iumASEJBV5rXJdiLJy+T5lnZtsr27XmxWsnqyKFjdIJMZKNLVEU3qI4aiKJn9Ire0YfxYrwZn8bXPLpiZDNH6A+MyQ+u7aA+</latexit>

✓̂MLE = argmax✓

⇣LL(✓)

⌘

<latexit sha1_base64="Ptbfrb57zgHU2WZ6s4+ybGGkEdM=">AAACRXicbVDPSxtBFJ5VazVWTe3Ry9AgxEvYFUEvBWkRPKRgoVEhG8LbyUsyODuzzLwthmX/OS/evfkf9NJDi3htZ5NQ/PVg4Hvf9z7emy/JlHQUhnfBwuLSm+W3K6u1tXfrG5v191tnzuRWYEcYZexFAg6V1NghSQovMouQJgrPk8svlX7+A62TRn+nSYa9FEZaDqUA8lS/HsdjoCKmMRKU/eJr+7jkn3ic64E34X+liE2GFshYDSkWYEcpXPGy5DHhFRXcg89y1Gy3mzPDbtXu9uuNsBVOi78E0Rw02LxO+/XbeGBEnqImocC5bhRm1PPrSAqFZS3OHWYgLmGEXQ+rW1yvmKZQ8h3PDPjQWP808Sn72FFA6twkTfxkCjR2z7WKfE3r5jQ87BVSZzmhFrNFw1xxMryKlA+kRUFq4gEIK/2tXIzBgiCfYc2HED3/8ktwtteKwlb0bb9xdDiPY4Vts4+sySJ2wI7YCTtlHSbYNfvJfrM/wU3wK7gPHmajC8Hc84E9qeDvPzFasxg=</latexit><latexit sha1_base64="Ptbfrb57zgHU2WZ6s4+ybGGkEdM=">AAACRXicbVDPSxtBFJ5VazVWTe3Ry9AgxEvYFUEvBWkRPKRgoVEhG8LbyUsyODuzzLwthmX/OS/evfkf9NJDi3htZ5NQ/PVg4Hvf9z7emy/JlHQUhnfBwuLSm+W3K6u1tXfrG5v191tnzuRWYEcYZexFAg6V1NghSQovMouQJgrPk8svlX7+A62TRn+nSYa9FEZaDqUA8lS/HsdjoCKmMRKU/eJr+7jkn3ic64E34X+liE2GFshYDSkWYEcpXPGy5DHhFRXcg89y1Gy3mzPDbtXu9uuNsBVOi78E0Rw02LxO+/XbeGBEnqImocC5bhRm1PPrSAqFZS3OHWYgLmGEXQ+rW1yvmKZQ8h3PDPjQWP808Sn72FFA6twkTfxkCjR2z7WKfE3r5jQ87BVSZzmhFrNFw1xxMryKlA+kRUFq4gEIK/2tXIzBgiCfYc2HED3/8ktwtteKwlb0bb9xdDiPY4Vts4+sySJ2wI7YCTtlHSbYNfvJfrM/wU3wK7gPHmajC8Hc84E9qeDvPzFasxg=</latexit><latexit sha1_base64="Ptbfrb57zgHU2WZ6s4+ybGGkEdM=">AAACRXicbVDPSxtBFJ5VazVWTe3Ry9AgxEvYFUEvBWkRPKRgoVEhG8LbyUsyODuzzLwthmX/OS/evfkf9NJDi3htZ5NQ/PVg4Hvf9z7emy/JlHQUhnfBwuLSm+W3K6u1tXfrG5v191tnzuRWYEcYZexFAg6V1NghSQovMouQJgrPk8svlX7+A62TRn+nSYa9FEZaDqUA8lS/HsdjoCKmMRKU/eJr+7jkn3ic64E34X+liE2GFshYDSkWYEcpXPGy5DHhFRXcg89y1Gy3mzPDbtXu9uuNsBVOi78E0Rw02LxO+/XbeGBEnqImocC5bhRm1PPrSAqFZS3OHWYgLmGEXQ+rW1yvmKZQ8h3PDPjQWP808Sn72FFA6twkTfxkCjR2z7WKfE3r5jQ87BVSZzmhFrNFw1xxMryKlA+kRUFq4gEIK/2tXIzBgiCfYc2HED3/8ktwtteKwlb0bb9xdDiPY4Vts4+sySJ2wI7YCTtlHSbYNfvJfrM/wU3wK7gPHmajC8Hc84E9qeDvPzFasxg=</latexit><latexit sha1_base64="Ptbfrb57zgHU2WZ6s4+ybGGkEdM=">AAACRXicbVDPSxtBFJ5VazVWTe3Ry9AgxEvYFUEvBWkRPKRgoVEhG8LbyUsyODuzzLwthmX/OS/evfkf9NJDi3htZ5NQ/PVg4Hvf9z7emy/JlHQUhnfBwuLSm+W3K6u1tXfrG5v191tnzuRWYEcYZexFAg6V1NghSQovMouQJgrPk8svlX7+A62TRn+nSYa9FEZaDqUA8lS/HsdjoCKmMRKU/eJr+7jkn3ic64E34X+liE2GFshYDSkWYEcpXPGy5DHhFRXcg89y1Gy3mzPDbtXu9uuNsBVOi78E0Rw02LxO+/XbeGBEnqImocC5bhRm1PPrSAqFZS3OHWYgLmGEXQ+rW1yvmKZQ8h3PDPjQWP808Sn72FFA6twkTfxkCjR2z7WKfE3r5jQ87BVSZzmhFrNFw1xxMryKlA+kRUFq4gEIK/2tXIzBgiCfYc2HED3/8ktwtteKwlb0bb9xdDiPY4Vts4+sySJ2wI7YCTtlHSbYNfvJfrM/wU3wK7gPHmajC8Hc84E9qeDvPzFasxg=</latexit>

Likelihood of d

ata

Log Likelihood

Max Likelihood

Think: observations of n IID random variables:

Where: has likelihood (PDF) function:

You have now estimated parameters…

Side Plot

argmaxargmax of log

Mother of optimizations?

Gradient Ascent

Linear Regression (simple)

(x(1), y(1)), (x(2), y(2)), . . . (x(n), y(n))

N training datapoints

Linear Regression Lite Model

Y = ✓ ·X + Z Y |X ⇠ N(✓X,�2)Z ⇠ N(0,�2)

X = CO2 level

Y = Average Global Temperature

(x(1), y(1)), (x(2), y(2)), . . . (x(n), y(n))N training datapoints:

@LL(✓)

@✓=

nX

i=1

2(y(i) � ✓x(i))(x(i))<latexit sha1_base64="W1jF0sJdUY9hRZ8xXchawJ45ySM=">AAACTXicbZHPSxtBFMdnY6sxtjbqsZdHg7A5NOyKUC+C6KWHHCw0JpBNwuxkNhmcnV1m3oph2X/Qi+DN/6IXD4qIk2RL648HA18+3/eYN98JUykMet6tU1n58HF1rbpe2/j0efNLfWv7zCSZZrzDEpnoXkgNl0LxDgqUvJdqTuNQ8m54fjL3uxdcG5Go3zhL+SCmEyUiwShaNKqPg0hTlgcp1SiohHbbDXDKkTah+EeXqIBDgMBk8SgXh34xVAB74M6GuSuaBXwvu+ByCZrg/lWjesNreYuCt8IvRYOUdTqq3wTjhGUxV8gkNabveykO8vk2TPKiFmSGp5Sd0wnvW6lozM0gX6RRwK4lY4gSbY9CWND/J3IaGzOLQ9sZU5ya194cvuf1M4wOBrlQaYZcseVFUSYBE5hHC2OhOUM5s4IyLeyuwKbUxov2A2o2BP/1k9+Ks72W77X8X/uNo+Myjir5Sr4Rl/jkBzkiP8kp6RBGrsgfck8enGvnznl0npatFaec2SEvqrL2DExksW8=</latexit><latexit sha1_base64="W1jF0sJdUY9hRZ8xXchawJ45ySM=">AAACTXicbZHPSxtBFMdnY6sxtjbqsZdHg7A5NOyKUC+C6KWHHCw0JpBNwuxkNhmcnV1m3oph2X/Qi+DN/6IXD4qIk2RL648HA18+3/eYN98JUykMet6tU1n58HF1rbpe2/j0efNLfWv7zCSZZrzDEpnoXkgNl0LxDgqUvJdqTuNQ8m54fjL3uxdcG5Go3zhL+SCmEyUiwShaNKqPg0hTlgcp1SiohHbbDXDKkTah+EeXqIBDgMBk8SgXh34xVAB74M6GuSuaBXwvu+ByCZrg/lWjesNreYuCt8IvRYOUdTqq3wTjhGUxV8gkNabveykO8vk2TPKiFmSGp5Sd0wnvW6lozM0gX6RRwK4lY4gSbY9CWND/J3IaGzOLQ9sZU5ya194cvuf1M4wOBrlQaYZcseVFUSYBE5hHC2OhOUM5s4IyLeyuwKbUxov2A2o2BP/1k9+Ks72W77X8X/uNo+Myjir5Sr4Rl/jkBzkiP8kp6RBGrsgfck8enGvnznl0npatFaec2SEvqrL2DExksW8=</latexit><latexit sha1_base64="W1jF0sJdUY9hRZ8xXchawJ45ySM=">AAACTXicbZHPSxtBFMdnY6sxtjbqsZdHg7A5NOyKUC+C6KWHHCw0JpBNwuxkNhmcnV1m3oph2X/Qi+DN/6IXD4qIk2RL648HA18+3/eYN98JUykMet6tU1n58HF1rbpe2/j0efNLfWv7zCSZZrzDEpnoXkgNl0LxDgqUvJdqTuNQ8m54fjL3uxdcG5Go3zhL+SCmEyUiwShaNKqPg0hTlgcp1SiohHbbDXDKkTah+EeXqIBDgMBk8SgXh34xVAB74M6GuSuaBXwvu+ByCZrg/lWjesNreYuCt8IvRYOUdTqq3wTjhGUxV8gkNabveykO8vk2TPKiFmSGp5Sd0wnvW6lozM0gX6RRwK4lY4gSbY9CWND/J3IaGzOLQ9sZU5ya194cvuf1M4wOBrlQaYZcseVFUSYBE5hHC2OhOUM5s4IyLeyuwKbUxov2A2o2BP/1k9+Ks72W77X8X/uNo+Myjir5Sr4Rl/jkBzkiP8kp6RBGrsgfck8enGvnznl0npatFaec2SEvqrL2DExksW8=</latexit><latexit sha1_base64="W1jF0sJdUY9hRZ8xXchawJ45ySM=">AAACTXicbZHPSxtBFMdnY6sxtjbqsZdHg7A5NOyKUC+C6KWHHCw0JpBNwuxkNhmcnV1m3oph2X/Qi+DN/6IXD4qIk2RL648HA18+3/eYN98JUykMet6tU1n58HF1rbpe2/j0efNLfWv7zCSZZrzDEpnoXkgNl0LxDgqUvJdqTuNQ8m54fjL3uxdcG5Go3zhL+SCmEyUiwShaNKqPg0hTlgcp1SiohHbbDXDKkTah+EeXqIBDgMBk8SgXh34xVAB74M6GuSuaBXwvu+ByCZrg/lWjesNreYuCt8IvRYOUdTqq3wTjhGUxV8gkNabveykO8vk2TPKiFmSGp5Sd0wnvW6lozM0gX6RRwK4lY4gSbY9CWND/J3IaGzOLQ9sZU5ya194cvuf1M4wOBrlQaYZcseVFUSYBE5hHC2OhOUM5s4IyLeyuwKbUxov2A2o2BP/1k9+Ks72W77X8X/uNo+Myjir5Sr4Rl/jkBzkiP8kp6RBGrsgfck8enGvnznl0npatFaec2SEvqrL2DExksW8=</latexit>

✓̂ = argmax✓

�nX

i=1

(y(i) � ✓x(i))2

<latexit sha1_base64="av8GdCA/JzJToLwlt7s51dpCrhA=">AAACTnicbZA9axtBEIb3ZCeRlQ8rSelmsUiQi4g7E0gag0malDZEtkEnibnVSFq8H8fuXLA47hemCe7yM9KkSDD2niSCPzKw8PK8M8zsm+VKeorjn1FjY/PR4yfNrdbTZ89fbLdfvjrxtnAC+8Iq684y8KikwT5JUniWOwSdKTzNzj/X/uk3dF5a85UWOQ41zIycSgEU0LiN6RyoTGmOBBV/e8DTwkxCP/6DZWpzdEDWGdBYgptpuKgqzt/x1Bd6XMqDpBoZ3l2Myq7cq2q+nOQXK7A32h+3O3EvXhZ/KJK16LB1HY3bl+nEikKjIaHA+0ES5zQMy0kKhVUrLTzmIM5hhoMg68v8sFzGUfE3gUz41LrwDPElvT1RgvZ+obPQqYHm/r5Xw/95g4KmH4elNHlBaMRq0bRQnCyvs+UT6VCQWgQBwslwKxdzcCAoJNoKIST3v/xQnOz3kriXHL/vHH5ax9FkO2yXdVnCPrBD9oUdsT4T7Dv7xf6wv9GP6Hd0FV2vWhvReuY1u1ON5g3bRrSW</latexit><latexit sha1_base64="av8GdCA/JzJToLwlt7s51dpCrhA=">AAACTnicbZA9axtBEIb3ZCeRlQ8rSelmsUiQi4g7E0gag0malDZEtkEnibnVSFq8H8fuXLA47hemCe7yM9KkSDD2niSCPzKw8PK8M8zsm+VKeorjn1FjY/PR4yfNrdbTZ89fbLdfvjrxtnAC+8Iq684y8KikwT5JUniWOwSdKTzNzj/X/uk3dF5a85UWOQ41zIycSgEU0LiN6RyoTGmOBBV/e8DTwkxCP/6DZWpzdEDWGdBYgptpuKgqzt/x1Bd6XMqDpBoZ3l2Myq7cq2q+nOQXK7A32h+3O3EvXhZ/KJK16LB1HY3bl+nEikKjIaHA+0ES5zQMy0kKhVUrLTzmIM5hhoMg68v8sFzGUfE3gUz41LrwDPElvT1RgvZ+obPQqYHm/r5Xw/95g4KmH4elNHlBaMRq0bRQnCyvs+UT6VCQWgQBwslwKxdzcCAoJNoKIST3v/xQnOz3kriXHL/vHH5ax9FkO2yXdVnCPrBD9oUdsT4T7Dv7xf6wv9GP6Hd0FV2vWhvReuY1u1ON5g3bRrSW</latexit><latexit sha1_base64="av8GdCA/JzJToLwlt7s51dpCrhA=">AAACTnicbZA9axtBEIb3ZCeRlQ8rSelmsUiQi4g7E0gag0malDZEtkEnibnVSFq8H8fuXLA47hemCe7yM9KkSDD2niSCPzKw8PK8M8zsm+VKeorjn1FjY/PR4yfNrdbTZ89fbLdfvjrxtnAC+8Iq684y8KikwT5JUniWOwSdKTzNzj/X/uk3dF5a85UWOQ41zIycSgEU0LiN6RyoTGmOBBV/e8DTwkxCP/6DZWpzdEDWGdBYgptpuKgqzt/x1Bd6XMqDpBoZ3l2Myq7cq2q+nOQXK7A32h+3O3EvXhZ/KJK16LB1HY3bl+nEikKjIaHA+0ES5zQMy0kKhVUrLTzmIM5hhoMg68v8sFzGUfE3gUz41LrwDPElvT1RgvZ+obPQqYHm/r5Xw/95g4KmH4elNHlBaMRq0bRQnCyvs+UT6VCQWgQBwslwKxdzcCAoJNoKIST3v/xQnOz3kriXHL/vHH5ax9FkO2yXdVnCPrBD9oUdsT4T7Dv7xf6wv9GP6Hd0FV2vWhvReuY1u1ON5g3bRrSW</latexit><latexit sha1_base64="av8GdCA/JzJToLwlt7s51dpCrhA=">AAACTnicbZA9axtBEIb3ZCeRlQ8rSelmsUiQi4g7E0gag0malDZEtkEnibnVSFq8H8fuXLA47hemCe7yM9KkSDD2niSCPzKw8PK8M8zsm+VKeorjn1FjY/PR4yfNrdbTZ89fbLdfvjrxtnAC+8Iq684y8KikwT5JUniWOwSdKTzNzj/X/uk3dF5a85UWOQ41zIycSgEU0LiN6RyoTGmOBBV/e8DTwkxCP/6DZWpzdEDWGdBYgptpuKgqzt/x1Bd6XMqDpBoZ3l2Myq7cq2q+nOQXK7A32h+3O3EvXhZ/KJK16LB1HY3bl+nEikKjIaHA+0ES5zQMy0kKhVUrLTzmIM5hhoMg68v8sFzGUfE3gUz41LrwDPElvT1RgvZ+obPQqYHm/r5Xw/95g4KmH4elNHlBaMRq0bRQnCyvs+UT6VCQWgQBwslwKxdzcCAoJNoKIST3v/xQnOz3kriXHL/vHH5ax9FkO2yXdVnCPrBD9oUdsT4T7Dv7xf6wv9GP6Hd0FV2vWhvReuY1u1ON5g3bRrSW</latexit>


Initialize: θ = 0

Repeat many times:

For each training example (x, y):

! += h * gradient

gradient = 0

gradient += 2(y - θ x)(x)


X1 = Temperature

X2 = Elevation

X3 = CO2 level yesterday

X4 = GDP of region

X5 = Acres of forest growth

Y = CO2 levels

Linear Regression (regular)

Training: Gradient ascent to chose the best thetas to describe your data

Problem: Predict real value Y based on observing variable X

Model: Linear weight every feature

Y = ✓1X1 + · · ·+ ✓mXm + Z

= ✓TX+ Z<latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit>


⇣�

nX

i=1

(y(i) � ✓x(i))2⌘

<latexit sha1_base64="kTCCsSOsuCXV9698bh7dzD2Oels=">AAACX3icbVBdaxNBFJ1dP1rT2q76JL4MBiV5MOwWQV8KpSL4oFDBtIVsstydTJKh87HM3C0Nw/5J3wRf/CfOboJo64WBwzn3cM+cspLCYZr+iOJ79x883Nl91Nvbf3xwmDx5eu5MbRkfMyONvSzBcSk0H6NAyS8ry0GVkl+UVx9a/eKaWyeM/obrik8VLLVYCAYYqCK5zleAPscVR2gK/+Xzx4a+PqZ5refBxf9IPjcVt4DGalDcg10quGkamp+K5YC+obmrVeHFcdbMNB2sZ34ghk3Ld3Z6syGGs6POMSySfjpKu6F3QbYFfbKdsyL5ns8NqxXXyCQ4N8nSCqchBwomedPLa8crYFew5JMA25Bu6rt+GvoqMHO6MDY8jbRj/3Z4UM6tVRk2FeDK3dZa8n/apMbF+6kXuqqRa7Y5tKglRUPbsulcWM5QrgMAZkXIStkKLDAM5fZCCdntL98F50ejLB1lX9/2T063deySF+QlGZCMvCMn5BM5I2PCyM8ojvai/ehXvBMfxMlmNY62nmfkn4mf/wZ2grTI</latexit><latexit sha1_base64="kTCCsSOsuCXV9698bh7dzD2Oels=">AAACX3icbVBdaxNBFJ1dP1rT2q76JL4MBiV5MOwWQV8KpSL4oFDBtIVsstydTJKh87HM3C0Nw/5J3wRf/CfOboJo64WBwzn3cM+cspLCYZr+iOJ79x883Nl91Nvbf3xwmDx5eu5MbRkfMyONvSzBcSk0H6NAyS8ry0GVkl+UVx9a/eKaWyeM/obrik8VLLVYCAYYqCK5zleAPscVR2gK/+Xzx4a+PqZ5refBxf9IPjcVt4DGalDcg10quGkamp+K5YC+obmrVeHFcdbMNB2sZ34ghk3Ld3Z6syGGs6POMSySfjpKu6F3QbYFfbKdsyL5ns8NqxXXyCQ4N8nSCqchBwomedPLa8crYFew5JMA25Bu6rt+GvoqMHO6MDY8jbRj/3Z4UM6tVRk2FeDK3dZa8n/apMbF+6kXuqqRa7Y5tKglRUPbsulcWM5QrgMAZkXIStkKLDAM5fZCCdntL98F50ejLB1lX9/2T063deySF+QlGZCMvCMn5BM5I2PCyM8ojvai/ehXvBMfxMlmNY62nmfkn4mf/wZ2grTI</latexit><latexit sha1_base64="kTCCsSOsuCXV9698bh7dzD2Oels=">AAACX3icbVBdaxNBFJ1dP1rT2q76JL4MBiV5MOwWQV8KpSL4oFDBtIVsstydTJKh87HM3C0Nw/5J3wRf/CfOboJo64WBwzn3cM+cspLCYZr+iOJ79x883Nl91Nvbf3xwmDx5eu5MbRkfMyONvSzBcSk0H6NAyS8ry0GVkl+UVx9a/eKaWyeM/obrik8VLLVYCAYYqCK5zleAPscVR2gK/+Xzx4a+PqZ5refBxf9IPjcVt4DGalDcg10quGkamp+K5YC+obmrVeHFcdbMNB2sZ34ghk3Ld3Z6syGGs6POMSySfjpKu6F3QbYFfbKdsyL5ns8NqxXXyCQ4N8nSCqchBwomedPLa8crYFew5JMA25Bu6rt+GvoqMHO6MDY8jbRj/3Z4UM6tVRk2FeDK3dZa8n/apMbF+6kXuqqRa7Y5tKglRUPbsulcWM5QrgMAZkXIStkKLDAM5fZCCdntL98F50ejLB1lX9/2T063deySF+QlGZCMvCMn5BM5I2PCyM8ojvai/ehXvBMfxMlmNY62nmfkn4mf/wZ2grTI</latexit><latexit sha1_base64="kTCCsSOsuCXV9698bh7dzD2Oels=">AAACX3icbVBdaxNBFJ1dP1rT2q76JL4MBiV5MOwWQV8KpSL4oFDBtIVsstydTJKh87HM3C0Nw/5J3wRf/CfOboJo64WBwzn3cM+cspLCYZr+iOJ79x883Nl91Nvbf3xwmDx5eu5MbRkfMyONvSzBcSk0H6NAyS8ry0GVkl+UVx9a/eKaWyeM/obrik8VLLVYCAYYqCK5zleAPscVR2gK/+Xzx4a+PqZ5refBxf9IPjcVt4DGalDcg10quGkamp+K5YC+obmrVeHFcdbMNB2sZ34ghk3Ld3Z6syGGs6POMSySfjpKu6F3QbYFfbKdsyL5ns8NqxXXyCQ4N8nSCqchBwomedPLa8crYFew5JMA25Bu6rt+GvoqMHO6MDY8jbRj/3Z4UM6tVRk2FeDK3dZa8n/apMbF+6kXuqqRa7Y5tKglRUPbsulcWM5QrgMAZkXIStkKLDAM5fZCCdntL98F50ejLB1lX9/2T063deySF+QlGZCMvCMn5BM5I2PCyM8ojvai/ehXvBMfxMlmNY62nmfkn4mf/wZ2grTI</latexit>


Initialize: θj = 0 for all 0 ≤ j ≤ m

Repeat many times:

For each training example (x, y):

For each parameter j:

!j += h * gradient[j] for all 0 ≤ j ≤ m

gradient[j] = 0 for all 0 ≤ j ≤ m

gradient[j] += (y – θTx)(-x[j])


Predicting WarriorsY = Warriors points

X1 = Opposing team ELO

X2 = Points in last game

X3 = Curry playing?

X4 = Playing at home?

!1 = -2.3

!2 = +1.2

!3 = +10.2

!4 = +3.3

!5 = +95.4X5 = 1

Y = ✓1X1 + · · ·+ ✓mXm + Z

= ✓TX+ Z<latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit><latexit sha1_base64="Op4aU8HHVbhDfSncb+88R15unkU=">AAACQnicbVBNS8NAEN34WetX1aOXxaIIQklE0IOC4MVjBavVppbNdmIXd5OwOxFL6G/z4i/w5g/w4kERrx7cpD34NTDM2zdvmNkXJFIYdN0nZ2x8YnJqujRTnp2bX1isLC2fmTjVHBo8lrFuBsyAFBE0UKCEZqKBqUDCeXBzlPfPb0EbEUen2E+grdh1JELBGVqqU7m4oOWNA+pjD5B1PNq0uUX9bowmr0NaWVrZ5yX1/UIdxLJr+sqWoeIqOx1YMdxhEGbNQSHtVKpuzS2C/gXeCFTJKOqdyqNdy1MFEXLJjGl5boLtjGkUXMKg7KcGEsZv2DW0LIyYAtPOCgsGdN0yXRrG2maEtGC/T2RMmfxiq1QMe+Z3Lyf/67VSDPfamYiSFCHiw0VhKinGNPeTdoUGjrJvAeNa2Fsp7zHNOFrXy9YE7/eX/4Kz7Zrn1ryTnerh/siOElkla2STeGSXHJJjUicNwsk9eSav5M15cF6cd+djKB1zRjMr5Ec4n193r6y0</latexit>

Supervised LearningReal World Problem

Formal Model !

Prediction Function !*

Model the problem

Learning Algorithm

Training Data

NewData

Predictions

Modelling

Real World Problem

Formal Model !


Model the problem

Learning Algorithm

Training Data

NewData

Predictions

TrainingReal World Problem

Formal Model !


Model the problem

Learning Algorithm

Training Data

NewData

Predictions

Make Predictions

Real World Problem

Formal Model !


Model the problem

Learning Algorithm

NewData

Training Data

Predictions

Our Path

Linear

Regressio

n Naïve Baye

s Logistic

Regressio

n

Neural Networks

Unbiased

estimators Maxim

izing

likelihood Baye

sian

estimation

Episode 2 The Song of The Last Estimator

Something rottenin the world of MLE

Foreshadowing..

So good to see you again!

Need a Volunteer

• I have two envelopes, will allow you to have one§ One contains $X, the other contains $2X

§ Select an envelopeo Open it!

§ Now, would you like to switch for other envelope?

§ To help you decide, compute E[$ in other envelope]o Let Y = $ in envelope you selected

§ Before opening envelope, think either equally good

§ So, what happened by opening envelope?o And does it really make sense to switch?

YYE Y45

21

221 2]envelopeother in $[ =×+×=

Two Envelopes

• The “two envelopes” problem set-up§ Two envelopes: one contains $X, other contains $2X§ You select an envelope and open it

o Let Y = $ in envelope you selectedo Let Z = $ in other envelope

§ E[Z | Y] above assumes all values X (where 0 < X < ¥) are equally likelyo Note: there are infinitely many values of Xo So, not true probability distribution over X (doesn’t integrate to 1)

YYYZE Y45

21

221 2]|[ =×+×=

Thinking Deeper About Two Envelopes

All Values are Equally Likely?

X

p(X)

0 10 20 10060 8040

Infinite powers of two…

• Belief about contents of envelopes§ Since implied distribution over X is not a true probability

distribution, what is our distribution over X?o Frequentist: play game infinitely many times and see how often

different values come up.o Problem: I only allow you to play the game once

§ Bayesian probabilityo Have prior belief of distribution for X (or anything for that matter)o Prior belief is a subjective probability

• By extension, all probabilities are subjectiveo Allows us to answer question when we have no/limited data

• E.g., probability a coin you’ve never flipped lands on headso As we get more data, prior belief is “swamped” by data

Subjectivity of Probability

Subjectivity of Probability

X

p(X)

0 10 20 10060 8040

• Bayesian: have prior distribution over X, P(X)§ Let Y = $ in envelope you selected§ Let Z = $ in other envelope§ Open your envelope to determine Y§ If Y > E[Z | Y], keep your envelope, otherwise switch

o No inconsistency!§ Opening envelope provides data to compute P(X | Y)

and thereby compute E[Z | Y]§ Of course, there’s the issue of how you determined

your prior distribution over X…o Bayesian: Doesn’t matter how you determined prior, but you must have one (whatever it is)

o Imagine if envelope you opened contained $20.01

The Envelope, Please

Envelope Summary:Probabilities are beliefs

Incorporating prior beliefs is useful

Priors for Parameter Estimation?

• Bayes’ Theorem (q = model parameters, D = data):

§ Likelihood: you’ve seen this before (in context of MLE)

o Probability of data given probability model (parameter q)

§ Prior: before seeing any data, what is belief about model

o I.e., what is distribution over parameters q

§ Posterior: after seeing data, what is belief about model

o After data D observed, have posterior distribution p(q | D) over

parameters q conditioned on data. Use this to predict new data.

P(D | q) P(q)

P(D)P(q | D) =

“Prior”“Likelihood”“Posterior”

Flash Back: Bayes Theorem

Maximum Likelihood Estimation

Maximum A Posteriori

MLE vs MAP

✓̂MAP = argmax✓

f(⇥ = ✓|X(1) = x(1), . . . , X(n) = x(n))<latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit>


f(X(1) = x(1), . . . , X(n) = x(n)|✓)

= argmax✓

⇣X

i

log f(X(i) = x(i)|✓)⌘

<latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit>


f(X(1) = x(1), . . . , X(n) = x(n)|✓)

= argmax✓

⇣X

i

log f(X(i) = x(i)|✓)⌘

<latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit><latexit sha1_base64="aUsK+mvq/05/+zTPCuGMo3jxx+k=">AAAC1niclVHLihNBFK1uX2N8RV26uRiUBIbQPQjOZmBQBBcKI5qZQCo21ZVKp5jqqqbqtiS07UIRt36bOz/Cf7A6HcM448YLBYdz7q1zH2mhpMMo+hmEly5fuXpt53rnxs1bt+907947dqa0XIy4UcaOU+aEklqMUKIS48IKlqdKnKSnzxv95IOwThr9DleFmOYs03IuOUNPJd1fdMGworgQyOqkev3qRQ2PD4CWeuarxFaqqCmEZWisZrmomM1ytqxrAIpiiRXU8/74fdWPBzUcwLJFu0BnBt0uNIreKg36CO2/MKC08z9+f+yAPpNZH6gr80QCVSaDtgO59ZFnfAbr/EHS7UXDaB1wEcQb0CObOEq6P/wIvMyFRq6Yc5M4KnDq20HJlag7tHSiYPyUZWLiYdOrm1brs9TwyDMzmBvrn0ZYs2crKpY7t8pTn5kzXLjzWkP+S5uUON+fVlIXJQrNW6N5qQANNDeGmbSCo1p5wLiVvlfgC2YZR7/jjl9CfH7ki+B4bxhHw/jNk97h/mYdO+QBeUj6JCZPySF5SY7IiPDgbbAKPgdfwnH4KfwafmtTw2BTc5/8FeH33x8V4U0=</latexit>

x(1), . . . , x(n)<latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit><latexit sha1_base64="GiChI1fA8t4wAwAIcrFSc/9D7vk=">AAACAHicbVDLSgMxFM3UV62vqgsXboJFaKGUGRF04aLgxmUF+4B2LJk004ZmMkNyRyzDbPwVNy4UcetnuPNvzLRdaOuBwMk5997kHi8SXINtf1u5ldW19Y38ZmFre2d3r7h/0NJhrChr0lCEquMRzQSXrAkcBOtEipHAE6ztja8zv/3AlOahvINJxNyADCX3OSVgpH7x6PE+KTuVtIp7gxB0FWd3WUn7xZJds6fAy8SZkxKao9EvfpkBNA6YBCqI1l3HjsBNiAJOBUsLvViziNAxGbKuoZIETLvJdIEUnxplgP1QmSMBT9XfHQkJtJ4EnqkMCIz0opeJ/3ndGPxLN+EyioFJOnvIjwWGEGdp4AFXjIKYGEKo4uavmI6IIhRMZgUTgrO48jJpndUcu+bcnpfqV/M48ugYnaAyctAFqqMb1EBNRFGKntErerOerBfr3fqYleasec8h+gPr8weWF5UW</latexit>

Data:

Notation Shorthand


f(⇥ = ✓|X(1) = x(1), . . . , X(n) = x(n))<latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit><latexit sha1_base64="CThWmp611GyF340xXnLlj+lHfLg=">AAACcnicbVBdi9NAFJ3Er7V+VcUXBR0tShdKSURwX4QVX3wRKmx3C00tN9ObdtjJTJi5kZaYH+Df881f4Ys/wEk2LLrrgYHDOffM3DlpoaSjKPoZhFeuXrt+Y+9m79btO3fv9e8/OHamtAKnwihjZyk4VFLjlCQpnBUWIU8VnqSnHxr/5CtaJ40+ol2BixzWWmZSAHlp2f+ebICqhDZIUC+rT+8nNX/1jielXvkUnltVYgq0QMZqyLECu85hW9ecJ4RbqnidDZOjZpL7cBvh3/jsSzWM92svbc/YiCcrQ27UOvrcadj+sj+IxlELfpnEHRmwDpNl/4e/TJQ5ahIKnJvHUUELvxpJobDuJaXDAsQprHHuabO3W1RtZTV/6ZUVz4z1RxNv1b8TFeTO7fLUT+ZAG3fRa8T/efOSsoNFJXVREmpx9lBWKk6GN/3zlbQoSO08AWGl35WLDVgQ5Pvu+RLii1++TI5fj+NoHH9+Mzg86OrYY0/YCzZkMXvLDtlHNmFTJtiv4FHwNHgW/A4fh8/Drrsw6DIP2T8IR38AlbO7VQ==</latexit>

x(i) is shorthand for the event: X(i) = x(i)<latexit sha1_base64="Lc8LdtVOnd31m6WsQdhiC9Ir3ow=">AAACJ3icbVDLSgMxFM34tr6qLt0Ei6CbMiOCIiiCG5cK9gFtLZn0jhPMJENyRyxD/8aNv+JGUBFd+iem7Sx83dXhPJJ7T5hKYdH3P7yJyanpmdm5+dLC4tLySnl1rW51ZjjUuJbaNENmQQoFNRQooZkaYEkooRHenA71xi0YK7S6xH4KnYRdKxEJztBR3fLx3VW+LXYGtI1whzkVltpYG4yZ6tFIG4oxULgFhYd0QJuF+YgWsW654lf90dC/IChAhRRz3i0/t3uaZ4l7kEtmbSvwU+zkzKDgEgaldmYhZfyGXUPLQcUSsJ18dOeAbjlmvFWkFdIR+z2Rs8TafhI6Z8Iwtr+1Ifmf1sowOujkQqUZguLjj6JMUtR0WBrtCQMcZd8Bxo1wu1IeM8M4umpLroTg98l/QX23GvjV4GKvcnJQ1DFHNsgm2SYB2Scn5Iyckxrh5J48khfy6j14T96b9z62TnhFZp38GO/zC+s+pKY=</latexit><latexit sha1_base64="Lc8LdtVOnd31m6WsQdhiC9Ir3ow=">AAACJ3icbVDLSgMxFM34tr6qLt0Ei6CbMiOCIiiCG5cK9gFtLZn0jhPMJENyRyxD/8aNv+JGUBFd+iem7Sx83dXhPJJ7T5hKYdH3P7yJyanpmdm5+dLC4tLySnl1rW51ZjjUuJbaNENmQQoFNRQooZkaYEkooRHenA71xi0YK7S6xH4KnYRdKxEJztBR3fLx3VW+LXYGtI1whzkVltpYG4yZ6tFIG4oxULgFhYd0QJuF+YgWsW654lf90dC/IChAhRRz3i0/t3uaZ4l7kEtmbSvwU+zkzKDgEgaldmYhZfyGXUPLQcUSsJ18dOeAbjlmvFWkFdIR+z2Rs8TafhI6Z8Iwtr+1Ifmf1sowOujkQqUZguLjj6JMUtR0WBrtCQMcZd8Bxo1wu1IeM8M4umpLroTg98l/QX23GvjV4GKvcnJQ1DFHNsgm2SYB2Scn5Iyckxrh5J48khfy6j14T96b9z62TnhFZp38GO/zC+s+pKY=</latexit><latexit sha1_base64="Lc8LdtVOnd31m6WsQdhiC9Ir3ow=">AAACJ3icbVDLSgMxFM34tr6qLt0Ei6CbMiOCIiiCG5cK9gFtLZn0jhPMJENyRyxD/8aNv+JGUBFd+iem7Sx83dXhPJJ7T5hKYdH3P7yJyanpmdm5+dLC4tLySnl1rW51ZjjUuJbaNENmQQoFNRQooZkaYEkooRHenA71xi0YK7S6xH4KnYRdKxEJztBR3fLx3VW+LXYGtI1whzkVltpYG4yZ6tFIG4oxULgFhYd0QJuF+YgWsW654lf90dC/IChAhRRz3i0/t3uaZ4l7kEtmbSvwU+zkzKDgEgaldmYhZfyGXUPLQcUSsJ18dOeAbjlmvFWkFdIR+z2Rs8TafhI6Z8Iwtr+1Ifmf1sowOujkQqUZguLjj6JMUtR0WBrtCQMcZd8Bxo1wu1IeM8M4umpLroTg98l/QX23GvjV4GKvcnJQ1DFHNsgm2SYB2Scn5Iyckxrh5J48khfy6j14T96b9z62TnhFZp38GO/zC+s+pKY=</latexit><latexit sha1_base64="Lc8LdtVOnd31m6WsQdhiC9Ir3ow=">AAACJ3icbVDLSgMxFM34tr6qLt0Ei6CbMiOCIiiCG5cK9gFtLZn0jhPMJENyRyxD/8aNv+JGUBFd+iem7Sx83dXhPJJ7T5hKYdH3P7yJyanpmdm5+dLC4tLySnl1rW51ZjjUuJbaNENmQQoFNRQooZkaYEkooRHenA71xi0YK7S6xH4KnYRdKxEJztBR3fLx3VW+LXYGtI1whzkVltpYG4yZ6tFIG4oxULgFhYd0QJuF+YgWsW654lf90dC/IChAhRRz3i0/t3uaZ4l7kEtmbSvwU+zkzKDgEgaldmYhZfyGXUPLQcUSsJ18dOeAbjlmvFWkFdIR+z2Rs8TafhI6Z8Iwtr+1Ifmf1sowOujkQqUZguLjj6JMUtR0WBrtCQMcZd8Bxo1wu1IeM8M4umpLroTg98l/QX23GvjV4GKvcnJQ1DFHNsgm2SYB2Scn5Iyckxrh5J48khfy6j14T96b9z62TnhFZp38GO/zC+s+pKY=</latexit>

✓ is shorthand for the event: ⇥ = ✓<latexit sha1_base64="oCuU1hhK/Az3jg80lcCu+mGiTT4=">AAACJ3icbVBNaxsxENWmaeu4H3HbYy5DTKEns1sKCYGUQC49umA7AdsYrTzrFdZKizQbYhb/m176V3oJNCEkx/yTyPYeEjvvosd7M6OZF+dKOgrD+2Dr1fbrN29rO/V37z983G18+txzprACu8IoY89j7lBJjV2SpPA8t8izWOFZPD1d+GcXaJ00ukOzHIcZn2iZSMHJS6PGT4ABpUjcP3hJJUgHLjWWUq7HkBgL3gW8QE1HAHMYdJbFx1XXqNEMW+ESsEmiijRZhfao8X8wNqLI/DyhuHP9KMxpWHJLUiic1weFw5yLKZ9g31PNM3TDcnnnHL56ZbVUYjTBUn3aUfLMuVkW+8qMU+rWvYX4ktcvKDkcllLnBaEWq4+SQgEZWIQGY2lRkJp5woWVflcQKbdckI+27kOI1k/eJL3vrShsRb9/NE8OqzhqbI/ts28sYgfshP1ibdZlgv1h/9g1uwn+BlfBbXC3Kt0Kqp4v7BmCh0cLVKTE</latexit><latexit sha1_base64="oCuU1hhK/Az3jg80lcCu+mGiTT4=">AAACJ3icbVBNaxsxENWmaeu4H3HbYy5DTKEns1sKCYGUQC49umA7AdsYrTzrFdZKizQbYhb/m176V3oJNCEkx/yTyPYeEjvvosd7M6OZF+dKOgrD+2Dr1fbrN29rO/V37z983G18+txzprACu8IoY89j7lBJjV2SpPA8t8izWOFZPD1d+GcXaJ00ukOzHIcZn2iZSMHJS6PGT4ABpUjcP3hJJUgHLjWWUq7HkBgL3gW8QE1HAHMYdJbFx1XXqNEMW+ESsEmiijRZhfao8X8wNqLI/DyhuHP9KMxpWHJLUiic1weFw5yLKZ9g31PNM3TDcnnnHL56ZbVUYjTBUn3aUfLMuVkW+8qMU+rWvYX4ktcvKDkcllLnBaEWq4+SQgEZWIQGY2lRkJp5woWVflcQKbdckI+27kOI1k/eJL3vrShsRb9/NE8OqzhqbI/ts28sYgfshP1ibdZlgv1h/9g1uwn+BlfBbXC3Kt0Kqp4v7BmCh0cLVKTE</latexit><latexit sha1_base64="oCuU1hhK/Az3jg80lcCu+mGiTT4=">AAACJ3icbVBNaxsxENWmaeu4H3HbYy5DTKEns1sKCYGUQC49umA7AdsYrTzrFdZKizQbYhb/m176V3oJNCEkx/yTyPYeEjvvosd7M6OZF+dKOgrD+2Dr1fbrN29rO/V37z983G18+txzprACu8IoY89j7lBJjV2SpPA8t8izWOFZPD1d+GcXaJ00ukOzHIcZn2iZSMHJS6PGT4ABpUjcP3hJJUgHLjWWUq7HkBgL3gW8QE1HAHMYdJbFx1XXqNEMW+ESsEmiijRZhfao8X8wNqLI/DyhuHP9KMxpWHJLUiic1weFw5yLKZ9g31PNM3TDcnnnHL56ZbVUYjTBUn3aUfLMuVkW+8qMU+rWvYX4ktcvKDkcllLnBaEWq4+SQgEZWIQGY2lRkJp5woWVflcQKbdckI+27kOI1k/eJL3vrShsRb9/NE8OqzhqbI/ts28sYgfshP1ibdZlgv1h/9g1uwn+BlfBbXC3Kt0Kqp4v7BmCh0cLVKTE</latexit><latexit sha1_base64="oCuU1hhK/Az3jg80lcCu+mGiTT4=">AAACJ3icbVBNaxsxENWmaeu4H3HbYy5DTKEns1sKCYGUQC49umA7AdsYrTzrFdZKizQbYhb/m176V3oJNCEkx/yTyPYeEjvvosd7M6OZF+dKOgrD+2Dr1fbrN29rO/V37z983G18+txzprACu8IoY89j7lBJjV2SpPA8t8izWOFZPD1d+GcXaJ00ukOzHIcZn2iZSMHJS6PGT4ABpUjcP3hJJUgHLjWWUq7HkBgL3gW8QE1HAHMYdJbFx1XXqNEMW+ESsEmiijRZhfao8X8wNqLI/DyhuHP9KMxpWHJLUiic1weFw5yLKZ9g31PNM3TDcnnnHL56ZbVUYjTBUn3aUfLMuVkW+8qMU+rWvYX4ktcvKDkcllLnBaEWq4+SQgEZWIQGY2lRkJp5woWVflcQKbdckI+27kOI1k/eJL3vrShsRb9/NE8OqzhqbI/ts28sYgfshP1ibdZlgv1h/9g1uwn+BlfBbXC3Kt0Kqp4v7BmCh0cLVKTE</latexit>


f(✓|x(1), . . . , x(n))<latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit>

Our shorthand notation

MAP, now with shorthand

MAP, without shorthand






f(x(1), . . . , x(n)|✓)

= argmax✓

⇣X

i

log f(x(i)|✓)⌘

<latexit sha1_base64="hWTJai6ZJ9txHDV1epW014CBypQ=">AAACuHiclVFdi9QwFE3r1zp+zeqjLxcHZQrL0C7KLoKwKIIPCis4uwuTsaZp2gmbJiW5lRlqf6Pgm//GdGaUddcXLwROzrkfJzdZraTDOP4ZhNeu37h5a+f24M7de/cfDHcfnjjTWC6m3ChjzzLmhJJaTFGiEme1FazKlDjNzt/0+ulXYZ00+hOuajGvWKllITlDT6XD73TBsKW4EMi6tP3w/m0Hz14BbXTuq8QfqaWmFpahsZpVomW2rNiy6wAoiiW20BXj5ed2nETdHtDcoNuD/q6jDr7BpgdElA7+p/fv1kBfy3IM1DVVKoEqU8JmmrzQPVpnRelwFE/idcBVkGzBiGzjOB3+8HZ5UwmNXDHnZklc49ybQMmV6Aa0caJm/JyVYuZh79DN2/XiO3jqmRwKY/3RCGv2YkXLKudWVeYzK4YLd1nryX9pswaLw3krdd2g0HwzqGgUoIH+FyGXVnBUKw8Yt9J7Bb5glnH0mx34JSSXn3wVnOxPkniSfHw+OjrcrmOHPCZPyJgk5IAckXfkmEwJD14EsyAPRPgy/BKWodykhsG25hH5K0L7Cx4816U=</latexit><latexit sha1_base64="hWTJai6ZJ9txHDV1epW014CBypQ=">AAACuHiclVFdi9QwFE3r1zp+zeqjLxcHZQrL0C7KLoKwKIIPCis4uwuTsaZp2gmbJiW5lRlqf6Pgm//GdGaUddcXLwROzrkfJzdZraTDOP4ZhNeu37h5a+f24M7de/cfDHcfnjjTWC6m3ChjzzLmhJJaTFGiEme1FazKlDjNzt/0+ulXYZ00+hOuajGvWKllITlDT6XD73TBsKW4EMi6tP3w/m0Hz14BbXTuq8QfqaWmFpahsZpVomW2rNiy6wAoiiW20BXj5ed2nETdHtDcoNuD/q6jDr7BpgdElA7+p/fv1kBfy3IM1DVVKoEqU8JmmrzQPVpnRelwFE/idcBVkGzBiGzjOB3+8HZ5UwmNXDHnZklc49ybQMmV6Aa0caJm/JyVYuZh79DN2/XiO3jqmRwKY/3RCGv2YkXLKudWVeYzK4YLd1nryX9pswaLw3krdd2g0HwzqGgUoIH+FyGXVnBUKw8Yt9J7Bb5glnH0mx34JSSXn3wVnOxPkniSfHw+OjrcrmOHPCZPyJgk5IAckXfkmEwJD14EsyAPRPgy/BKWodykhsG25hH5K0L7Cx4816U=</latexit><latexit sha1_base64="hWTJai6ZJ9txHDV1epW014CBypQ=">AAACuHiclVFdi9QwFE3r1zp+zeqjLxcHZQrL0C7KLoKwKIIPCis4uwuTsaZp2gmbJiW5lRlqf6Pgm//GdGaUddcXLwROzrkfJzdZraTDOP4ZhNeu37h5a+f24M7de/cfDHcfnjjTWC6m3ChjzzLmhJJaTFGiEme1FazKlDjNzt/0+ulXYZ00+hOuajGvWKllITlDT6XD73TBsKW4EMi6tP3w/m0Hz14BbXTuq8QfqaWmFpahsZpVomW2rNiy6wAoiiW20BXj5ed2nETdHtDcoNuD/q6jDr7BpgdElA7+p/fv1kBfy3IM1DVVKoEqU8JmmrzQPVpnRelwFE/idcBVkGzBiGzjOB3+8HZ5UwmNXDHnZklc49ybQMmV6Aa0caJm/JyVYuZh79DN2/XiO3jqmRwKY/3RCGv2YkXLKudWVeYzK4YLd1nryX9pswaLw3krdd2g0HwzqGgUoIH+FyGXVnBUKw8Yt9J7Bb5glnH0mx34JSSXn3wVnOxPkniSfHw+OjrcrmOHPCZPyJgk5IAckXfkmEwJD14EsyAPRPgy/BKWodykhsG25hH5K0L7Cx4816U=</latexit><latexit sha1_base64="hWTJai6ZJ9txHDV1epW014CBypQ=">AAACuHiclVFdi9QwFE3r1zp+zeqjLxcHZQrL0C7KLoKwKIIPCis4uwuTsaZp2gmbJiW5lRlqf6Pgm//GdGaUddcXLwROzrkfJzdZraTDOP4ZhNeu37h5a+f24M7de/cfDHcfnjjTWC6m3ChjzzLmhJJaTFGiEme1FazKlDjNzt/0+ulXYZ00+hOuajGvWKllITlDT6XD73TBsKW4EMi6tP3w/m0Hz14BbXTuq8QfqaWmFpahsZpVomW2rNiy6wAoiiW20BXj5ed2nETdHtDcoNuD/q6jDr7BpgdElA7+p/fv1kBfy3IM1DVVKoEqU8JmmrzQPVpnRelwFE/idcBVkGzBiGzjOB3+8HZ5UwmNXDHnZklc49ybQMmV6Aa0caJm/JyVYuZh79DN2/XiO3jqmRwKY/3RCGv2YkXLKudWVeYzK4YLd1nryX9pswaLw3krdd2g0HwzqGgUoIH+FyGXVnBUKw8Yt9J7Bb5glnH0mx34JSSXn3wVnOxPkniSfHw+OjrcrmOHPCZPyJgk5IAckXfkmEwJD14EsyAPRPgy/BKWodykhsG25hH5K0L7Cx4816U=</latexit>

MLE vs MAP


f(x(1), . . . , x(n)|✓)

= argmax✓

⇣X

i

log f(x(i)|✓)⌘



Data:

Most important slide of today

Maximum A Posteriori✓̂MAP = argmax

✓f(✓|x(1), . . . , x(n))

<latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit>

✓̂MAP =argmax✓

f(x(1), x(2), . . . , x(n)|✓)g(✓)h(x(1), x(2), . . . x(n))

<latexit sha1_base64="CB7kxzJ54tosqxTnIH+ZRBQ7was=">AAACoXicbVFdb9MwFHUyYKN8rMAjLxYRqJWmKilI2wvSEC/wgFQQ3SbFoXLcmyZa4kT2DbQy/in8MH4E/wGnjRCsXMnW8bn3XF8fp01ZaAzDn55/cOv2ncOju4N79x88PB4+enyh61YJmIu6rNVVyjWUhYQ5FljCVaOAV2kJl+n12y5/+RWULmr5GTcNJBVfySIrBEdHLYY/BiznaBjmgNwuzIc3M0tfv6CslUsngz8pw+oGFMdaSV6B4WpV8TW1ljKENRrqQKa4MNlozSA6oW6fnlC2rFFvD0Za+p3umo3patQja/J9QV8/tothEE7CbdB9EPUgIH3MFsNfroFoK5AoSq51HIUNJm5aLEQJdsBaDQ0X13wFsYPdU3RitjZa+twxS5rVyi2JdMv+rTC80npTpa6y4pjrm7mO/F8ubjE7S0whmxZBit1FWVtSrGn3J3RZKBBYbhzgQhVuVipy7txE9wUDJuGbqKuKy6VhYOMoMV/MKIjGtvMnuunGPriYTqJwEn18FZyf9U4dkafkGRmRiJySc/KOzMicCO/AG3tT76Uf+O/9mf9pV+p7veYJ+Sf8+Dd4Q82P</latexit><latexit sha1_base64="CB7kxzJ54tosqxTnIH+ZRBQ7was=">AAACoXicbVFdb9MwFHUyYKN8rMAjLxYRqJWmKilI2wvSEC/wgFQQ3SbFoXLcmyZa4kT2DbQy/in8MH4E/wGnjRCsXMnW8bn3XF8fp01ZaAzDn55/cOv2ncOju4N79x88PB4+enyh61YJmIu6rNVVyjWUhYQ5FljCVaOAV2kJl+n12y5/+RWULmr5GTcNJBVfySIrBEdHLYY/BiznaBjmgNwuzIc3M0tfv6CslUsngz8pw+oGFMdaSV6B4WpV8TW1ljKENRrqQKa4MNlozSA6oW6fnlC2rFFvD0Za+p3umo3patQja/J9QV8/tothEE7CbdB9EPUgIH3MFsNfroFoK5AoSq51HIUNJm5aLEQJdsBaDQ0X13wFsYPdU3RitjZa+twxS5rVyi2JdMv+rTC80npTpa6y4pjrm7mO/F8ubjE7S0whmxZBit1FWVtSrGn3J3RZKBBYbhzgQhVuVipy7txE9wUDJuGbqKuKy6VhYOMoMV/MKIjGtvMnuunGPriYTqJwEn18FZyf9U4dkafkGRmRiJySc/KOzMicCO/AG3tT76Uf+O/9mf9pV+p7veYJ+Sf8+Dd4Q82P</latexit><latexit sha1_base64="CB7kxzJ54tosqxTnIH+ZRBQ7was=">AAACoXicbVFdb9MwFHUyYKN8rMAjLxYRqJWmKilI2wvSEC/wgFQQ3SbFoXLcmyZa4kT2DbQy/in8MH4E/wGnjRCsXMnW8bn3XF8fp01ZaAzDn55/cOv2ncOju4N79x88PB4+enyh61YJmIu6rNVVyjWUhYQ5FljCVaOAV2kJl+n12y5/+RWULmr5GTcNJBVfySIrBEdHLYY/BiznaBjmgNwuzIc3M0tfv6CslUsngz8pw+oGFMdaSV6B4WpV8TW1ljKENRrqQKa4MNlozSA6oW6fnlC2rFFvD0Za+p3umo3patQja/J9QV8/tothEE7CbdB9EPUgIH3MFsNfroFoK5AoSq51HIUNJm5aLEQJdsBaDQ0X13wFsYPdU3RitjZa+twxS5rVyi2JdMv+rTC80npTpa6y4pjrm7mO/F8ubjE7S0whmxZBit1FWVtSrGn3J3RZKBBYbhzgQhVuVipy7txE9wUDJuGbqKuKy6VhYOMoMV/MKIjGtvMnuunGPriYTqJwEn18FZyf9U4dkafkGRmRiJySc/KOzMicCO/AG3tT76Uf+O/9mf9pV+p7veYJ+Sf8+Dd4Q82P</latexit><latexit sha1_base64="CB7kxzJ54tosqxTnIH+ZRBQ7was=">AAACoXicbVFdb9MwFHUyYKN8rMAjLxYRqJWmKilI2wvSEC/wgFQQ3SbFoXLcmyZa4kT2DbQy/in8MH4E/wGnjRCsXMnW8bn3XF8fp01ZaAzDn55/cOv2ncOju4N79x88PB4+enyh61YJmIu6rNVVyjWUhYQ5FljCVaOAV2kJl+n12y5/+RWULmr5GTcNJBVfySIrBEdHLYY/BiznaBjmgNwuzIc3M0tfv6CslUsngz8pw+oGFMdaSV6B4WpV8TW1ljKENRrqQKa4MNlozSA6oW6fnlC2rFFvD0Za+p3umo3patQja/J9QV8/tothEE7CbdB9EPUgIH3MFsNfroFoK5AoSq51HIUNJm5aLEQJdsBaDQ0X13wFsYPdU3RitjZa+twxS5rVyi2JdMv+rTC80npTpa6y4pjrm7mO/F8ubjE7S0whmxZBit1FWVtSrGn3J3RZKBBYbhzgQhVuVipy7txE9wUDJuGbqKuKy6VhYOMoMV/MKIjGtvMnuunGPriYTqJwEn18FZyf9U4dkafkGRmRiJySc/KOzMicCO/AG3tT76Uf+O/9mf9pV+p7veYJ+Sf8+Dd4Q82P</latexit>


data:

prior

likelihood

posterior

Maximum A Posteriori✓̂MAP = argmax

✓f(✓|x(1), . . . , x(n))

<latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit><latexit sha1_base64="5TYPk9XPUUQ9E5+lWmRCj7zgILk=">AAACVXicbVBNaxsxENVu0iR1P+Kmx15ETYsNweyGQnMpJPTSS8GFOgl4XTMrz9oiWmmRZoONsn8yl9J/0kuh8gelTTogeHpvnmb08kpJR0nyI4p3dh/t7R88bj15+uz5YfvF0YUztRU4FEYZe5WDQyU1DkmSwqvKIpS5wsv8+uNKv7xB66TRX2lZ4biEmZaFFECBmrRVNgfyGc2RoJn4z+eDhr/9wLNaT4ML/0g+MxVaIGM1lOjBzkpYNA3nGeGCPG+K7qaT3/LFN99Ne80xz6aG3PH6rnsN703anaSfrIs/BOkWdNi2BpP2XXhC1CVqEgqcG6VJReMwnaRQ2LSy2mEF4hpmOApwtZob+3UqDX8TmCkvjA1HE1+zfzs8lM4tyzx0lkBzd19bkf/TRjUVp2MvdVUTarEZVNSKk+GriPlUWhSklgGAsDLsysUcLAgKkbZCCOn9Lz8EFyf9NOmnX951zk63cRywV+w167KUvWdn7BMbsCET7I79jKIojr5Hv+LdeG/TGkdbz0v2T8WHvwHtFLPL</latexit>


data:

=argmax✓

g(✓)Qn

i=1 f(x(i)|✓)

h(x(1), x(2), . . . , x(n))<latexit sha1_base64="MaqkCy/6+1kI4WPUFvEIjtGkWf4=">AAACiXicbVBda9swFJW9ry7d1nR7W1/EwkYCJVhlsDAoFPqyxw6WthC7QZGvE1FbMtL1lqDpcT9yv2HsP0xO/LC1uyBxdM650tVZ1KW0mCQ/o/jBw0ePn+w97e0/e/7ioH/48tLqxgiYCl1qc73gFkqpYIoSS7iuDfBqUcLV4va81a++grFSqy+4qSGr+FLJQgqOgZr3f/RO39G0UXnwALoUV4Dcu1TXYDhqo3gFjptlxdfUe5oirNHRAArDhVsOdw2jtDY6nzt5yvyNosVwnYKTnn6nne7dquXYMQ37yTFNc412e3DKj/y8P0jGybbofcA6MCBdXcz7v8MNoqlAoSi5tTOW1JiFQVGKEnwvbSzUXNzyJcwCbH9hM7eNy9O3gclpoU1YCumW/bvD8craTbUIzorjyt7VWvJ/2qzBYpI5qeoGQYndQ0VTUtS0zZ7m0oDAchMAF0aGWalY8RAkhvR7qYJvQlcVV7lLwc9Y5m7ccMBGvs2H3U3jPrg8GbNkzD6/H5xNuqT2yBF5Q4aEkQ/kjHwiF2RKBPkVHUavo6N4P2bxJP64s8ZR1/OK/FPx+R9LzcWp</latexit><latexit sha1_base64="MaqkCy/6+1kI4WPUFvEIjtGkWf4=">AAACiXicbVBda9swFJW9ry7d1nR7W1/EwkYCJVhlsDAoFPqyxw6WthC7QZGvE1FbMtL1lqDpcT9yv2HsP0xO/LC1uyBxdM650tVZ1KW0mCQ/o/jBw0ePn+w97e0/e/7ioH/48tLqxgiYCl1qc73gFkqpYIoSS7iuDfBqUcLV4va81a++grFSqy+4qSGr+FLJQgqOgZr3f/RO39G0UXnwALoUV4Dcu1TXYDhqo3gFjptlxdfUe5oirNHRAArDhVsOdw2jtDY6nzt5yvyNosVwnYKTnn6nne7dquXYMQ37yTFNc412e3DKj/y8P0jGybbofcA6MCBdXcz7v8MNoqlAoSi5tTOW1JiFQVGKEnwvbSzUXNzyJcwCbH9hM7eNy9O3gclpoU1YCumW/bvD8craTbUIzorjyt7VWvJ/2qzBYpI5qeoGQYndQ0VTUtS0zZ7m0oDAchMAF0aGWalY8RAkhvR7qYJvQlcVV7lLwc9Y5m7ccMBGvs2H3U3jPrg8GbNkzD6/H5xNuqT2yBF5Q4aEkQ/kjHwiF2RKBPkVHUavo6N4P2bxJP64s8ZR1/OK/FPx+R9LzcWp</latexit><latexit sha1_base64="MaqkCy/6+1kI4WPUFvEIjtGkWf4=">AAACiXicbVBda9swFJW9ry7d1nR7W1/EwkYCJVhlsDAoFPqyxw6WthC7QZGvE1FbMtL1lqDpcT9yv2HsP0xO/LC1uyBxdM650tVZ1KW0mCQ/o/jBw0ePn+w97e0/e/7ioH/48tLqxgiYCl1qc73gFkqpYIoSS7iuDfBqUcLV4va81a++grFSqy+4qSGr+FLJQgqOgZr3f/RO39G0UXnwALoUV4Dcu1TXYDhqo3gFjptlxdfUe5oirNHRAArDhVsOdw2jtDY6nzt5yvyNosVwnYKTnn6nne7dquXYMQ37yTFNc412e3DKj/y8P0jGybbofcA6MCBdXcz7v8MNoqlAoSi5tTOW1JiFQVGKEnwvbSzUXNzyJcwCbH9hM7eNy9O3gclpoU1YCumW/bvD8craTbUIzorjyt7VWvJ/2qzBYpI5qeoGQYndQ0VTUtS0zZ7m0oDAchMAF0aGWalY8RAkhvR7qYJvQlcVV7lLwc9Y5m7ccMBGvs2H3U3jPrg8GbNkzD6/H5xNuqT2yBF5Q4aEkQ/kjHwiF2RKBPkVHUavo6N4P2bxJP64s8ZR1/OK/FPx+R9LzcWp</latexit><latexit sha1_base64="MaqkCy/6+1kI4WPUFvEIjtGkWf4=">AAACiXicbVBda9swFJW9ry7d1nR7W1/EwkYCJVhlsDAoFPqyxw6WthC7QZGvE1FbMtL1lqDpcT9yv2HsP0xO/LC1uyBxdM650tVZ1KW0mCQ/o/jBw0ePn+w97e0/e/7ioH/48tLqxgiYCl1qc73gFkqpYIoSS7iuDfBqUcLV4va81a++grFSqy+4qSGr+FLJQgqOgZr3f/RO39G0UXnwALoUV4Dcu1TXYDhqo3gFjptlxdfUe5oirNHRAArDhVsOdw2jtDY6nzt5yvyNosVwnYKTnn6nne7dquXYMQ37yTFNc412e3DKj/y8P0jGybbofcA6MCBdXcz7v8MNoqlAoSi5tTOW1JiFQVGKEnwvbSzUXNzyJcwCbH9hM7eNy9O3gclpoU1YCumW/bvD8craTbUIzorjyt7VWvJ/2qzBYpI5qeoGQYndQ0VTUtS0zZ7m0oDAchMAF0aGWalY8RAkhvR7qYJvQlcVV7lLwc9Y5m7ccMBGvs2H3U3jPrg8GbNkzD6/H5xNuqT2yBF5Q4aEkQ/kjHwiF2RKBPkVHUavo6N4P2bxJP64s8ZR1/OK/FPx+R9LzcWp</latexit>

✓̂MAP =argmax✓

g(✓)f(x(1), x(2), . . . , x(n)|✓)h(x(1), x(2), . . . x(n))

<latexit sha1_base64="4+GHILtFrJgDHdnJ7WGPtwAW+Pc=">AAAConicbVFdb9MwFHXCgFG+CjzyYlFALZqqZExiL0hDvCAkpE7QbVIdKse9aawlTmTfQCvjv8L/4kfwH3DagGDlSraO7z3Hvj43rQtpMIp+BOG1ves3bu7f6t2+c/fe/f6Dh2emarSAqaiKSl+k3EAhFUxRYgEXtQZepgWcp5dv2/r5F9BGVuoTrmtISr5UMpOCo0/N+997LOdoGeaA3M3thzcTR18/p6xRCy+DPyXLqho0x0orXoLlelnyFXWOMoQVWupBprmwy+FWMKLZcMUgPqB+PzygbFGh2RyscvQb/c1yNt/ldbSRm/cH0TjaBN0FcQcGpIvJvP/TXyCaEhSKghszi6MaE98uSlGA67HGQM3FJV/CzMP2LyaxGx8dfeYzC5pV2i+FdJP9W2F5acy6TD2z5Jibq7U2+b/arMHsOLFS1Q2CEtuHsqagWNF2KHQhNQgs1h5woaXvlYqcezvRz6DHFHwVVVlytbAM3CxO7Gc7HMQj1/oTX3VjF5wdjuNoHJ8eDU6OO6f2yWPyhAxJTF6RE/KOTMiUiGAveBG8DI7Cp+H78DT8uKWGQad5RP6JkP0CBV3NuQ==</latexit><latexit sha1_base64="4+GHILtFrJgDHdnJ7WGPtwAW+Pc=">AAAConicbVFdb9MwFHXCgFG+CjzyYlFALZqqZExiL0hDvCAkpE7QbVIdKse9aawlTmTfQCvjv8L/4kfwH3DagGDlSraO7z3Hvj43rQtpMIp+BOG1ves3bu7f6t2+c/fe/f6Dh2emarSAqaiKSl+k3EAhFUxRYgEXtQZepgWcp5dv2/r5F9BGVuoTrmtISr5UMpOCo0/N+997LOdoGeaA3M3thzcTR18/p6xRCy+DPyXLqho0x0orXoLlelnyFXWOMoQVWupBprmwy+FWMKLZcMUgPqB+PzygbFGh2RyscvQb/c1yNt/ldbSRm/cH0TjaBN0FcQcGpIvJvP/TXyCaEhSKghszi6MaE98uSlGA67HGQM3FJV/CzMP2LyaxGx8dfeYzC5pV2i+FdJP9W2F5acy6TD2z5Jibq7U2+b/arMHsOLFS1Q2CEtuHsqagWNF2KHQhNQgs1h5woaXvlYqcezvRz6DHFHwVVVlytbAM3CxO7Gc7HMQj1/oTX3VjF5wdjuNoHJ8eDU6OO6f2yWPyhAxJTF6RE/KOTMiUiGAveBG8DI7Cp+H78DT8uKWGQad5RP6JkP0CBV3NuQ==</latexit><latexit sha1_base64="4+GHILtFrJgDHdnJ7WGPtwAW+Pc=">AAAConicbVFdb9MwFHXCgFG+CjzyYlFALZqqZExiL0hDvCAkpE7QbVIdKse9aawlTmTfQCvjv8L/4kfwH3DagGDlSraO7z3Hvj43rQtpMIp+BOG1ves3bu7f6t2+c/fe/f6Dh2emarSAqaiKSl+k3EAhFUxRYgEXtQZepgWcp5dv2/r5F9BGVuoTrmtISr5UMpOCo0/N+997LOdoGeaA3M3thzcTR18/p6xRCy+DPyXLqho0x0orXoLlelnyFXWOMoQVWupBprmwy+FWMKLZcMUgPqB+PzygbFGh2RyscvQb/c1yNt/ldbSRm/cH0TjaBN0FcQcGpIvJvP/TXyCaEhSKghszi6MaE98uSlGA67HGQM3FJV/CzMP2LyaxGx8dfeYzC5pV2i+FdJP9W2F5acy6TD2z5Jibq7U2+b/arMHsOLFS1Q2CEtuHsqagWNF2KHQhNQgs1h5woaXvlYqcezvRz6DHFHwVVVlytbAM3CxO7Gc7HMQj1/oTX3VjF5wdjuNoHJ8eDU6OO6f2yWPyhAxJTF6RE/KOTMiUiGAveBG8DI7Cp+H78DT8uKWGQad5RP6JkP0CBV3NuQ==</latexit><latexit sha1_base64="4+GHILtFrJgDHdnJ7WGPtwAW+Pc=">AAAConicbVFdb9MwFHXCgFG+CjzyYlFALZqqZExiL0hDvCAkpE7QbVIdKse9aawlTmTfQCvjv8L/4kfwH3DagGDlSraO7z3Hvj43rQtpMIp+BOG1ves3bu7f6t2+c/fe/f6Dh2emarSAqaiKSl+k3EAhFUxRYgEXtQZepgWcp5dv2/r5F9BGVuoTrmtISr5UMpOCo0/N+997LOdoGeaA3M3thzcTR18/p6xRCy+DPyXLqho0x0orXoLlelnyFXWOMoQVWupBprmwy+FWMKLZcMUgPqB+PzygbFGh2RyscvQb/c1yNt/ldbSRm/cH0TjaBN0FcQcGpIvJvP/TXyCaEhSKghszi6MaE98uSlGA67HGQM3FJV/CzMP2LyaxGx8dfeYzC5pV2i+FdJP9W2F5acy6TD2z5Jibq7U2+b/arMHsOLFS1Q2CEtuHsqagWNF2KHQhNQgs1h5woaXvlYqcezvRz6DHFHwVVVlytbAM3CxO7Gc7HMQj1/oTX3VjF5wdjuNoHJ8eDU6OO6f2yWPyhAxJTF6RE/KOTMiUiGAveBG8DI7Cp+H78DT8uKWGQad5RP6JkP0CBV3NuQ==</latexit>

=argmax✓

g(✓)nY

i=1

f(x(i)|✓)<latexit sha1_base64="zcVa1VKAdZS39AzYjQ2c7bKKZ/w=">AAACZ3icbVBLaxsxGJS3r9R9xG2hFHpRa1qci9ktgeQSCPTSYwp1ErA2Rqv91hZZSYv0bWKj6gf2J/Qn9FB6TW6VH4U26YBgmJkPDVM0tXSYpt87yZ279+4/2HrYffT4ydPt3rPnx860VsBImNrY04I7qKWGEUqs4bSxwFVRw0lx/nHpn1yAddLoL7hoIFd8qmUlBccoTXri4D1lrS5jBNAznAHy4JlpwHI0VnMFntup4nMaAmUIc/Q00OlgHd2hrLGmnHhJD2gWzjStBnMGXgb6lf6JTHr9dJiuQG+TbEP6ZIOjSe8nK41oFWgUNXdunKUN5rEHSlFD6LLWQcPFOZ/CONJlSZf71RiBvotKSStj49NIV+rfF54r5xaqiEnFceZuekvxf964xWo/91I3LYIW64+qtqZo6HJZWkoLAutFJFxYGbtSMeOWC4zjdpmGS2GU4rr0DMI4y/2ZH/SznRDiPtnNNW6T4w/DLB1mn3f7h/ubpbbIa/KWDEhG9sgh+USOyIgI8o38IlfkuvMj2U5eJq/W0aSzuXlB/kHy5jfF8ru4</latexit><latexit sha1_base64="zcVa1VKAdZS39AzYjQ2c7bKKZ/w=">AAACZ3icbVBLaxsxGJS3r9R9xG2hFHpRa1qci9ktgeQSCPTSYwp1ErA2Rqv91hZZSYv0bWKj6gf2J/Qn9FB6TW6VH4U26YBgmJkPDVM0tXSYpt87yZ279+4/2HrYffT4ydPt3rPnx860VsBImNrY04I7qKWGEUqs4bSxwFVRw0lx/nHpn1yAddLoL7hoIFd8qmUlBccoTXri4D1lrS5jBNAznAHy4JlpwHI0VnMFntup4nMaAmUIc/Q00OlgHd2hrLGmnHhJD2gWzjStBnMGXgb6lf6JTHr9dJiuQG+TbEP6ZIOjSe8nK41oFWgUNXdunKUN5rEHSlFD6LLWQcPFOZ/CONJlSZf71RiBvotKSStj49NIV+rfF54r5xaqiEnFceZuekvxf964xWo/91I3LYIW64+qtqZo6HJZWkoLAutFJFxYGbtSMeOWC4zjdpmGS2GU4rr0DMI4y/2ZH/SznRDiPtnNNW6T4w/DLB1mn3f7h/ubpbbIa/KWDEhG9sgh+USOyIgI8o38IlfkuvMj2U5eJq/W0aSzuXlB/kHy5jfF8ru4</latexit><latexit sha1_base64="zcVa1VKAdZS39AzYjQ2c7bKKZ/w=">AAACZ3icbVBLaxsxGJS3r9R9xG2hFHpRa1qci9ktgeQSCPTSYwp1ErA2Rqv91hZZSYv0bWKj6gf2J/Qn9FB6TW6VH4U26YBgmJkPDVM0tXSYpt87yZ279+4/2HrYffT4ydPt3rPnx860VsBImNrY04I7qKWGEUqs4bSxwFVRw0lx/nHpn1yAddLoL7hoIFd8qmUlBccoTXri4D1lrS5jBNAznAHy4JlpwHI0VnMFntup4nMaAmUIc/Q00OlgHd2hrLGmnHhJD2gWzjStBnMGXgb6lf6JTHr9dJiuQG+TbEP6ZIOjSe8nK41oFWgUNXdunKUN5rEHSlFD6LLWQcPFOZ/CONJlSZf71RiBvotKSStj49NIV+rfF54r5xaqiEnFceZuekvxf964xWo/91I3LYIW64+qtqZo6HJZWkoLAutFJFxYGbtSMeOWC4zjdpmGS2GU4rr0DMI4y/2ZH/SznRDiPtnNNW6T4w/DLB1mn3f7h/ubpbbIa/KWDEhG9sgh+USOyIgI8o38IlfkuvMj2U5eJq/W0aSzuXlB/kHy5jfF8ru4</latexit><latexit sha1_base64="zcVa1VKAdZS39AzYjQ2c7bKKZ/w=">AAACZ3icbVBLaxsxGJS3r9R9xG2hFHpRa1qci9ktgeQSCPTSYwp1ErA2Rqv91hZZSYv0bWKj6gf2J/Qn9FB6TW6VH4U26YBgmJkPDVM0tXSYpt87yZ279+4/2HrYffT4ydPt3rPnx860VsBImNrY04I7qKWGEUqs4bSxwFVRw0lx/nHpn1yAddLoL7hoIFd8qmUlBccoTXri4D1lrS5jBNAznAHy4JlpwHI0VnMFntup4nMaAmUIc/Q00OlgHd2hrLGmnHhJD2gWzjStBnMGXgb6lf6JTHr9dJiuQG+TbEP6ZIOjSe8nK41oFWgUNXdunKUN5rEHSlFD6LLWQcPFOZ/CONJlSZf71RiBvotKSStj49NIV+rfF54r5xaqiEnFceZuekvxf964xWo/91I3LYIW64+qtqZo6HJZWkoLAutFJFxYGbtSMeOWC4zjdpmGS2GU4rr0DMI4y/2ZH/SznRDiPtnNNW6T4w/DLB1mn3f7h/ubpbbIa/KWDEhG9sgh+USOyIgI8o38IlfkuvMj2U5eJq/W0aSzuXlB/kHy5jfF8ru4</latexit>

=argmax✓

log(g(✓)) +

nX

i=1

log(f(x(i)|✓))!

<latexit sha1_base64="5+ZrdjVIbPglVta+3XK5c+xnamI=">AAACgXicbVBbaxQxFM5OvbTrpWv76EtwUWYQlpkiWJBCwRcfK7htYbNdMtkzs6GTZEjO6C4xP9Rnn/0PZi+Ith4I+fgunMNXto10mOc/esneg4ePHu8f9J88ffb8cPDi6NKZzgoYC9MYe11yB43UMEaJDVy3FrgqG7gqbz+u9auvYJ00+guuWpgqXmtZScExUrNBOHtDWafn0QLoGS4AefDMtGA5Gqu5As9trfiShkAZwhI9jaCBCtP4mZqmdbrNZRl9S5nr1MzLsyLc6I2eVumSgZeBfqd/fMzKeoHZbDDMR/lm6H1Q7MCQ7OZiNvjF5kZ0CjSKhjs3KfIWp/FClKKB0Gedg5aLW17DJML1+W7qNzUF+joyc1oZG59GumH/TniunFupMjoVx4W7q63J/2mTDqvTqZe67RC02C6quoaioevO6VxaENisIuDCyngrFQtuucBYe59p+CaMUlzPPYMwKab+xqfDIgsh9lPcbeM+uDwZFfmo+PxueH66a2qfvCSvSEoK8p6ck0/kgoyJID97B72j3nGyl2RJnpxsrUlvlzkm/0zy4Tf7N8Fb</latexit><latexit sha1_base64="5+ZrdjVIbPglVta+3XK5c+xnamI=">AAACgXicbVBbaxQxFM5OvbTrpWv76EtwUWYQlpkiWJBCwRcfK7htYbNdMtkzs6GTZEjO6C4xP9Rnn/0PZi+Ith4I+fgunMNXto10mOc/esneg4ePHu8f9J88ffb8cPDi6NKZzgoYC9MYe11yB43UMEaJDVy3FrgqG7gqbz+u9auvYJ00+guuWpgqXmtZScExUrNBOHtDWafn0QLoGS4AefDMtGA5Gqu5As9trfiShkAZwhI9jaCBCtP4mZqmdbrNZRl9S5nr1MzLsyLc6I2eVumSgZeBfqd/fMzKeoHZbDDMR/lm6H1Q7MCQ7OZiNvjF5kZ0CjSKhjs3KfIWp/FClKKB0Gedg5aLW17DJML1+W7qNzUF+joyc1oZG59GumH/TniunFupMjoVx4W7q63J/2mTDqvTqZe67RC02C6quoaioevO6VxaENisIuDCyngrFQtuucBYe59p+CaMUlzPPYMwKab+xqfDIgsh9lPcbeM+uDwZFfmo+PxueH66a2qfvCSvSEoK8p6ck0/kgoyJID97B72j3nGyl2RJnpxsrUlvlzkm/0zy4Tf7N8Fb</latexit><latexit sha1_base64="5+ZrdjVIbPglVta+3XK5c+xnamI=">AAACgXicbVBbaxQxFM5OvbTrpWv76EtwUWYQlpkiWJBCwRcfK7htYbNdMtkzs6GTZEjO6C4xP9Rnn/0PZi+Ith4I+fgunMNXto10mOc/esneg4ePHu8f9J88ffb8cPDi6NKZzgoYC9MYe11yB43UMEaJDVy3FrgqG7gqbz+u9auvYJ00+guuWpgqXmtZScExUrNBOHtDWafn0QLoGS4AefDMtGA5Gqu5As9trfiShkAZwhI9jaCBCtP4mZqmdbrNZRl9S5nr1MzLsyLc6I2eVumSgZeBfqd/fMzKeoHZbDDMR/lm6H1Q7MCQ7OZiNvjF5kZ0CjSKhjs3KfIWp/FClKKB0Gedg5aLW17DJML1+W7qNzUF+joyc1oZG59GumH/TniunFupMjoVx4W7q63J/2mTDqvTqZe67RC02C6quoaioevO6VxaENisIuDCyngrFQtuucBYe59p+CaMUlzPPYMwKab+xqfDIgsh9lPcbeM+uDwZFfmo+PxueH66a2qfvCSvSEoK8p6ck0/kgoyJID97B72j3nGyl2RJnpxsrUlvlzkm/0zy4Tf7N8Fb</latexit><latexit sha1_base64="5+ZrdjVIbPglVta+3XK5c+xnamI=">AAACgXicbVBbaxQxFM5OvbTrpWv76EtwUWYQlpkiWJBCwRcfK7htYbNdMtkzs6GTZEjO6C4xP9Rnn/0PZi+Ith4I+fgunMNXto10mOc/esneg4ePHu8f9J88ffb8cPDi6NKZzgoYC9MYe11yB43UMEaJDVy3FrgqG7gqbz+u9auvYJ00+guuWpgqXmtZScExUrNBOHtDWafn0QLoGS4AefDMtGA5Gqu5As9trfiShkAZwhI9jaCBCtP4mZqmdbrNZRl9S5nr1MzLsyLc6I2eVumSgZeBfqd/fMzKeoHZbDDMR/lm6H1Q7MCQ7OZiNvjF5kZ0CjSKhjs3KfIWp/FClKKB0Gedg5aLW17DJML1+W7qNzUF+joyc1oZG59GumH/TniunFupMjoVx4W7q63J/2mTDqvTqZe67RC02C6quoaioevO6VxaENisIuDCyngrFQtuucBYe59p+CaMUlzPPYMwKab+xqfDIgsh9lPcbeM+uDwZFfmo+PxueH66a2qfvCSvSEoK8p6ck0/kgoyJID97B72j3nGyl2RJnpxsrUlvlzkm/0zy4Tf7N8Fb</latexit>

monotonic

Fury K


Estimatedparameter

Sum of log likelihood

Log prior

Chose the value of thetathat maximizes:

✓̂MAP =argmax✓

log(g(✓)) +

nX

i=1

log(f(x(i)|✓))!

<latexit sha1_base64="rH5tZudDPhfCXQEQFM6M0YqTyCI=">AAAClHicbVFdixMxFE3Hr7V+bFXwRR+CZZcpQpkRwX1wYWURfFEq2N2FplvS9M407CQZkjvaEvMf/Hv+CF99Nv1g0V0vhBzOuYccTqZ1JR1m2c9WcuPmrdt3du62791/8HC38+jxiTONFTAUpjL2bModVFLDECVWcFZb4Gpawen04niln34F66TRX3BZw1jxUstCCo6RmnR+sDlHz3AOyMPEf3w3CPRwn7JGz6ILLiXPTA2Wo7GaK/DcloovaAiUISzQ0wgqKDCNlylpWqYbX69HX1LmGjXx8jAP53qtp0W6YOBloN/p5R6zspxjb9LpZv1sPfQ6yLegS7YzmHR+sZkRjQKNouLOjfKsxnFMiFJUENqscVBzccFLGEW4iu/Gft1coHuRmdHC2Hg00jX7t8Nz5dxSTeOm4jh3V7UV+T9t1GBxMPZS1w2CFpuHiqaiaOjqG+hMWhBYLSPgwsqYlYo5t1xgrL3NNHwTRimuZ55BGOVjf+7Tbt4LIfaTX23jOjh51c+zfv75dffoYNvUDnlGXpCU5OQNOSIfyIAMiSC/W89be6395GnyNjlO3m9Wk9bW84T8M8mnP5SkyXE=</latexit><latexit sha1_base64="rH5tZudDPhfCXQEQFM6M0YqTyCI=">AAAClHicbVFdixMxFE3Hr7V+bFXwRR+CZZcpQpkRwX1wYWURfFEq2N2FplvS9M407CQZkjvaEvMf/Hv+CF99Nv1g0V0vhBzOuYccTqZ1JR1m2c9WcuPmrdt3du62791/8HC38+jxiTONFTAUpjL2bModVFLDECVWcFZb4Gpawen04niln34F66TRX3BZw1jxUstCCo6RmnR+sDlHz3AOyMPEf3w3CPRwn7JGz6ILLiXPTA2Wo7GaK/DcloovaAiUISzQ0wgqKDCNlylpWqYbX69HX1LmGjXx8jAP53qtp0W6YOBloN/p5R6zspxjb9LpZv1sPfQ6yLegS7YzmHR+sZkRjQKNouLOjfKsxnFMiFJUENqscVBzccFLGEW4iu/Gft1coHuRmdHC2Hg00jX7t8Nz5dxSTeOm4jh3V7UV+T9t1GBxMPZS1w2CFpuHiqaiaOjqG+hMWhBYLSPgwsqYlYo5t1xgrL3NNHwTRimuZ55BGOVjf+7Tbt4LIfaTX23jOjh51c+zfv75dffoYNvUDnlGXpCU5OQNOSIfyIAMiSC/W89be6395GnyNjlO3m9Wk9bW84T8M8mnP5SkyXE=</latexit><latexit sha1_base64="rH5tZudDPhfCXQEQFM6M0YqTyCI=">AAAClHicbVFdixMxFE3Hr7V+bFXwRR+CZZcpQpkRwX1wYWURfFEq2N2FplvS9M407CQZkjvaEvMf/Hv+CF99Nv1g0V0vhBzOuYccTqZ1JR1m2c9WcuPmrdt3du62791/8HC38+jxiTONFTAUpjL2bModVFLDECVWcFZb4Gpawen04niln34F66TRX3BZw1jxUstCCo6RmnR+sDlHz3AOyMPEf3w3CPRwn7JGz6ILLiXPTA2Wo7GaK/DcloovaAiUISzQ0wgqKDCNlylpWqYbX69HX1LmGjXx8jAP53qtp0W6YOBloN/p5R6zspxjb9LpZv1sPfQ6yLegS7YzmHR+sZkRjQKNouLOjfKsxnFMiFJUENqscVBzccFLGEW4iu/Gft1coHuRmdHC2Hg00jX7t8Nz5dxSTeOm4jh3V7UV+T9t1GBxMPZS1w2CFpuHiqaiaOjqG+hMWhBYLSPgwsqYlYo5t1xgrL3NNHwTRimuZ55BGOVjf+7Tbt4LIfaTX23jOjh51c+zfv75dffoYNvUDnlGXpCU5OQNOSIfyIAMiSC/W89be6395GnyNjlO3m9Wk9bW84T8M8mnP5SkyXE=</latexit><latexit sha1_base64="rH5tZudDPhfCXQEQFM6M0YqTyCI=">AAAClHicbVFdixMxFE3Hr7V+bFXwRR+CZZcpQpkRwX1wYWURfFEq2N2FplvS9M407CQZkjvaEvMf/Hv+CF99Nv1g0V0vhBzOuYccTqZ1JR1m2c9WcuPmrdt3du62791/8HC38+jxiTONFTAUpjL2bModVFLDECVWcFZb4Gpawen04niln34F66TRX3BZw1jxUstCCo6RmnR+sDlHz3AOyMPEf3w3CPRwn7JGz6ILLiXPTA2Wo7GaK/DcloovaAiUISzQ0wgqKDCNlylpWqYbX69HX1LmGjXx8jAP53qtp0W6YOBloN/p5R6zspxjb9LpZv1sPfQ6yLegS7YzmHR+sZkRjQKNouLOjfKsxnFMiFJUENqscVBzccFLGEW4iu/Gft1coHuRmdHC2Hg00jX7t8Nz5dxSTeOm4jh3V7UV+T9t1GBxMPZS1w2CFpuHiqaiaOjqG+hMWhBYLSPgwsqYlYo5t1xgrL3NNHwTRimuZ55BGOVjf+7Tbt4LIfaTX23jOjh51c+zfv75dffoYNvUDnlGXpCU5OQNOSIfyIAMiSC/W89be6395GnyNjlO3m9Wk9bW84T8M8mnP5SkyXE=</latexit>






f(x(1), . . . , x(n)|✓)

= argmax✓

⇣X

i

log f(x(i)|✓)⌘


MLE vs MAP


f(x(1), . . . , x(n)|✓)

= argmax✓

⇣X

i

log f(x(i)|✓)⌘



Data:

= argmax✓

log(g(✓)) +

nX

i=1

log(f(x(i)|✓))!

<latexit sha1_base64="2Vyh+F2J1WNRU3hIn/gc2jRPn/U=">AAACgHicbVBba9swFJa9W5bd0u5xL2Jh4DDI7DJYGRQKe9ljB0tbiNwgy8eOqCUZ6XhLUP1D97r3/YcpF8bW7oDQx3fhHL6ibaTDNP0RxffuP3j4aPB4+OTps+cvRgeH5850VsBMmMbYy4I7aKSGGUps4LK1wFXRwEVx/WmjX3wD66TRX3HdQq54rWUlBcdALUY3J5R1ugwOQM9wCch7z0wLlqOxmivw3NaKr2jfU4awQk8DaKDCJHympkmd7HKTCX1LmevUwsuTrL/SWz2pkhUDL3t6Q//4mJX1EieL0Tidptuhd0G2B2Oyn7PF6BcrjegUaBQNd26epS3m4UKUooF+yDoHLRfXvIZ5gJvzXe63LfX0TWBKWhkbnka6Zf9OeK6cW6siOBXHpbutbcj/afMOq+PcS912CFrsFlVdQ9HQTeW0lBYENusAuLAy3ErFklsuMNQ+ZBq+C6MU16Vn0M+z3F/5ZJxN+j70k91u4y44P5pm6TT78n58erxvakBekdckIRn5QE7JZ3JGZkSQn9EgOogO4zhO4ndxtrPG0T7zkvwz8cffhQrBKw==</latexit><latexit sha1_base64="2Vyh+F2J1WNRU3hIn/gc2jRPn/U=">AAACgHicbVBba9swFJa9W5bd0u5xL2Jh4DDI7DJYGRQKe9ljB0tbiNwgy8eOqCUZ6XhLUP1D97r3/YcpF8bW7oDQx3fhHL6ibaTDNP0RxffuP3j4aPB4+OTps+cvRgeH5850VsBMmMbYy4I7aKSGGUps4LK1wFXRwEVx/WmjX3wD66TRX3HdQq54rWUlBcdALUY3J5R1ugwOQM9wCch7z0wLlqOxmivw3NaKr2jfU4awQk8DaKDCJHympkmd7HKTCX1LmevUwsuTrL/SWz2pkhUDL3t6Q//4mJX1EieL0Tidptuhd0G2B2Oyn7PF6BcrjegUaBQNd26epS3m4UKUooF+yDoHLRfXvIZ5gJvzXe63LfX0TWBKWhkbnka6Zf9OeK6cW6siOBXHpbutbcj/afMOq+PcS912CFrsFlVdQ9HQTeW0lBYENusAuLAy3ErFklsuMNQ+ZBq+C6MU16Vn0M+z3F/5ZJxN+j70k91u4y44P5pm6TT78n58erxvakBekdckIRn5QE7JZ3JGZkSQn9EgOogO4zhO4ndxtrPG0T7zkvwz8cffhQrBKw==</latexit><latexit sha1_base64="2Vyh+F2J1WNRU3hIn/gc2jRPn/U=">AAACgHicbVBba9swFJa9W5bd0u5xL2Jh4DDI7DJYGRQKe9ljB0tbiNwgy8eOqCUZ6XhLUP1D97r3/YcpF8bW7oDQx3fhHL6ibaTDNP0RxffuP3j4aPB4+OTps+cvRgeH5850VsBMmMbYy4I7aKSGGUps4LK1wFXRwEVx/WmjX3wD66TRX3HdQq54rWUlBcdALUY3J5R1ugwOQM9wCch7z0wLlqOxmivw3NaKr2jfU4awQk8DaKDCJHympkmd7HKTCX1LmevUwsuTrL/SWz2pkhUDL3t6Q//4mJX1EieL0Tidptuhd0G2B2Oyn7PF6BcrjegUaBQNd26epS3m4UKUooF+yDoHLRfXvIZ5gJvzXe63LfX0TWBKWhkbnka6Zf9OeK6cW6siOBXHpbutbcj/afMOq+PcS912CFrsFlVdQ9HQTeW0lBYENusAuLAy3ErFklsuMNQ+ZBq+C6MU16Vn0M+z3F/5ZJxN+j70k91u4y44P5pm6TT78n58erxvakBekdckIRn5QE7JZ3JGZkSQn9EgOogO4zhO4ndxtrPG0T7zkvwz8cffhQrBKw==</latexit><latexit sha1_base64="2Vyh+F2J1WNRU3hIn/gc2jRPn/U=">AAACgHicbVBba9swFJa9W5bd0u5xL2Jh4DDI7DJYGRQKe9ljB0tbiNwgy8eOqCUZ6XhLUP1D97r3/YcpF8bW7oDQx3fhHL6ibaTDNP0RxffuP3j4aPB4+OTps+cvRgeH5850VsBMmMbYy4I7aKSGGUps4LK1wFXRwEVx/WmjX3wD66TRX3HdQq54rWUlBcdALUY3J5R1ugwOQM9wCch7z0wLlqOxmivw3NaKr2jfU4awQk8DaKDCJHympkmd7HKTCX1LmevUwsuTrL/SWz2pkhUDL3t6Q//4mJX1EieL0Tidptuhd0G2B2Oyn7PF6BcrjegUaBQNd26epS3m4UKUooF+yDoHLRfXvIZ5gJvzXe63LfX0TWBKWhkbnka6Zf9OeK6cW6siOBXHpbutbcj/afMOq+PcS912CFrsFlVdQ9HQTeW0lBYENusAuLAy3ErFklsuMNQ+ZBq+C6MU16Vn0M+z3F/5ZJxN+j70k91u4y44P5pm6TT78n58erxvakBekdckIRn5QE7JZ3JGZkSQn9EgOogO4zhO4ndxtrPG0T7zkvwz8cffhQrBKw==</latexit>

Gotta get that intuition

l Prior: q ~ Beta(a, b); data = {n heads, m tails}l Estimate p, aka q

P(θ | D) For Bernoulli

This is the

beta PDF

This is ???

l Prior: q ~ Beta(a, b); data = {n heads, m tails}l Estimate p, aka q


= argmax✓

log1

�+ (a� 1) log ✓ + (b� 1) log(1� ✓) + n log ✓ +m log(1� ✓)

= argmax✓

logh 1�✓a�1(1� ✓)b�1

i

+n log f(heads|✓)+m log f(tails|✓)

= argmax✓

(a� 1 + n) log ✓ + (b� 1 +m) log(1� ✓)

This is the

beta PDF

Product of thetas and (1-theta)s

l Prior: q ~ Beta(a, b); D = {n heads, m tails}l Estimate p, aka q


✓MAP =n+ a� 1

n+m+ a+ b� 2

That’s the mode of the

updated beta

= argmax✓

(a� 1 + n) log ✓ + (b� 1 +m) log(1� ✓)

Hyper Parameters

X1

X2Xn

p

a b

p ⇠ Beta(a, b)

Xi ⇠ Bern(p)

a, b are fixed

Prior

Hyperparameter

Data distribution

MAP will estimate the most likely value of p for this model

l q is the probability a coin turns up headsl Model q with 2 different priors:

l P1(q) is Beta(3,8) (blue)l P2(q) is Beta(7,4) (red)

l They look pretty different!

l Now flip 100 coins; get 58 heads and 42 tailsl What do posteriors look like?

Where’d Ya Get Them P(!)?

l As long as we collect enough data, posteriors will converge to the true value!

It’s Like Having Twinsargmax returns the mode

• Just for review…• Have coin with unknown probability q of heads

§ Our prior (subjective) belief is that q ~ Beta(a, b)§ Now flip coin k = n + m times, getting n heads, m tails§ Posterior density: (q | n heads, m tails) ~ Beta(a+n,b+\m)

o Beta is conjugate for Bernoulli, Binomial, Geometric, and Negative Binomial

§ a and b are called “hyperparameters”o Saw (a + b – 2) imaginary trials, of those (a – 1) are “successes”

§ For a coin you never flipped before, use Beta(x, x) to denote you think coin likely to be fairo How strongly you feel coin is fair is a function of x

Conjugate Distributions Without Tears

Gonna Need Priors

Bernoulli p Beta

Binomial p Beta

Poisson � Gamma

Exponential � Gamma

Multinomial pi Dirichlet

Normal µ Normal

Normal �2 Inverse Gamma

Parameter Distribution for Parameter

Don’t need to know Inverse Gamma. But it will know you…

• Gamma(k, !) distribution§ Conjugate for Poisson Rate

o Also conjugate for Exponential, but we won’t delve into that

§ Intuitive understanding of hyperparameters:o Saw k total imaginary events during ! prior time periods

Good Times with Gamma

2.0

4.04.04.0

• Gamma(k, !) distribution§ Conjugate for Poisson Rate

o Also conjugate for Exponential, but we won’t delve into that

§ Intuitive understanding of hyperparameters:o Saw k total imaginary events during ! prior time periods

§ Updating with observationso After observing n events during next t time periods...

o ... posterior distribution is Gamma(k + n, ! + t)

o …MAP estimator for Poisson with Gamma prior is (k+n)/(! + t)

o Example: Prior for rate is Gamma(10, 5) o Saw 10 events in 5 time periods. Like observing at rate = 2o Now see 11 events in next 2 time periods à Gamma(21, 7)o MAP rate = 3

Good Times with Gamma

Reviving an Old Story Line

The Multinomial Distribution Mult(p1, … , pk)

p(x1, . . . , xk) =n!

x1! . . . xk!px11 . . . pxk

k

• Dirichlet(a1, a2, ..., am) distribution§ Conjugate for Multinomial

o Dirichlet generalizes Beta in same way Multinomial generalizes Bernoulli

§ Intuitive understanding of hyperparameters:o Saw imaginary trials, with (ai – 1) of outcome i

§ Updating to get the posterior distributiono After observing n1 + n2 + ... + nm, new trials with ni of outcome i...o ... posterior distribution is Dirichlet(a1 + n1, a2 + n2, ..., am + nm)

mai -å=

m

i 1

f(X1 = x1, X2 = x2, . . . , Xm = xm) = KmY

i=1

xai�1i

Multinomial is Multiple Times the Fun

• And now a cool animation of Dirichlet(a, a, a)§ This is actually log density (but you get the idea…)

Thanks Wikipedia!

Best Short Film in the Dirichlet Category

Example: Estimating Die Parameters

• Recall example of 6-sides die rolls:§ X ~ Multinomial(p1, p2, p3, p4, p5, p6)§ Roll n = 12 times§ Result: 3 ones, 2 twos, 0 threes, 3 fours, 1 fives, 3 sixes

o MLE: p1=3/12, p2=2/12, p3=0/12, p4=3/12, p5=1/12, p6=3/12

§ Dirichlet prior allows us to pretend we saw each outcome k times before. MAP estimate:o Laplace’s “law of succession”: idea above with k = 1

o Laplace estimate:

o Laplace: p1=4/18, p2=3/18, p3=1/18, p4=4/18, p5=2/18, p6=4/18

o No longer have 0 probability of rolling a three!

mknkXp i

i ++

=

mnXp i

i ++

=1

Your Happy Laplace

The last estimator has risen…

One Shot Learning

Single training example:

Test set:

Is Peer Grading Accurate Enough?

Peer Grading on CourseraHCI.

31,067 peer grades for 3,607 students.

Tuned Models of Peer Assessment. C Piech, J Huang, A Ng, D Koller


1. Defined random variables for:• True grade (si) for assignment i• Observed (zij) score for assign i• Bias (bj) for each grader j• Variance (rj) for each grader j


2. Designed a probabilistic model that defined the distributions for all random variableszji ⇠ N (µ = si + bj ,� =

prj)

si ⇠ N(µ0,�0)

bi ⇠ N(0, ⌘0)

= hyperparameter ri ⇠ InvGamma(↵0, ✓0)



2. Designed a probabilistic model that defined the distributions for all random variables

3. Found variable assignments using MAP estimation given the observed data

1. Defined random variables for:• True grade (si) for assignment i• Observed (zij) score for assign i• Bias (bj) for each grader j• Variance (rj) for each grader j

Inference or Machine Learning

Parent’s Club

-100 -80 -60 -40 -20 0 20 40 60 80-100 -80 -60 -40 -20 0 20 40 60 80

99% within 10pp

Error is based on ground truth assignments. Results are across all assignments (~10,000 submissions)

Some students

were getting very erroneous

grades

Before: After:

Improved Accuracy

Next time: Machine Learning algorithms

Maximum A Posteriori - MAP.pdfoFrequentist: play game infinitely many times and see how often...

Documents

Transcript of Maximum A Posteriori - MAP.pdfoFrequentist: play game infinitely many times and see how often...