Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate...

20
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander Robey Hamed Hassani, Manfred Morari, George J. Pappas NeurIPS 2019

Transcript of Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate...

Page 1: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Efficient and Accurate Estimation ofLipschitz Constants for Deep Neural Networks

Mahyar Fazlyab, Alexander RobeyHamed Hassani, Manfred Morari, George J. Pappas

NeurIPS 2019

Page 2: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Lipschitz Constant of Neural Networks

I A function f : Rnx → Rny is globally Lipschitz continuous on X ⊆ Rnx if

‖f(x)− f(y)‖ ≤ L‖x− y‖ ∀x, y ∈ X

I The smallest such L is called the Lipschitz constant.

L = supx,y∈X

‖f(x)− f(y)‖‖x− y‖ .

I A tight upper bound on L can be useful in several applications

– Robustness certification of classifiers– Closed-loop stability analysis of neural network controllers– Robust training– Generalization bounds

2

Page 3: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Lipschitz Constant as a Measure of Robustness

Linear Linear Linear f(x)<latexit sha1_base64="k7ufXGKAt77G/FaPA5tscWX9PSc=">AAAB7XicbVA9TwJBEJ3DL8Qv1NJmI5hgQ+6w0MKCxMYSE/lI4EL2lj1Y2du97O4ZyYX/YGOhMbb+Hzv/jQtcoeBLJnl5byYz84KYM21c99vJra1vbG7ltws7u3v7B8XDo5aWiSK0SSSXqhNgTTkTtGmY4bQTK4qjgNN2ML6Z+e1HqjST4t5MYupHeChYyAg2VmqVw8rTeblfLLlVdw60SryMlCBDo1/86g0kSSIqDOFY667nxsZPsTKMcDot9BJNY0zGeEi7lgocUe2n82un6MwqAxRKZUsYNFd/T6Q40noSBbYzwmakl72Z+J/XTUx45adMxImhgiwWhQlHRqLZ62jAFCWGTyzBRDF7KyIjrDAxNqCCDcFbfnmVtGpV76Lq3dVK9essjjycwClUwINLqMMtNKAJBB7gGV7hzZHOi/PufCxac042cwx/4Hz+ACPVjig=</latexit>

f(x?)<latexit sha1_base64="j3AFkUraq/AuU1MJs5rROJBN+MY=">AAAB83icbZBNT8JAEIan+IX4hXr00ggmeCEtFz2SePGIiXwktJLtsoUN222zOzWShr/hxYPGePXPePPfuEAPCr7JJk/emcnMvkEiuEbH+bYKG5tb2zvF3dLe/sHhUfn4pKPjVFHWprGIVS8gmgkuWRs5CtZLFCNRIFg3mNzM691HpjSP5T1OE+ZHZCR5yClBY3nVsPb04Gkk6rI6KFecurOQvQ5uDhXI1RqUv7xhTNOISaSCaN13nQT9jCjkVLBZyUs1SwidkBHrG5QkYtrPFjfP7AvjDO0wVuZJtBfu74mMRFpPo8B0RgTHerU2N/+r9VMMr/2MyyRFJulyUZgKG2N7HoA95IpRFFMDhCpubrXpmChC0cRUMiG4q19eh06j7hq+a1SazTyOIpzBOdTAhStowi20oA0UEniGV3izUuvFerc+lq0FK585hT+yPn8Ay5mQ2Q==</latexit><latexit sha1_base64="j3AFkUraq/AuU1MJs5rROJBN+MY=">AAAB83icbZBNT8JAEIan+IX4hXr00ggmeCEtFz2SePGIiXwktJLtsoUN222zOzWShr/hxYPGePXPePPfuEAPCr7JJk/emcnMvkEiuEbH+bYKG5tb2zvF3dLe/sHhUfn4pKPjVFHWprGIVS8gmgkuWRs5CtZLFCNRIFg3mNzM691HpjSP5T1OE+ZHZCR5yClBY3nVsPb04Gkk6rI6KFecurOQvQ5uDhXI1RqUv7xhTNOISaSCaN13nQT9jCjkVLBZyUs1SwidkBHrG5QkYtrPFjfP7AvjDO0wVuZJtBfu74mMRFpPo8B0RgTHerU2N/+r9VMMr/2MyyRFJulyUZgKG2N7HoA95IpRFFMDhCpubrXpmChC0cRUMiG4q19eh06j7hq+a1SazTyOIpzBOdTAhStowi20oA0UEniGV3izUuvFerc+lq0FK585hT+yPn8Ay5mQ2Q==</latexit><latexit sha1_base64="j3AFkUraq/AuU1MJs5rROJBN+MY=">AAAB83icbZBNT8JAEIan+IX4hXr00ggmeCEtFz2SePGIiXwktJLtsoUN222zOzWShr/hxYPGePXPePPfuEAPCr7JJk/emcnMvkEiuEbH+bYKG5tb2zvF3dLe/sHhUfn4pKPjVFHWprGIVS8gmgkuWRs5CtZLFCNRIFg3mNzM691HpjSP5T1OE+ZHZCR5yClBY3nVsPb04Gkk6rI6KFecurOQvQ5uDhXI1RqUv7xhTNOISaSCaN13nQT9jCjkVLBZyUs1SwidkBHrG5QkYtrPFjfP7AvjDO0wVuZJtBfu74mMRFpPo8B0RgTHerU2N/+r9VMMr/2MyyRFJulyUZgKG2N7HoA95IpRFFMDhCpubrXpmChC0cRUMiG4q19eh06j7hq+a1SazTyOIpzBOdTAhStowi20oA0UEniGV3izUuvFerc+lq0FK585hT+yPn8Ay5mQ2Q==</latexit><latexit sha1_base64="j3AFkUraq/AuU1MJs5rROJBN+MY=">AAAB83icbZBNT8JAEIan+IX4hXr00ggmeCEtFz2SePGIiXwktJLtsoUN222zOzWShr/hxYPGePXPePPfuEAPCr7JJk/emcnMvkEiuEbH+bYKG5tb2zvF3dLe/sHhUfn4pKPjVFHWprGIVS8gmgkuWRs5CtZLFCNRIFg3mNzM691HpjSP5T1OE+ZHZCR5yClBY3nVsPb04Gkk6rI6KFecurOQvQ5uDhXI1RqUv7xhTNOISaSCaN13nQT9jCjkVLBZyUs1SwidkBHrG5QkYtrPFjfP7AvjDO0wVuZJtBfu74mMRFpPo8B0RgTHerU2N/+r9VMMr/2MyyRFJulyUZgKG2N7HoA95IpRFFMDhCpubrXpmChC0cRUMiG4q19eh06j7hq+a1SazTyOIpzBOdTAhStowi20oA0UEniGV3izUuvFerc+lq0FK585hT+yPn8Ay5mQ2Q==</latexit>x?

<latexit sha1_base64="1HSaP89i6ZNuxL6ngZHnK0kxsVM=">AAAB8HicbZA9T8MwEIYvfJbyVWBkiWiRmKqkC4yVWBiLRD9QGyrHdVqrthPZF0QV9VewMIAQKz+HjX+D22aAlley9Oi9O/nuDRPBDXret7O2vrG5tV3YKe7u7R8clo6OWyZONWVNGotYd0JimOCKNZGjYJ1EMyJDwdrh+HpWbz8ybXis7nCSsECSoeIRpwStdV95eugZJLrSL5W9qjeXuwp+DmXI1eiXvnqDmKaSKaSCGNP1vQSDjGjkVLBpsZcalhA6JkPWtaiIZCbI5gtP3XPrDNwo1vYpdOfu74mMSGMmMrSdkuDILNdm5n+1borRVZBxlaTIFF18FKXCxdidXe8OuGYUxcQCoZrbXV06IppQtBkVbQj+8smr0KpVfcu3tXK9nsdRgFM4gwvw4RLqcAMNaAIFCc/wCm+Odl6cd+dj0brm5DMn8EfO5w8+Z5AE</latexit><latexit sha1_base64="1HSaP89i6ZNuxL6ngZHnK0kxsVM=">AAAB8HicbZA9T8MwEIYvfJbyVWBkiWiRmKqkC4yVWBiLRD9QGyrHdVqrthPZF0QV9VewMIAQKz+HjX+D22aAlley9Oi9O/nuDRPBDXret7O2vrG5tV3YKe7u7R8clo6OWyZONWVNGotYd0JimOCKNZGjYJ1EMyJDwdrh+HpWbz8ybXis7nCSsECSoeIRpwStdV95eugZJLrSL5W9qjeXuwp+DmXI1eiXvnqDmKaSKaSCGNP1vQSDjGjkVLBpsZcalhA6JkPWtaiIZCbI5gtP3XPrDNwo1vYpdOfu74mMSGMmMrSdkuDILNdm5n+1borRVZBxlaTIFF18FKXCxdidXe8OuGYUxcQCoZrbXV06IppQtBkVbQj+8smr0KpVfcu3tXK9nsdRgFM4gwvw4RLqcAMNaAIFCc/wCm+Odl6cd+dj0brm5DMn8EfO5w8+Z5AE</latexit><latexit sha1_base64="1HSaP89i6ZNuxL6ngZHnK0kxsVM=">AAAB8HicbZA9T8MwEIYvfJbyVWBkiWiRmKqkC4yVWBiLRD9QGyrHdVqrthPZF0QV9VewMIAQKz+HjX+D22aAlley9Oi9O/nuDRPBDXret7O2vrG5tV3YKe7u7R8clo6OWyZONWVNGotYd0JimOCKNZGjYJ1EMyJDwdrh+HpWbz8ybXis7nCSsECSoeIRpwStdV95eugZJLrSL5W9qjeXuwp+DmXI1eiXvnqDmKaSKaSCGNP1vQSDjGjkVLBpsZcalhA6JkPWtaiIZCbI5gtP3XPrDNwo1vYpdOfu74mMSGMmMrSdkuDILNdm5n+1borRVZBxlaTIFF18FKXCxdidXe8OuGYUxcQCoZrbXV06IppQtBkVbQj+8smr0KpVfcu3tXK9nsdRgFM4gwvw4RLqcAMNaAIFCc/wCm+Odl6cd+dj0brm5DMn8EfO5w8+Z5AE</latexit><latexit sha1_base64="1HSaP89i6ZNuxL6ngZHnK0kxsVM=">AAAB8HicbZA9T8MwEIYvfJbyVWBkiWiRmKqkC4yVWBiLRD9QGyrHdVqrthPZF0QV9VewMIAQKz+HjX+D22aAlley9Oi9O/nuDRPBDXret7O2vrG5tV3YKe7u7R8clo6OWyZONWVNGotYd0JimOCKNZGjYJ1EMyJDwdrh+HpWbz8ybXis7nCSsECSoeIRpwStdV95eugZJLrSL5W9qjeXuwp+DmXI1eiXvnqDmKaSKaSCGNP1vQSDjGjkVLBpsZcalhA6JkPWtaiIZCbI5gtP3XPrDNwo1vYpdOfu74mMSGMmMrSdkuDILNdm5n+1borRVZBxlaTIFF18FKXCxdidXe8OuGYUxcQCoZrbXV06IppQtBkVbQj+8smr0KpVfcu3tXK9nsdRgFM4gwvw4RLqcAMNaAIFCc/wCm+Odl6cd+dj0brm5DMn8EfO5w8+Z5AE</latexit>

kx � x?k2 ✏<latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="p3GAkxepYK0/hd8vlF9v81bI25M=">AAAB/XicbZBLTwIxFIXv4AsRFd26sBFM3Ehm2OjSxI1LTOSRMEg65QINnc7YdgwEWLrxr7hxoTH+DXf+G8tjoeBJmnw5p83tPUEsuDau++2k1tY3NrfS25md7O7efu4gW9VRohhWWCQiVQ+oRsElVgw3AuuxQhoGAmtB/3qa1x5RaR7JOzOMsRnSruQdzqixVit3XPDHg/PBva8NVf64VSK+wAfiY6y5iGShlcu7RXcmsgreAvKwULmV+/LbEUtClIYJqnXDc2PTHFFlOBM4yfiJxpiyPu1iw6KkIermaLbIhJxap006kbJHGjJzf78Y0VDrYRjYmyE1Pb2cTc3/skZiOpfNEZdxYlCy+aBOIoiJyLQV0uYKmRFDC5Qpbv9KWI8qyoztLmNL8JZXXoVqqehZvnUhDUdwAmfgwQVcwQ2UoQIMnuAF3uDdeXZenY95XSln0dsh/JHz+QM32Je8</latexit><latexit sha1_base64="p3GAkxepYK0/hd8vlF9v81bI25M=">AAAB/XicbZBLTwIxFIXv4AsRFd26sBFM3Ehm2OjSxI1LTOSRMEg65QINnc7YdgwEWLrxr7hxoTH+DXf+G8tjoeBJmnw5p83tPUEsuDau++2k1tY3NrfS25md7O7efu4gW9VRohhWWCQiVQ+oRsElVgw3AuuxQhoGAmtB/3qa1x5RaR7JOzOMsRnSruQdzqixVit3XPDHg/PBva8NVf64VSK+wAfiY6y5iGShlcu7RXcmsgreAvKwULmV+/LbEUtClIYJqnXDc2PTHFFlOBM4yfiJxpiyPu1iw6KkIermaLbIhJxap006kbJHGjJzf78Y0VDrYRjYmyE1Pb2cTc3/skZiOpfNEZdxYlCy+aBOIoiJyLQV0uYKmRFDC5Qpbv9KWI8qyoztLmNL8JZXXoVqqehZvnUhDUdwAmfgwQVcwQ2UoQIMnuAF3uDdeXZenY95XSln0dsh/JHz+QM32Je8</latexit><latexit sha1_base64="PsaHGvPWV2KMb4TyqF9oOMdYXqg=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr5x8uXyPI4sOkIn6Ay56AKV0TWqoCqi6BE9o1f0Zj1ZL9a79TFrzVjzmUP0R9bnD8sVmSk=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit><latexit sha1_base64="DoHRRz7mM2yqCp0yK1NCedRVspQ=">AAACCHicbZC7TsMwFIadcivlFmBkwKJFYqFKusBYiYWxSPQiNSVy3NPWquME20Gt0o4svAoLAwix8ghsvA3uZYCWX7L06T/n6Pj8QcyZ0o7zbWVWVtfWN7Kbua3tnd09e/+gpqJEUqjSiEeyERAFnAmoaqY5NGIJJAw41IP+1aRefwCpWCRu9TCGVki6gnUYJdpYvn1c8EaD88GdpzSR3sgvYY/DPfYgVoxHouDbeafoTIWXwZ1DHs1V8e0vrx3RJAShKSdKNV0n1q2USM0oh3HOSxTEhPZJF5oGBQlBtdLpIWN8apw27kTSPKHx1P09kZJQqWEYmM6Q6J5arE3M/2rNRHcuWykTcaJB0NmiTsKxjvAkFdxmEqjmQwOESmb+immPSEK1yS5nQnAXT16GWqnoGr4p5cvleRxZdIRO0Bly0QUqo2tUQVVE0SN6Rq/ozXqyXqx362PWmrHmM4foj6zPH8u1mSs=</latexit> kf(x) � f(x?)k2 �

<latexit sha1_base64="kcW+Rnzd5H2LQ0zdmiQW/3R8W1g=">AAACDHicbVDLTgIxFO34RHyhLt00gokuJDO40CWJG5eYyCNhkHTKHWjsPGzvGAnwAW78FTcuNMatH+DOv7HALBQ8SZuTc85Ne48XS6HRtr+thcWl5ZXVzFp2fWNzazu3s1vTUaI4VHkkI9XwmAYpQqiiQAmNWAELPAl17/Zi7NfvQWkRhdfYj6EVsG4ofMEZGqmdyxfcoX/0cHxirhtXI1PH7rBdoq6EO+p2QCIrmJRdtCeg88RJSZ6kqLRzX24n4kkAIXLJtG46doytAVMouIRR1k00xIzfsi40DQ1ZALo1mCwzoodG6VA/UuaESCfq74kBC7TuB55JBgx7etYbi/95zQT989ZAhHGCEPLpQ34iKUZ03AztCAUcZd8QxpUwf6W8xxTjaPrLmhKc2ZXnSa1UdE6LzlUpXy6ndWTIPjkgR8QhZ6RMLkmFVAknj+SZvJI368l6sd6tj2l0wUpn9sgfWJ8/PGqZ1A==</latexit>

x<latexit sha1_base64="BmVm2kBoL99/m27NxkUbvUDufsI=">AAAB6nicbZA9TwJBEIbn8AvxC7W02QgmVuSORksSG0uM8pHAhewte7Bhb++yO2ckF36CjYXG2PqL7Pw3LnCFgm+yyZN3ZrIzb5BIYdB1v53CxubW9k5xt7S3f3B4VD4+aZs41Yy3WCxj3Q2o4VIo3kKBkncTzWkUSN4JJjfzeueRayNi9YDThPsRHSkRCkbRWvfVp+qgXHFr7kJkHbwcKpCrOSh/9YcxSyOukElqTM9zE/QzqlEwyWelfmp4QtmEjnjPoqIRN362WHVGLqwzJGGs7VNIFu7viYxGxkyjwHZGFMdmtTY3/6v1Ugyv/UyoJEWu2PKjMJUEYzK/mwyF5gzl1AJlWthdCRtTTRnadEo2BG/15HVo12ue5bt6pdHI4yjCGZzDJXhwBQ24hSa0gMEInuEV3hzpvDjvzseyteDkM6fwR87nD5y6jVQ=</latexit><latexit sha1_base64="BmVm2kBoL99/m27NxkUbvUDufsI=">AAAB6nicbZA9TwJBEIbn8AvxC7W02QgmVuSORksSG0uM8pHAhewte7Bhb++yO2ckF36CjYXG2PqL7Pw3LnCFgm+yyZN3ZrIzb5BIYdB1v53CxubW9k5xt7S3f3B4VD4+aZs41Yy3WCxj3Q2o4VIo3kKBkncTzWkUSN4JJjfzeueRayNi9YDThPsRHSkRCkbRWvfVp+qgXHFr7kJkHbwcKpCrOSh/9YcxSyOukElqTM9zE/QzqlEwyWelfmp4QtmEjnjPoqIRN362WHVGLqwzJGGs7VNIFu7viYxGxkyjwHZGFMdmtTY3/6v1Ugyv/UyoJEWu2PKjMJUEYzK/mwyF5gzl1AJlWthdCRtTTRnadEo2BG/15HVo12ue5bt6pdHI4yjCGZzDJXhwBQ24hSa0gMEInuEV3hzpvDjvzseyteDkM6fwR87nD5y6jVQ=</latexit><latexit sha1_base64="BmVm2kBoL99/m27NxkUbvUDufsI=">AAAB6nicbZA9TwJBEIbn8AvxC7W02QgmVuSORksSG0uM8pHAhewte7Bhb++yO2ckF36CjYXG2PqL7Pw3LnCFgm+yyZN3ZrIzb5BIYdB1v53CxubW9k5xt7S3f3B4VD4+aZs41Yy3WCxj3Q2o4VIo3kKBkncTzWkUSN4JJjfzeueRayNi9YDThPsRHSkRCkbRWvfVp+qgXHFr7kJkHbwcKpCrOSh/9YcxSyOukElqTM9zE/QzqlEwyWelfmp4QtmEjnjPoqIRN362WHVGLqwzJGGs7VNIFu7viYxGxkyjwHZGFMdmtTY3/6v1Ugyv/UyoJEWu2PKjMJUEYzK/mwyF5gzl1AJlWthdCRtTTRnadEo2BG/15HVo12ue5bt6pdHI4yjCGZzDJXhwBQ24hSa0gMEInuEV3hzpvDjvzseyteDkM6fwR87nD5y6jVQ=</latexit><latexit sha1_base64="BmVm2kBoL99/m27NxkUbvUDufsI=">AAAB6nicbZA9TwJBEIbn8AvxC7W02QgmVuSORksSG0uM8pHAhewte7Bhb++yO2ckF36CjYXG2PqL7Pw3LnCFgm+yyZN3ZrIzb5BIYdB1v53CxubW9k5xt7S3f3B4VD4+aZs41Yy3WCxj3Q2o4VIo3kKBkncTzWkUSN4JJjfzeueRayNi9YDThPsRHSkRCkbRWvfVp+qgXHFr7kJkHbwcKpCrOSh/9YcxSyOukElqTM9zE/QzqlEwyWelfmp4QtmEjnjPoqIRN362WHVGLqwzJGGs7VNIFu7viYxGxkyjwHZGFMdmtTY3/6v1Ugyv/UyoJEWu2PKjMJUEYzK/mwyF5gzl1AJlWthdCRtTTRnadEo2BG/15HVo12ue5bt6pdHI4yjCGZzDJXhwBQ24hSa0gMEInuEV3hzpvDjvzseyteDkM6fwR87nD5y6jVQ=</latexit>

kf(x) � f(x?)k2 L2kx � x?k2<latexit sha1_base64="TZXeUxHiiPOPf+dFoMuZUOCv/+8=">AAACGXicbZC7TsMwFIYdrqXcAowsFi1SO7RKygBjJRYGhiLRi9SEyHGd1qpzwXZQq7SvwcKrsDCAECNMvA1OmwFajmTr0/+fI/v8bsSokIbxra2srq1vbOa28ts7u3v7+sFhS4Qxx6SJQxbyjosEYTQgTUklI52IE+S7jLTd4WXqtx8IFzQMbuU4IraP+gH1KEZSSY5uFK2JVxqVK+q6s4REvGxNnBq0GLmH1ylMRpXMSY2ioxeMqjEruAxmBgWQVcPRP61eiGOfBBIzJETXNCJpJ4hLihmZ5q1YkAjhIeqTrsIA+UTYyWyzKTxVSg96IVcnkHCm/p5IkC/E2HdVp4/kQCx6qfif142ld2EnNIhiSQI8f8iLGZQhTGOCPcoJlmysAGFO1V8hHiCOsFRh5lUI5uLKy9CqVc2zqnlTK9TrWRw5cAxOQAmY4BzUwRVogCbA4BE8g1fwpj1pL9q79jFvXdGymSPwp7SvH4KVnr8=</latexit>

activations activations

✏<latexit sha1_base64="i4udT/drOOO6oO7noIEGU/CaHW4=">AAAB8XicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsspCSxscREwAgXsrfMwYa9vcvungm58C9sLDTG1n9j579xgSsUfMkkL+/NZGZekAiujet+O4WNza3tneJuaW//4PCofHzS0XGqGLZZLGL1EFCNgktsG24EPiQKaRQI7AaTm7nffUKleSzvzTRBP6IjyUPOqLHSY7WPieYiltVBueLW3AXIOvFyUoEcrUH5qz+MWRqhNExQrXuemxg/o8pwJnBW6qcaE8omdIQ9SyWNUPvZ4uIZubDKkISxsiUNWai/JzIaaT2NAtsZUTPWq95c/M/rpSZs+BmXSWpQsuWiMBXExGT+PhlyhcyIqSWUKW5vJWxMFWXGhlSyIXirL6+TTr3mXdW8u3ql2cjjKMIZnMMleHANTbiFFrSBgYRneIU3RzsvzrvzsWwtOPnMKfyB8/kDBdqQcw==</latexit> �

<latexit sha1_base64="7vRx4A6LElEKTYMBacNXemCuKD8=">AAAB73icbVA9SwNBEJ3zM8avqKXNYiJYhbtYaCUBG8sI5gOSI+xt5pIle3vn7p4QjvwJGwtFbP07dv4bN8kVmvhg4PHeDDPzgkRwbVz321lb39jc2i7sFHf39g8OS0fHLR2nimGTxSJWnYBqFFxi03AjsJMopFEgsB2Mb2d++wmV5rF8MJME/YgOJQ85o8ZKnUpvgMLQSr9UdqvuHGSVeDkpQ45Gv/TVG8QsjVAaJqjWXc9NjJ9RZTgTOC32Uo0JZWM6xK6lkkao/Wx+75ScW2VAwljZkobM1d8TGY20nkSB7YyoGellbyb+53VTE177GZdJalCyxaIwFcTEZPY8GXCFzIiJJZQpbm8lbEQVZcZGVLQheMsvr5JWrepdVr37Wrl+k8dRgFM4gwvw4ArqcAcNaAIDAc/wCm/Oo/PivDsfi9Y1J585gT9wPn8ASyOPdQ==</latexit>

Proposition 1:

The inequality

L2 ≤1

ε√2

min1≤j≤k, j 6=i?

|fj(x?)− fi?(x?)|,

implies C(x) := argmax1≤i≤kfi(x) = C(x?) for all ‖x− x?‖2 ≤ ε.

• A global measure of robustness 3

Page 4: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Incremental Quadratic Constraints

Definition 1: Incremental Quadratic Constraint [Acıkmese and Corless, 2011]

A function φ : Rn → Rn satisfies the incremental quadratic constraint defined byQ ⊂ S2nif for any Q ∈ Q and x, y ∈ Rn, φ satisfies

[x− y

φ(x)− φ(y)

]>Q

[x− y

φ(x)− φ(y)

]≥ 0.

I Lipschitz continuity (in `2 norm) is a special case

[x− y

φ(x)− φ(y)

]> [L2In 00 −In

] [x− y

φ(x)− φ(y)

]≥ 0.

4

Page 5: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Incremental QC for Activation Functions

Lemma 1: Incremental QC for activation functions

Suppose ϕ is slope-restricted on [α, β]. Then φ(x) = [ϕ(x1) · · ·ϕ(xn)]> satisfies

[x− y

φ(x)− φ(y)

]> [ −2αβT (α+ β)T(α+ β)T −2T

] [x− y

φ(x)− φ(y)

]≥ 0,

for all x, y ∈ Rn and all T ∈ Tn, where

Tn = {T ∈ Sn | T =

n∑

i=1

λiieie>i +

∑

i<j

λij(ei−ej)(ei−ej)>, λij ≥ 0}.

5

Page 6: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Visual “Proof”

-5 0 5-1

-0.5

0

0.5

1

(xj ,'(xj))<latexit sha1_base64="7KVgF9Hepxz+AlzTMESMIJp61E0=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64="7KVgF9Hepxz+AlzTMESMIJp61E0=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64="7KVgF9Hepxz+AlzTMESMIJp61E0=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64="7KVgF9Hepxz+AlzTMESMIJp61E0=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit>

(xi,'(xi))<latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit>

↵ '(xj) � '(xi)

xj � xi �

<latexit sha1_base64="bXpVOSwBhezioWcgI+XxN1XSrkk=">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64="bXpVOSwBhezioWcgI+XxN1XSrkk=">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64="bXpVOSwBhezioWcgI+XxN1XSrkk=">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64="bXpVOSwBhezioWcgI+XxN1XSrkk=">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit>

xi<latexit sha1_base64="pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64="pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64="pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64="pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit>

'(xi)<latexit sha1_base64="2RSP5OM1rKvlHPSZBvlymYbyN3E=">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64="2RSP5OM1rKvlHPSZBvlymYbyN3E=">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64="2RSP5OM1rKvlHPSZBvlymYbyN3E=">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64="2RSP5OM1rKvlHPSZBvlymYbyN3E=">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit>

'(yi)<latexit sha1_base64="RG1iAtaTSYwavALhRg+orYL4LFU=">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64="RG1iAtaTSYwavALhRg+orYL4LFU=">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64="RG1iAtaTSYwavALhRg+orYL4LFU=">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64="RG1iAtaTSYwavALhRg+orYL4LFU=">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit>

yi<latexit sha1_base64="WmgSYaT4KZFZh7wSYgzHlBn+laY=">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64="WmgSYaT4KZFZh7wSYgzHlBn+laY=">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64="WmgSYaT4KZFZh7wSYgzHlBn+laY=">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64="WmgSYaT4KZFZh7wSYgzHlBn+laY=">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit>

-5 0 5-1

-0.5

0

0.5

1

(xi,'(xi))<latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64="GllpdvBtNW9LH6e2gGOX+rXlP0k=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit>

(yi,'(yi))<latexit sha1_base64="OwRczwPm0a80lXMROfDF4JnscN8=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64="OwRczwPm0a80lXMROfDF4JnscN8=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64="OwRczwPm0a80lXMROfDF4JnscN8=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64="OwRczwPm0a80lXMROfDF4JnscN8=">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit>

↵ '(yi) � '(xi)

yi � xi �

<latexit sha1_base64="71vqPKjY+TK9gITBZZq4AVlX2HI=">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64="71vqPKjY+TK9gITBZZq4AVlX2HI=">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64="71vqPKjY+TK9gITBZZq4AVlX2HI=">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64="71vqPKjY+TK9gITBZZq4AVlX2HI=">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit>

xj<latexit sha1_base64="Qmcs1an4x+++o9Cy9ZfnlIj1FDM=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64="Qmcs1an4x+++o9Cy9ZfnlIj1FDM=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64="Qmcs1an4x+++o9Cy9ZfnlIj1FDM=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64="Qmcs1an4x+++o9Cy9ZfnlIj1FDM=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit>

yj<latexit sha1_base64="GXeTI3hCKBcAvNjSYFW2ns3xRoM=">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64="GXeTI3hCKBcAvNjSYFW2ns3xRoM=">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64="GXeTI3hCKBcAvNjSYFW2ns3xRoM=">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64="GXeTI3hCKBcAvNjSYFW2ns3xRoM=">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit>

'(xj)<latexit sha1_base64="oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64="oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64="oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64="oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit>

'(yj)<latexit sha1_base64="I7R4a32uvuL7gDjagUmI6UlCRsU=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64="I7R4a32uvuL7gDjagUmI6UlCRsU=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64="I7R4a32uvuL7gDjagUmI6UlCRsU=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64="I7R4a32uvuL7gDjagUmI6UlCRsU=">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit>

nonnegative combination

�(x) =

264'(x1)

...'(xn)

375

<latexit sha1_base64="guxvl0q2bU+xVoPoliEsRkzbL0M=">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64="guxvl0q2bU+xVoPoliEsRkzbL0M=">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64="guxvl0q2bU+xVoPoliEsRkzbL0M=">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64="guxvl0q2bU+xVoPoliEsRkzbL0M=">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit>

x � y

�(x) � �(y)

�> �2↵�T (↵ + �)T

(↵ + �)T �2T

� x � y

�(x) � �(y)

�� 0

<latexit sha1_base64="xE4TKIfpLj8g4gjosp4+9y8sOeo=">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64="xE4TKIfpLj8g4gjosp4+9y8sOeo=">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64="xE4TKIfpLj8g4gjosp4+9y8sOeo=">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64="xE4TKIfpLj8g4gjosp4+9y8sOeo=">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit>

6

Page 7: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Main Idea

I Original network with Lipschitz constant L2

Linear Linear

y1<latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit>

x1<latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit>x0

<latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit>

y2<latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit>

x0<latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit> y1

<latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit>

x1<latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit>

y2<latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit>

I Incrementally Quadratically Constrained Linear Network (IQCLN)

Linear Linear

x0<latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit><latexit sha1_base64="C6tOYRQ7vOtaRX8BQQC0iQJnarU=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04Nb6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHv12N9A==</latexit>

y2<latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit><latexit sha1_base64="yxxj+G8OPCSpuuevqkaSKw6jqu0=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pkhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3R0nCpCfRLzWPVCrClnkvqGGU57iaJYhJx2w+ntvN59okqzWD6YLKGBwGPJIkawsZZfzx6b9WG15jbchdA6eAXUoFB7WP0ajGKSCioN4VjrvucmJsixMoxwOqsMUk0TTKZ4TPsWJRZUB/li2Rm6sM4IRbGyTxq0cH9P5FhonYnQdgpsJnq1Njf/q/VTE10HOZNJaqgky4+ilCMTo/nlaMQUJYZnFjBRzO6KyAQrTIzNp2JD8FZPXodOs+FZvm/WWjdFHGU4g3O4BA+uoAV30AYfCDB4hld4c6Tz4rw7H8vWklPMnMIfOZ8/w+6N9w==</latexit>

x0<latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit><latexit sha1_base64="9J+ynOBFN0XXGcAqpkKrLN982n8=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRguLgI1lBHOB7BpmZ88mQ2YvzMyKYclr2FgoYuvL2Pk2TpItNPGHgY//nMM58/up4Erb9re1tr6xubVd2inv7u0fHFaOjjsqySTDNktEIns+VSh4jG3NtcBeKpFGvsCuP76Z1buPKBVP4ns9SdGL6DDmIWdUG8utuZqLAMnTg10bVKp23Z6LrIJTQBUKtQaVLzdIWBZhrJmgSvUdO9VeTqXmTOC07GYKU8rGdIh9gzGNUHn5/OYpOTdOQMJEmhdrMnd/T+Q0UmoS+aYzonqklmsz879aP9PhlZfzOM00xmyxKMwE0QmZBUACLpFpMTFAmeTmVsJGVFKmTUxlE4Kz/OVV6DTqjuG7RrV5XcRRglM4gwtw4BKacAstaAODFJ7hFd6szHqx3q2PReuaVcycwB9Znz+0mZDG</latexit>

y2<latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit><latexit sha1_base64="65sCX0O1Vp7SoObv8xiFdvgtIZ4=">AAAB83icbZC7SgNBFIbPxluMt6ilzWAiWIXdNFpJwMYygrlAdg2zs7PJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmWK8wxKZqL5PNZci5h0UKHk/VZxGvuQ9f3I7r/eeuNIiiR8wT7kX0VEsQsEoGsutuyhkwEn+2KwPqzW7YS9E1sEpoAaF2sPqlxskLIt4jExSrQeOnaI3pQoFk3xWcTPNU8omdMQHBmMace1NFzfPyIVxAhImyrwYycL9PTGlkdZ55JvOiOJYr9bm5n+1QYbhtTcVcZohj9lyUZhJggmZB0ACoThDmRugTAlzK2FjqihDE1PFhOCsfnkdus2GY/i+WWvdFHGU4QzO4RIcuIIW3EEbOsAghWd4hTcrs16sd+tj2VqyiplT+CPr8we5xJDL</latexit>

y1<latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit><latexit sha1_base64="eUbFnEfyWMjYTAUgCeR9bsRZVvk=">AAAB7HicbZDNSgMxFIXv1L9a/6ou3QRbwVWZ6UZXUnDjsoLTFtqxZNJMG5pJhiQjDEOfwY0LRdz6QO58G9N2Ftp6IPBx7r3k3hMmnGnjut9OaWNza3unvFvZ2z84PKoen3S0TBWhPpFcql6INeVMUN8ww2kvURTHIafdcHo7r3efqNJMigeTJTSI8ViwiBFsrOXXs0evPqzW3Ia7EFoHr4AaFGoPq1+DkSRpTIUhHGvd99zEBDlWhhFOZ5VBqmmCyRSPad+iwDHVQb5YdoYurDNCkVT2CYMW7u+JHMdaZ3FoO2NsJnq1Njf/q/VTE10HORNJaqggy4+ilCMj0fxyNGKKEsMzC5goZndFZIIVJsbmU7EheKsnr0On2fAs3zdrrZsijjKcwTlcggdX0II7aIMPBBg8wyu8OcJ5cd6dj2VrySlmTuGPnM8fwmmN9g==</latexit>

y1<latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit><latexit sha1_base64="JPSKS7ln6VHlWD+W3XtlejUo+3w=">AAAB83icbZC7SgNBFIZn4y3GW9TSZjARrMJuGq0kYGMZwVwgu4bZ2bPJkNkLM2eFJeQ1bCwUsfVl7HwbJ8kWmvjDwMd/zuGc+f1UCo22/W2VNja3tnfKu5W9/YPDo+rxSVcnmeLQ4YlMVN9nGqSIoYMCJfRTBSzyJfT8ye283nsCpUUSP2CeghexUSxCwRkay627KGQANH906sNqzW7YC9F1cAqokULtYfXLDRKeRRAjl0zrgWOn6E2ZQsElzCpupiFlfMJGMDAYswi0N13cPKMXxglomCjzYqQL9/fElEVa55FvOiOGY71am5v/1QYZhtfeVMRphhDz5aIwkxQTOg+ABkIBR5kbYFwJcyvlY6YYRxNTxYTgrH55HbrNhmP4vllr3RRxlMkZOSeXxCFXpEXuSJt0CCcpeSav5M3KrBfr3fpYtpasYuaU/JH1+QO4P5DK</latexit>

x1<latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit><latexit sha1_base64="GLHp/tQR2hCLFqJ7sxcm2CFPS9U=">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmj0cqQ2Fhi4gEJnGRvWWDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3jCRwqDrfjtr6xubW9uFneLu3v7BYenouGniVDPus1jGuh1Sw6VQ3EeBkrcTzWkUSt4KxzezeuuRayNidY+ThAcRHSoxEIyitfzK04NX6ZXKbtWdi6yCl0MZcjV6pa9uP2ZpxBUySY3peG6CQUY1Cib5tNhNDU8oG9Mh71hUNOImyObLTsm5dfpkEGv7FJK5+3sio5Exkyi0nRHFkVmuzcz/ap0UB1dBJlSSIlds8dEglQRjMruc9IXmDOXEAmVa2F0JG1FNGdp8ijYEb/nkVWjWqp7lu1q5fp3HUYBTOIML8OAS6nALDfCBgYBneIU3RzkvzrvzsWhdc/KZE/gj5/MHwOKN9Q==</latexit>

x1<latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit><latexit sha1_base64="SVxPp3MhBV5bWqUypPfvEiT2R0o=">AAAB83icbZC7SgNBFIbPeo3xFrW0GUwEq7CbRisJ2FhGMBfIrmF29mwyZPbCzKwYlryGjYUitr6MnW/jJNlCE38Y+PjPOZwzv58KrrRtf1tr6xubW9ulnfLu3v7BYeXouKOSTDJss0QksudThYLH2NZcC+ylEmnkC+z645tZvfuIUvEkvteTFL2IDmMecka1sdyaq7kIkDw9OLVBpWrX7bnIKjgFVKFQa1D5coOEZRHGmgmqVN+xU+3lVGrOBE7LbqYwpWxMh9g3GNMIlZfPb56Sc+MEJEykebEmc/f3RE4jpSaRbzojqkdquTYz/6v1Mx1eeTmP00xjzBaLwkwQnZBZACTgEpkWEwOUSW5uJWxEJWXaxFQ2ITjLX16FTqPuGL5rVJvXRRwlOIUzuAAHLqEJt9CCNjBI4Rle4c3KrBfr3fpYtK5ZxcwJ/JH1+QO2uJDJ</latexit>

y1 � y1

x1 � x1

�>Q

y1 � y1

x1 � x1

�� 0

<latexit sha1_base64="1stDpQy7SxKa9NLTSeAdhEKIWLI=">AAACd3icpZE9T8MwEIad8FXCV4ENBixaEAsoYYEJIbEwFokCUl0qx7kWC8cJ9gVRRf0L/Dg2/gcLG26p+N44ydKj996zz3dxrqTFMHz2/InJqemZymwwN7+wuFRdXrmwWWEENEWmMnMVcwtKamiiRAVXuQGexgou49uTYf7yHoyVmT7Hfg7tlPe07ErB0Umd6mOdxdCTuoxTjkY+DIL+dbTLUKoEqEPKGH34VBwGDHTyYb9mmOX0jP7vFsp6cEfDoN6p1sK9cBT0N0RjqJFxNDrVJ5ZkokhBo1Dc2lYU5tguuUEpFAwCVljIubjlPWg51DwF2y5HcxvQLacktJsZdzTSkfq1ouSptf00dk7X6I39mRuKf+VaBXYP26XUeYGgxftD3UJRzOhwCTSRBgSqvgMujHS9UnHDDRfoVhW4IUQ/v/wbLvb3Isdn+7Xjo/E4KmSdbJIdEpEDckxOSYM0iSAv3ppX8+req7/hb/s771bfG9eskm/hR2/5qr0V</latexit><latexit sha1_base64="1stDpQy7SxKa9NLTSeAdhEKIWLI=">AAACd3icpZE9T8MwEIad8FXCV4ENBixaEAsoYYEJIbEwFokCUl0qx7kWC8cJ9gVRRf0L/Dg2/gcLG26p+N44ydKj996zz3dxrqTFMHz2/InJqemZymwwN7+wuFRdXrmwWWEENEWmMnMVcwtKamiiRAVXuQGexgou49uTYf7yHoyVmT7Hfg7tlPe07ErB0Umd6mOdxdCTuoxTjkY+DIL+dbTLUKoEqEPKGH34VBwGDHTyYb9mmOX0jP7vFsp6cEfDoN6p1sK9cBT0N0RjqJFxNDrVJ5ZkokhBo1Dc2lYU5tguuUEpFAwCVljIubjlPWg51DwF2y5HcxvQLacktJsZdzTSkfq1ouSptf00dk7X6I39mRuKf+VaBXYP26XUeYGgxftD3UJRzOhwCTSRBgSqvgMujHS9UnHDDRfoVhW4IUQ/v/wbLvb3Isdn+7Xjo/E4KmSdbJIdEpEDckxOSYM0iSAv3ppX8+req7/hb/s771bfG9eskm/hR2/5qr0V</latexit><latexit sha1_base64="1stDpQy7SxKa9NLTSeAdhEKIWLI=">AAACd3icpZE9T8MwEIad8FXCV4ENBixaEAsoYYEJIbEwFokCUl0qx7kWC8cJ9gVRRf0L/Dg2/gcLG26p+N44ydKj996zz3dxrqTFMHz2/InJqemZymwwN7+wuFRdXrmwWWEENEWmMnMVcwtKamiiRAVXuQGexgou49uTYf7yHoyVmT7Hfg7tlPe07ErB0Umd6mOdxdCTuoxTjkY+DIL+dbTLUKoEqEPKGH34VBwGDHTyYb9mmOX0jP7vFsp6cEfDoN6p1sK9cBT0N0RjqJFxNDrVJ5ZkokhBo1Dc2lYU5tguuUEpFAwCVljIubjlPWg51DwF2y5HcxvQLacktJsZdzTSkfq1ouSptf00dk7X6I39mRuKf+VaBXYP26XUeYGgxftD3UJRzOhwCTSRBgSqvgMujHS9UnHDDRfoVhW4IUQ/v/wbLvb3Isdn+7Xjo/E4KmSdbJIdEpEDckxOSYM0iSAv3ppX8+req7/hb/s771bfG9eskm/hR2/5qr0V</latexit><latexit sha1_base64="1stDpQy7SxKa9NLTSeAdhEKIWLI=">AAACd3icpZE9T8MwEIad8FXCV4ENBixaEAsoYYEJIbEwFokCUl0qx7kWC8cJ9gVRRf0L/Dg2/gcLG26p+N44ydKj996zz3dxrqTFMHz2/InJqemZymwwN7+wuFRdXrmwWWEENEWmMnMVcwtKamiiRAVXuQGexgou49uTYf7yHoyVmT7Hfg7tlPe07ErB0Umd6mOdxdCTuoxTjkY+DIL+dbTLUKoEqEPKGH34VBwGDHTyYb9mmOX0jP7vFsp6cEfDoN6p1sK9cBT0N0RjqJFxNDrVJ5ZkokhBo1Dc2lYU5tguuUEpFAwCVljIubjlPWg51DwF2y5HcxvQLacktJsZdzTSkfq1ouSptf00dk7X6I39mRuKf+VaBXYP26XUeYGgxftD3UJRzOhwCTSRBgSqvgMujHS9UnHDDRfoVhW4IUQ/v/wbLvb3Isdn+7Xjo/E4KmSdbJIdEpEDckxOSYM0iSAv3ppX8+req7/hb/s771bfG9eskm/hR2/5qr0V</latexit>

infQ∈Q

LIQCLN2 (Q) ≥ L2 (1)

I Main result: we can pose (1) as a semidefinite program

7

Page 8: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Main Result

Theorem 1: LipSDP for one-layer neural networks

Consider a single-layer neural network described by f(x) =W 1φ(W 0x+b0)+b1. Supposeφ(x) = [ϕ(x1) · · ·ϕ(xn)], where ϕ is slope-restricted on [α, β]. Suppose there exists aρ > 0 such that the matrix inequality

M(ρ, T ) :=

[−2αβW 0>TW 0 − ρIn0 (α+ β)W 0>T

(α+ β)TW 0 −2T +W 1>W 1

]� 0,

holds for some T ∈ Tn. Then ‖f(x)− f(y)‖2 ≤ √ρ‖x− y‖2 for all x, y ∈ Rn0 .

A semidefinite program for finding the best upper bound:

minimize ρ subject to M(ρ, T ) � 0, T ∈ Tn. (2)

8

Page 9: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Multi-layer Case

x0 = x, xk+1 = φ(W kxk + bk) k = 0, · · · , `− 1, f(x) =W `x` + b`. (3)

I Introduce x = [x0> · · ·x`>]>. We can write (3) compactly as

x = Dx, Bx = φ(Ax+ b), f(x) = Cx+ b`,

where

A =

W 0 0 · · · 0 00 W 1 · · · 0 0...

.... . .

......

0 0 · · · W `−1 0

b =

b0

b1

...b`−1

, B =

0 In1 · · · 0 0...

.... . .

......

0 0 · · · In`−1 00 0 · · · 0 In`

C =

[0 0 · · · 0 W `

]D =

[In0 0 · · · 0 0

].

9

Page 10: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Lipschitz Certificates for Multi-layer Neural Networks

I For multi-layer neural networks

M(ρ, T )=

[AB

]> [ −2αβT (α+ β)T(α+ β)T −2T

] [AB

]+

−ρI0 0 . . . 00 0 . . . 0...

.... . .

...0 0 . . . (W `)>W `

I We can find the smallest ρ by semidefinite programming!

minimize ρ subject to M(ρ, T ) � 0 and T ∈ Tn,

10

Page 11: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Complexity Analysis

I Parameters: nx input dimension, n total number of neurons, ` number of layers

I Number of decision variables:

– LipSDP-Network imposes constraints on all possible pairs of activation functions and has O(n2)decision variables. It is the least scalable but the most accurate method.

– LipSDP-Neuron ignores the cross coupling constraints among different neurons and has O(n)decision variables. It is more scalable and less accurate than LipSDP-Network. For this case, wehave T = diag(λ11, · · · , λnn).

– LipSDP-Layer considers only one constraint per layer, resulting in O(`) decision variables. It isthe most scalable and least accurate method. For this variant, we haveT = blkdiag(λ1In1 , · · · , λ`In`).

I A hierarchy of SDP relaxations

I Size of the PSD cone: nx + n

11

Page 12: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Experiments

I Platform: MATLAB, CVX toolbox, and MOSEK on a 9-core CPU with 16GB of RAM

I Methods:

– LipSDP:I LipSDP-NetworkI LipSDP-NeuronI LipSDP-Layer

– CPLip: Combettes, Patrick L., and Jean-Christophe Pesquet. ”Lipschitz Certificates for NeuralNetwork Structures Driven by Averaged Activation Operators.”arXiv preprintarXiv:1903.01014(2019).

– SeqLip: Virmaux, Aladin, and Kevin Scaman. ”Lipschitz regularity of deep neural networks:analysis and efficient estimation.”Advances in Neural Information Processing Systems. 2018.

12

Page 13: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Experiments

I Training Methods: two robust and one standard training procedures

– Adam: Kingma, Diederik P., and Jimmy Ba. ”Adam: A method for stochasticoptimization.”arXiv preprint arXiv:1412.6980(2014).

– LP-Train: Wong, Eric, and J. Zico Kolter. ”Provable defenses against adversarial examples viathe convex outer adversarial polytope.”arXiv preprint arXiv:1711.00851(2017).

– PGD-Train: Madry, Aleksander, et al. ”Towards deep learning models resistant to adversarialattacks.”arXiv preprint arXiv:1706.06083(2017).

13

Page 14: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Tightness of the Bounds

Figure: Comparison of Lipschitz bounds found by various methods for five-hidden-layer networkstrained on MNIST with the Adam optimizer. Each network had a test accuracy above 97%.

• Our bounds are the tightest in the literature14

Page 15: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Run Time

nLipSDP-Neuron

LipSDP-Layer

500 5.22 2.851000 27.91 17.881500 82.12 58.612000 200.88 146.092500 376.07 245.943000 734.63 473.25

Table: Computation time in seconds forevaluating Lipschitz bounds ofone-hidden-layer neural networks with avarying number of hidden units. A plotshowing the Lipschitz constant for eachnetwork tested in this table has beenprovided in the Appendix.

`LipSDP-Neuron

LipSDP-Layer

5 20.33 3.4110 32.18 7.0650 87.45 25.88

100 135.85 40.39200 221.2 64.90500 707.56 216.49

Table: Computation time in seconds forcomputing Lipschitz bounds of`-hidden-layer neural networks with 100activation functions per layer. ForLipSDP-Neuron and LipSDP-Layer, wesplit each network up into 5-layersub-networks.

15

Page 16: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Reconciling Accuracy and Efficiency

Figure: LipSDP-Network Lipschitz bounds and computation time for a one-hidden-layer network with100 neurons. The weights for this network were obtained by sampling from a normal distribution.

16

Page 17: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Effect of Robust Training

Figure: (Left) Lipschitz bounds for a one-hidden-layer neural networks trained on the MNIST datasetwith the Adam optimizer and LP-Train and PGD-Train for two values of the robustness parameter ε.Each network reached an accuracy of 95% or higher. (Right) Histograms showing the local robustness(in `∞ norm) around each correctly-classified test instance from the MNIST dataset. The neuralnetworks had three hidden layers with 100, 50, 20 neurons, respectively. All classifiers had a testaccuracy of 97%.

17

Page 18: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Conclusion and Future Work

I Proposed a unified framework based on convex optimization for reachability analysis,robustness analysis, and safety verification of feedforward neural networks

I Ongoing work

– Extension to Convolutional Nets– Improve scalability– Robust training

18

Page 19: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

Efficient and Accurate Estimation ofLipschitz Constants for Deep Neural Networks

Mahyar Fazlyab, Alexander RobeyHamed Hassani, Manfred Morari, George J. Pappas

NeurIPS 2019

Page 20: Efficient and Accurate Estimation of Lipschitz …mahyarfa/files/Slides...E cient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab, Alexander

References I

Acıkmese, B. and Corless, M. (2011).Observers for systems with nonlinearities satisfying incremental quadratic constraints.Automatica, 47(7):1339–1348.


A Tractable Approach to Probabilistically Accurate Mode Estimation

A Tractable Approach to Probabilistically Accurate Mode Estimation

Accurate Disparity Estimation for Plenoptic Imagesvigir.missouri.edu/~gdesouza/Research/Conference_CDs/...Accurate Disparity Estimation for Plenoptic Images Neus Sabater, Mozhdeh Seifi,

Accurate Disparity Estimation for Plenoptic Imagesvigir.missouri.edu/~gdesouza/Research/Conference_CDs/...Accurate Disparity Estimation for Plenoptic Images Neus Sabater, Mozhdeh Seifi,

Detailed, Accurate, Human Shape Estimation From Clothed ...openaccess.thecvf.com/content_cvpr_2017/papers/Zhang...Detailed, accurate, human shape estimation from clothed 3D scan sequences

Detailed, Accurate, Human Shape Estimation From Clothed ...openaccess.thecvf.com/content_cvpr_2017/papers/Zhang...Detailed, accurate, human shape estimation from clothed 3D scan sequences

Accurate Patient Charge Estimation a Fantasy?

Accurate Patient Charge Estimation a Fantasy?

Efficient and Accurate Path Cost Estimation Using ... · Efficient and Accurate Path Cost Estimation Using Trajectory Data ... sector accounts for substantial greenhouse gas ...

Efficient and Accurate Path Cost Estimation Using ... · Efficient and Accurate Path Cost Estimation Using Trajectory Data ... sector accounts for substantial greenhouse gas ...

Accurate estimation of microbial communities using 16S tags

Accurate estimation of microbial communities using 16S tags

AccuRate: Constellation Based Rate Estimation in Wireless … · 2019-02-25 · AccuRate: Constellation Based Rate Estimation in Wireless Networks Souvik Sen Duke University Naveen

AccuRate: Constellation Based Rate Estimation in Wireless … · 2019-02-25 · AccuRate: Constellation Based Rate Estimation in Wireless Networks Souvik Sen Duke University Naveen

Accurate range estimation for an electric vehicle ...

Accurate range estimation for an electric vehicle ...

Accurate Estimation of Ligand Binding Affinity Changes ...€¦ · Accurate Estimation of Ligand Binding Affinity Changes upon Protein Mutation Matteo Aldeghi, Vytautas Gapsys, and

Accurate Estimation of Ligand Binding Affinity Changes ...€¦ · Accurate Estimation of Ligand Binding Affinity Changes upon Protein Mutation Matteo Aldeghi, Vytautas Gapsys, and

Accurate halo–galaxy mocks from automatic bias estimation ...

Accurate halo–galaxy mocks from automatic bias estimation ...

Accurate Estimation of Center of Gravity of Spacecraft ...

Accurate Estimation of Center of Gravity of Spacecraft ...

ON ACCURATE ESTIMATION OF TRANSVERSE STRESSES IN ...

ON ACCURATE ESTIMATION OF TRANSVERSE STRESSES IN ...

Fast and Accurate Permittivity Estimation Algorithm … and Accurate Permittivity Estimation Algorithm for UWB Internal Imaging Radar Ryunosuke Souma, Shouhei Kidera, and Tetsuo Kirimoto

Fast and Accurate Permittivity Estimation Algorithm … and Accurate Permittivity Estimation Algorithm for UWB Internal Imaging Radar Ryunosuke Souma, Shouhei Kidera, and Tetsuo Kirimoto

Fast and Accurate Skew Estimation Based on Distance Transform

Fast and Accurate Skew Estimation Based on Distance Transform

ACCURATE ESTIMATION OF ORIENTATION PARAMETERS OF …

ACCURATE ESTIMATION OF ORIENTATION PARAMETERS OF …

Accurate Angular Velocity Estimation with an Event Camerarpg.ifi.uzh.ch/docs/RAL16_Gallego.pdf · Accurate Angular Velocity Estimation With an Event Camera Guillermo Gallego and Davide

Accurate Angular Velocity Estimation with an Event Camerarpg.ifi.uzh.ch/docs/RAL16_Gallego.pdf · Accurate Angular Velocity Estimation With an Event Camera Guillermo Gallego and Davide

Accurate Shielded Interconnect Delay Estimation by ...

Accurate Shielded Interconnect Delay Estimation by ...

GEOMETRY OF LIPSCHITZ PERCOLATION 1. Lipschitz percolation ...

GEOMETRY OF LIPSCHITZ PERCOLATION 1. Lipschitz percolation ...

HLScope+: Fast and Accurate Performance Estimation for ...

HLScope+: Fast and Accurate Performance Estimation for ...

ACCURATE ESTIMATION OF START-UP PULSATING TORQUE OF …

ACCURATE ESTIMATION OF START-UP PULSATING TORQUE OF …