Bethe free energy, Kikuchi approximations, and belief propagation algorithms
This is anupdatedandexpandedversionof TR2000-26,but it is still in draft form. Beliefpropagation(BP)wasonlysupposedtowork for tree-likenetworksbutworks surprisinglywell in many applicationsinvolving networkswith loops,includingturbo codes.However, therehasbeenlittle understandingof thealgorithmor thenatureof thesolutionsit findsfor generalgraphs. We show that BP canonly converge to a stationarypoint of an approximatefree energy, known astheBethefreeenergy in statisticalphysics.This resultcharacterizes BP fixed-pointsand makes connectionswith variationalapproachesto approximate inference. More importantly, our analysislets us build on the progressmadein statistical physicssinceBethe’ s approximationwasintroducedin 1935.Kikuchi andothershave shown how to constructmoreaccuratefree energy approximations,of which Bethe’ s approximationis the simplest. Exploiting the insightsfrom our analysis,we derive generalizedbeliefpropagation(GBP)versionsof theseKikuchi approximations. These new messagepassingalgorithmscanbesignificantlymoreaccuratethanordinaryBP, at anadjustableincreasein complexity. We illustratesucha new GBPalgorithmon a grid Markov network andshow thatit givesmuchmoreaccuratemarginalprobabilities thanthosefoundusingordinaryBP. This work may not be copiedor reproducedin whole or in part for any commercialpurpose.Permissionto copy in wholeor in partwithoutpaymentof feeis grantedfor nonprofiteducational andresearchpurposesprovidedthatall such wholeor partialcopiesincludethefollowing: anoticethatsuchcopying is by permissionof MitsubishiElectricResearch Laboratories, Inc.;anacknowledgmentof theauthorsandindividualcontributionsto thework; andall applicableportions of thecopyright notice.Copying, reproduction,or republishingfor any otherpurposeshallrequirealicensewith payment of feeto MitsubishiElectricResearchLaboratories, Inc. All rightsreserved. Copyright c MitsubishiElectricResearchLaboratories, Inc., 2001 201Broadway, Cambridge,Massachusetts 02139 This is anupdatedandexpandedversionof TR2000-26,but is still in draft form asof its releaseon April 4, 2001.Smallfurtherupdate(section2.3)onMay 15,2001. ! "$#% '& (") * + ,. ' " (") .' /" 0 ( & * 132547698;:<6=4?>A@CB D E<FGE<FG6 H$IKJML NPORQTSVUXW(Y(Z9[VY]\ ^ Y`_ba=UXcdZ9e`f(g Hih O(N=Qkj`l mon W(p=f`qAr=Qts;uvr(N=QwuxsPytz(z {}|;~  ~9 PM|`G wR FGGF6=@C3tD D /6=4 H$IKJL NPORQTSVUXW(Y`Z=[VY]\ ^ Y`_ba=UXcZ=e`f`g Hih O(N9Q;j`l mKn W`pRf`qAr=Qts;uvr(N9Q}us`y`N's  d|}|`'t|P }= B 6=FD F; ^ W`_= 9¡Xf}U¢=£wcf}pR£}f¥¤c¦'cd§ ̈c©W(p a ^ SfkU ̈«(f}¬f}\(g'z'­(y¢ W9Z=Y®MY`¬©¬ SVf}UX«`fk¬©fk\`g ^ h l`z sPN`Otu ̄Q]s(str mon W`p=f(qAy=QkOPu°r`z'N]uyPO'NPl {k±o|t 2x2 ̄Cw2k3P|P ́μ|P |}{k |;~=¶ ·1 ̧To`»91⁄4=1⁄2A3⁄4¿» SVf}¬c©f}ÀM=UXW`RY`e(Yt¡Xc©W(pÂÁS mVÃ [ÄY`§ÅW`p=¬\/§ ̈ ==¿W(§ ̈fkZ ¡XW![W(U ̈«ÆÀÇW`U ¡ ̈UXf}f}u¬c«`f p=fw¡[VW`UX«9§ a= 9¡<[W(U ̈«9§3§ ̈ =UX=U ̈cd§ ̈c©p=e(¬©\[fk¬©¬(c©pb_Y`p \YPR=¬©cd£}YP¡ ̈cW`pR§Èc©p ¦(W`¬¦'cp=e p=fw¡[VW`UX«9§0[Éc©¡ n ¬©W W(R§}gKc©pR£}¬© 5Z9c©pReÆ¡X =UXa5W!£wW9Z9f;§}Ê/®MWt[fk¦`f}U;gC¡ n f}UXf n Y(§ a5fkf}p%¬©c©¡x¡X¬©fË =pRZ9fkUX§x¡XY`pRZ9cp=e W`À ¡ n f!YP¬e`W(U ̈c©¡ n _ÌW(U¡ n fÆp5Yt¡ ̈ RU ̈fÆWPÀ0¡ n f §xW(¬© =¡ ̈cW`pR§Äc©¡ÉÍRpRZ=§ÉÀÇW`UÉe`fkp=f}U ̄YP¬Îe(UXY` n §kÊ Ï f § n Wt[-¡ n YP¡S m £kYPp/W(p=¬©\ £}W`p ¦`fkU ̈e(f¡ ̈WÆYÆ§¡ ̄Yt¡Xc©W(pRYPUX\Æ¿W`cp'¡ÅW`ÀMYPp YP=RU ̈W]Ð9c_YP¡ ̈fMÀÇU ̈fkffkp=f}UXe`\(g'« p=Wt[Ép Y`§¡ n f¥SVf}¡ n fTÀÇUXf}fTf}p=fkU ̈e(\c©p §¡ ̄Yt¡Xc§xu ¡ ̈cd£}Y`¬3 n \9§xcd£}§kÊÑ n cd§UXfk§ ̈ =¬©¡M£ n YPU ̄Y`£w¡ ̈fkU ̈cÒ}f;§VS m Í=Ð9f;Z u¿W`cp'¡X§Y`pRZ _YP«`f;§ £wW(p=p=fk£w¡ ̈cW`pR§Ä[Éc©¡ n ¦tYPUXcdYt¡ ̈cW`p5YP¬ YP=RU ̈W'Y`£ n f;§o¡XWY`==UXW]Ð9c©_Yt¡Xfcp9ÀÇfkU ̈fkpR£wf(Ê H W`UXf¥c©_¿W`U ̈¡XY`p(¡X¬©\(g=W` RUYPpRY`¬©\9§ ̈c§Ä¬fw¡ ̄§É R§Éa= Rc©¬dZ W`pÓ¡ n fb=U ̈W(e`UXfk§X§K_Y(Z9f c©p/§x¡XYP¡ ̈cd§¡Xc£kYP¬A n \9§ ̈c£k§0§xcpR£}fSf}¡ n f(Ô §ÅYP=RU ̈W]Ð9c_YP¡ ̈cW`pÆ[VY(§¥c©p'¡ ̈UXW9Z9 R£}fkZ c©p Q;l`j'y9Ê<Õ¥c«  R£ n cRY`pRZWP¡ n f}U ̄§ n Y]¦(f§ n Wt[Ép n Wt[)¡XW0£wW(pR§¡XU ̈ 5£μ¡K_W`UXfY`£wu £w =U ̄Yt¡XfKÀÇUXf}fÄfkp=f}UXe`\0Y`==UXW]Ð c_Yt¡Xc©W(pR§}g]W`ÀR[ n cd£ n SVfw¡ n f`Ô §AYPR=U ̈W]Ð9c_Yt¡ ̈cW`p c§T¡ n f§ ̈c©_=¬fk§x¡kÊ I Ð9=¬W`c©¡ ̈cp=eÓ¡ n fcpR§ ̈c©e n ¡ ̄§TÀÇU ̈W(_ÖW` RU¥Y`pRYP¬\9§xcd§kg3[fZ9f}u U ̈c¦`fVe`fkp=f}U ̄YP¬cÒ}fkZ0a5fk¬©cfwÀ¿=U ̈W(RYPe'Yt¡Xc©W(p×ÁGØ¥S mÃ ¦(f}U ̄§xcW`pR§ÈWPÀ5¡ n f;§xfÉÕ¥c«' 5£ n c YP=RU ̈W]Ð9c_YP¡ ̈cW`pR§kÊ Ñ n f;§xfÆp=f}[Ù_fk§X§XYPe`fiRY`§X§xcp=e Y`¬©e(W`UXc¡ n _§×£kYPpÚa¿f §xce`pRcÍ5£kYPp'¡ ̈¬\_W`UXf¥Y`£}£} =U ̄Yt¡ ̈fT¡ n YPp W`U ̄Z9cpRYPUX\S m g=Yt¡MYPp Y`ZtÛ R§¡ ̄YPa=¬f¥c©p=u £wUXfkY(§xf¥cpi£wW(_ R¬©f}Ð c©¡\`Ê Ï fbc¬©¬ R§x¡ ̈U ̄Yt¡ ̈fÅ§x R£ n YpRf}[ Ø¥S m YP¬e`W(U ̈c©¡ n _ÜW`p Y0e(U ̈cdZ H Y`U ̈«(Wt¦bp=f}¡[W(U ̈«YPpRZ×§ n Wt[1¡ n Yt¡Vc¡Äe`c¦`f;§K_Å R£ n _W`UXfMY(£}£w RUXYP¡ ̈f _YPUXe`cpRYP¬Î=UXW`aRY`a=c¬©c©¡ ̈cfk§V¡ n YPp ¡ n W(§ ̈fÀÇW( =pRZ  R§ ̈c©pRe W(UXZ=c©pRY`U ̈\S m Ê Ý Þ(ß »91⁄4=àÉáMâM3⁄4¿»=ãxà ß L W9£kYP¬Mä ̈a5fk¬©cfwÀ=U ̈W(RYPe'Yt¡Xc©W(pRå ÁS mÃ YP¬e`W`UXc©¡ n _§M§ ̈ R£ n Y(§¡ n W(§ ̈f0cp(¡XU ̈W9Z9 5£wfkZ$a'\ m f;YPUX¬ÈYPUXf e` 5YPU ̄YPp'¡ ̈fkfkZ ¡XW$£wW(p'¦(f}UXe`fb¡ ̈W ¡ n f£wW`UXUXfk£μ¡¥_YPUXe`cpRYP¬A¿W(§x¡ ̈fkU ̈cW`U¥=UXW`a5YPa=c¬©c©¡ ̈cfk§¥cpË¡ ̈UXf}f}u¬c©«(f e`U ̄YP n c£kYP¬5_ W9Z9fk¬§kÊAæ=W(Ue(f}p=fkUXY`¬Rp=f}¡[W(U ̈«9§A[Éc©¡ n ¬©W W`5§}g'¡ n f§ ̈c¡X RYt¡Xc©W(p×cd§K_Å R£ n ¬fk§X§£}¬©f;YPU;Ê ç p1¡ n fiW`pRf n YPpRZ3gMY!p  =_ba¿f}UÓWPÀ U ̈f;§xf;YPU ̄£ n f}U ̄§ n Y]¦(f f}_=cU ̈cd£}Y`¬©¬\)Z9fk_W`pR§x¡ ̈U ̄Yt¡XfkZ1e`W W9Z ¿f}U ̈ÀÇW`UX_Y`pR£wfÀÇW`UÉS m YP¬e`W`UXc©¡ n _§VYP==¬cfkZ×¡ ̈W e`U ̄YP n §[Éc©¡ n ¬©W W`5§}Ê ç p=f Z9U ̄YP_Yt¡Xc££}Y(§xfTc§ ¡ n fÉp=fkY`U¢ n Y`p=p=W`p=u¬c©_c©¡o5fkUxÀÇW(U ̈_Y`pR£wfÄWPÀÄä ̈Ñ< =UXa5W £}W Z=fk§Xå=gP[ n W'§xfZ=fk£wW9Z9cp=e YP¬e`W(U ̈c©¡ n _ c§ fkè' =c¦tYP¬f}p'¡V¡ ̈WS m W`p Y0¬W W` \e`U ̄YP nÆé j9g Q;O]ê°Ê ç ¡ n f}UÉ§ ̈ R£}£}fk§X§ÀÇ R¬Î£kY`§ ̈fk§c©p5£w¬ RZ9f¥£wW`_= =¡ ̈f}U ¦ c§ ̈cW`p =UXW`a=¬f}_§ÅYPpRZ!_fkZ=c£kYP¬Z=cY`e`W`p5§xcd§ é j=gAN gQ]N;êV§ ̈ =e(e`fk§x¡ ̈cp=e ¡ n fÓ§ ̈ R£}£}fk§X§0W`ÀÉS m W(p ¬W'W( \ie`U ̄YP n §¥c§0p=WP¡0¬c©_c©¡ ̈f;Z!¡ ̈W$£}W Z=c©p=e$Y`==¬c£kYt¡Xc©W(pR§}Ê ç pË¡ n fW`¡ n fkU n Y`pRZ g<ÀÇW`U0W`¡ n fkU e`U ̄YP n §V[Éc¡ n ¬W W`R§kg=S m _Y]\×e`c¦`f¥5W W(UÉU ̈f;§x =¬©¡X§ÄW(UÄÀYPc¬Î¡ ̈W£}W`p ¦`fkU ̈e(f é Q;N;ê°Ê æ=W(U Y e`fkp=f}U ̄YP¬e`U ̄YP n gM¬©c©¡x¡X¬©f n Y(§×a¿f}fkpÂ =pRZ=f}U ̄§¡XW'W9ZÚY`a5W( 9¡Ó[ n Yt¡ YP==UXW]Ð9c©_YP¡ ̈cW`pÚS m UXf}=UXfk§ ̈f}p'¡X§kgoYPp5Z n Wt[Üc¡_c©e n ¡ a¿f c©_=UXWt¦`f;Z Ê Ñ n cd§ RYP¿f}U;Ô § e(W(YP¬Vcd§ ¡ ̈WË=U ̈Wt¦ cdZ9fÓ¡ n YP¡ =pRZ=f}U ̄§¡ ̄YPpRZ9cp=e!YPpRZ c©p'¡XU ̈W9Z9 R£}f YË§ ̈fw¡WPÀMp=fk[ÜYP¬e`W(U ̈c©¡ n _§ÅUXfk§ ̈ =¬¡Xc©pReiÀÇUXW`_ë¡ n Yt¡ =pRZ=f}U ̈u §x¡XYPp5Z9c©pReRÊ Ï f×§ n Wt[ ¡ n Yt¡ S m cd§ ¡ n fÍRU ̄§¡bcp Y =UXW`e`UXfk§X§ ̈c©W(piWPÀÄ¬W £kYP¬o_f;§ ̈§XYPe(fwu°RY`§X§xcp=e YP¬e`W(U ̈c©¡ n _§kg3f;Y`£ n e`c¦'cp=e fkè' =c¦tYP¬f}p'¡0U ̈f;§x R¬¡ ̄§T¡ ̈W$Y £wW(U ̈UXfk§ ̈5W(pRZ9cp=e YP==UXW]Ð9c©_YP¡ ̈cW`p ÀÇUXW`_ §x¡XYt¡Xc§x¡ ̈cd£}Y`¬5 n \9§ ̈c£k§V«'pRWt[Ép×Y`§K¡ n f äXÕ¥c©«  R£ n cdå0Y`==UXW]Ð c_Yt¡Xc©W(p¡ ̈WÅ¡ n f ØTc©aRaR§ÀÇU ̈fkffkp=f}UXe`\(Ê Ñ n f;§xf YP¬e`W(U ̈c©¡ n _§ n Y]¦`fÓ¡ n f YP¡x¡XUXY(£μ¡ ̈c¦`fÓ=UXW`¿f}U ̈¡\ËW`Àa¿f}cp=eË R§xfkUxuvY`ZtÛ R§¡ ̄YPa=¬f`qÓa \ RY]\'u cp=e×§ ̈W`_f0Y`Z=Z=c¡Xc©W(pRYP¬7£wW`_= =¡XYt¡Xc©W(pRYP¬<£wW(§x¡kg5W`p=fÅ£}YPp W(a9¡XY`c©p$£}W`pR§ ̈cdZ9f}U ̄YPa=¬f¥c©_=UXWt¦`fk_f}p'¡ cp ¡ n f$Y`£k£w =U ̄Y`£}\ËW`ÀW(p=f`Ô §YP==UXW]Ð9c©_YP¡ ̈cW`p3gKY`pRZì£}YPpì§xW(_fw¡ ̈c_fk§W(a9¡XY`c©p1YÆ£}W`p ¦`fkU ̈e(f}p'¡ _fk§X§ ̈Y`e`f}uRY(§ ̈§ ̈cp=eÅY`¬©e(W`UXc¡ n _í[ n f}p W(UXZ9cpRY`U ̈\S m Z9W fk§Äp=W`¡M£wW(p'¦(f}UXe`f(Ê î ï!ðÈñ ã ð3òÅó 1⁄4=à ó 1⁄27ôK1⁄2<»=ãxà ßöõ÷Éð áÄø ó àKã ß »=o 1⁄2C1⁄4 ðÂù ð 1⁄4=à ôo1⁄4=1⁄2Aáã ðÈß » ó àKã ß »=oËà ò »9ú ð?ï!ð »9ú ð ò 1⁄4 ð3ð%ð<ßð 1⁄4=ôoû æ=W(U ¡ n f= =UX5W'§xfW`ÀÄY`pRYP¬\ Ò}cp=e YPpRZ!Z9f;§ ̈£}U ̈ca=cp=e S m [VfY`§X§ ̈ =_fÅ¡ n Yt¡Å[fY`U ̈fe`c¦`fkpËY`p =pRZ=c©UXfk£w¡ ̈fkZe(UXY` n cd£}Y`¬9_ W9Z9fk¬=WPÀ3üýp=W9Z9f;§o[Éc©¡ n RY`c©UX[Éc§ ̈fÄ5W`¡ ̈fkp(¡XcY`¬§MÁY RY`c©UX[Écd§xf H Y`U ̈«(Wt¦ J Y`pRZ9W`_öæ7cf}¬dZ Ã Ê h p \e`U ̄YP n c£kYP¬¿_W Z=f}¬3£}Y`p×a¿f0£wW(p'¦(f}U ̈¡ ̈f;Zc©p'¡XWÅ¡ n cd§VÀÇW`UX_ía5f}ÀÇW`UXf¥Z=W`cp=e cp9ÀÇf}UXf}pR£}fÄ¡ n UXW` Re n Y §x =c©¡XY`a=¬f£w¬ R§¡Xf}UXc©pRe WPÀÎp=W9Z9fk§Acp'¡ ̈W ¬dYPUXe`fkU7pRW Z=fk§ é Q]skê°ÊAÑ n UXW` =e n ¡ n c§ ¡ ̈U ̄YPp5§ÀÇW(U ̈_Yt¡Xc©W(p3gÈW(p=f£}Y`p/YP=R¬©\ËYP¬¬o¡ n fU ̈f;§x R¬¡ ̄§ c©p/¡ n cd§0RY`5fkU ¡ ̈WÆYPUXa=c¡XUXY`U ̈\ie`U ̄YP n cd£}Y`¬ _W9Z9f}¬d§KcpR£}¬© RZ=c©p=eb¡ n W'§xfM[Éc¡ n×n ce n fkUxu°W`U ̄Z9fkUA5W`¡ ̈f}p'¡XcY`¬§kÊ Ï f[Éc©¬¬5e(c©¦(ffwÐ9=¬cd£wc©¡fwÐ=Y`_ R¬©f;§ WPÀ7¡ n c§Äcp ¬YP¡ ̈f}U§ ̈fk£w¡ ̈cW`pR§kÊ Ñ n f§¡ ̄Yt¡XfÅW`ÀKfkY(£ n  =p=W(aR§xfkU ̈¦(fkZ p=W9Z9f þÉcd§Z9fkp=WP¡XfkZia'\ ÿ xg3Y`pRZ$[VfÅY(§ ̈§ ̈ =_fbf;Y`£ n  Rp=W`a9u § ̈f}UX¦`fkZpRW Z=fMcd§£}W`p=p=f;£μ¡XfkZ¡XWÅYPp×W`aR§ ̈f}UX¦`f;Zp=W9Z9f xÊAÑ n fKÛW(c©p'¡V=U ̈W(aRYPaRc©¬c¡\Z9cd§¡XU ̈ca= 9¡Xc©W(p ÀÇ =pR£w¡ ̈cW`p cd§Ée`c¦`fkp×a \ ÁÇÿ xÿ  ̈ÿ Ã Q ÁÇÿ xÿ Ã Áÿ Ã ÁQ Ã [ n fkU ̈f  ̈Áÿ Ã c§T¡ n f¬©W9£}Y`¬ äxfk¦'cdZ9fkpR£wf;åÀÇW`U pRW Z=f þ ̄g (ÁÇÿ  ̈ÿ! Ã cd§T¡ n f£wW(_RYt¡Xc©a=c¬c¡\ _Yt¡XU ̈c©Ða5f}¡[fkf}p p=W9Z9fk§ÄþYPpRZ "5g=YPpRZ c§ÉY p=W(U ̈_YP¬cÒkYt¡Xc©W(p×£wW(pR§x¡XYPp'¡;Ê$#vp [ n YP¡VÀÇW(¬©¬Wt[§kg [Vf¥§ ̈c_ R¬©c©ÀÇ\×p=W`¡XYP¡ ̈cW`p YPpRZ [ÉUXc¡Xf XÁÇÿ Ã Y(§É§ n W(Ux¡ n YPp5ZÀÇW(U XÁÇÿ Ã Ê Ñ n f0§x¡XY`pRZ=YPU ̄Z×S m  =ÎZ=Yt¡Xf U ̈ R¬©f;§ÉYPUXf`q % (Áÿ! Ã'& (*),+.'ÁÇÿ  ̈ÿ! Ã XÁÇÿ Ã /.0 21 3546 % / XÁÇÿ Ã ÁGN Ã 7  ̈ÁÇÿ Ã'& (  ̈Áÿ Ã /.0 21 3 % /  ̈Áÿ Ã Áj Ã [ n fkU ̈f ( Z=f}p=W`¡ ̈fk§ÄYÅp=W`UX_YP¬c©Ò;Yt¡Xc©W(pÓ£wW(pR§x¡XYPp'¡ÉYPpRZÓü/ÁÇþ Ã 8 " _fkY`pR§ÉYP¬¬Îp=W9Z9fk§Vp=fkc©e n a¿W`UXcp=e p=W9Z9f þ ̄g5f}Ð9£}f}9¡*"5ÊT®Mf}UXf % ÅU ̈f}ÀÇf}U ̄§É¡XW¡ n fÅ_fk§X§XYPe`f0¡ n YP¡Tp=W9Z9f þÉ§xfkpRZ=§M¡ ̈WÓp=W9Z9f9" Y`pRZ 7 7cd§K¡ n f¥a5