From e13264862bbb2131684860fb389a0fee4761ce1f Mon Sep 17 00:00:00 2001 From: Heidi Steen <1612796+HeidiSteen@users.noreply.github.com> Date: Thu, 9 Jan 2025 16:59:44 -0800 Subject: [PATCH 1/7] Vector index size by algorithm --- .../vector-index-size-by-algorithm.png | Bin 0 -> 22997 bytes .../vector-search-how-to-create-index.md | 14 ++++++------- articles/search/vector-search-index-size.md | 19 +++++++++++++++--- 3 files changed, 23 insertions(+), 10 deletions(-) create mode 100644 articles/search/media/vector-search-index-size/vector-index-size-by-algorithm.png diff --git a/articles/search/media/vector-search-index-size/vector-index-size-by-algorithm.png b/articles/search/media/vector-search-index-size/vector-index-size-by-algorithm.png new file mode 100644 index 0000000000000000000000000000000000000000..e49ddcfba65a6caf2c22f467a5b2504bd31654ff GIT binary patch literal 22997 zcmdSB2T)U8)F_GtumLs@k)l!*kPbrVic&)F5V~|i51|K8KLL>9kqy!Q=1f;*9-~IphX8!r--MRO^nRjL~?6c24d#|&wl4rbT~H1FN8Vj8F?A0t4UhBx$s-rxV^IF_jPe6kEWoImh*MDvIg6F z-G60k@8~M?XuTf&=)R+k%p*fFbwPD^1zQJ4Wq(gwU4IQdYk#mcz~+&hEX@^ZUrF)| zF1B7)_kCTQT_KXbG7tYuEJ<#kbqhR{mG-o;lhk>k`0ohvoyE9kbM>hVy|p$#Q0uWiP-Q73E5cjS=opP@!5!o*ocb>NeJ5sy?XfX@?MU1|GB;^ zpv)S zMyCIuuB|JXq&{TYJxNHuNbs`BEQp0D}pae%MhLgVQvS`erO7(0KH<1+_ON&#gK zWlmS~O?hFi?z%M*27D5^Fdy9QB*RNkPx$iWdEyF6qh71)ds#?M6V54i1j%$@sat#kx2s0@^SNTCZK-o9@PM?_d10{S>V! zAJQ)TZHD;DTVDIy^mzY`@o($b&DTQzj^sRV{kIjadd8Gz*Y%`Jk>~!j?(^SZ{5y}l z=|jOYZl$0I`TF92enP~4pll|@a?nE$&9~vSiS6XMH_!&SDWO0uwie672c`8!!PieX z+P~cbnfU9mxHaj>G(pO(nGtWwWM@9X*ZuXkk=CsX;!C^QHHBB@{=&rUtN(%-TsYN_ z8;|t-&~P2L&8t@1g3@e6riS6hU*5S68YZ%yDIMt#jsL@I<}nw>`8AF%ka^Q zcd{q>5T`#VU5b2AiqeZ=MGk-dl^oPWyvJ!a`Md=L$QKgR9AIN57WJ;V6`Ihp8RGKg zMtX5_vuRiCTWa#5J4UD`_dtzdW#CseLvvVE>1ObRMc&2XB`@gGjs>$E-}ltB`LE*~ zfbS)C-H@<_%&w!2hcXhoPRA#UZab6b%RI}jjV28In!E+7Khm{b`)y!sT{ax&ngqy~ zx0XCAO+;=k@!~h?oR$Y-6mdlwJ+o&MhJ1ZydCj9pF@u!(xKR%Q*mT0G)IM=<8fIdQ z=QuWTJ6)GOb}$s7KRS|>nG$LSoMXbALHqmMG3C)+Cgzx78p-%yd@@7Z z8HU5jN`a@CJ<$W77hc*Blp`z!As(O5XNq%~E}~^)zn=JU42W~_d?pk#kvltGuuteiPM05!b-#tr)Q< zBKpB!SL+k4`m6d|=0IKgP}ACAsRd?Z-Z`Vx*H19{LC?C6{a0CQGHTb1U4q@hs?k9f zQ(`xykM#xJ#thqwp*voJY~!0{>AIO&TQxgfssf?M4f<np90`!BI>uTkzgtUox<3(kP(2d#mugdw$ z)5zLKi$BN6NbrEWsc6i1D=L@@y9QNy4gF29VuZyP_yT|Bp^lx!xINxR*Z`oEwX{id z+jAgC3@nfgdh3yUjARmNM)LE?4Ms1lXHpfIn(694nqQSK~b+Ij)btqnycgF%1L1Q)C7C(J>wRBJ7yK8|E#kJ2#dlr)faen=O$+@*F`prsGD2pLE2!~9f zCdOij7c6}M9G<9bV{}uUVe>FQ6XTE`W6P5F`>LHLI*Wpp`>!6+Ff%oAPg>aR^@V@u zPfXML!Bp)Uk+I(=WVKv2v|WlT#2i_!73eFBIyqTAaciDUAg0v&JGdxr1h5ef67wef zDMjC~>B{ZDGoTz1z%Ngy~oP z92>tDL1{YF4ahlR@-)sx4}_C9{y2rxFQ{W*uOw_oDoQZslrB>++R9Kx0qdBta>sOj z;%@DLsrB!H%{Cpd6t&&~FDb2BiLUXkbl4pmsw8fegAE@h4-AN=@<}!k!JT!k`s*=JelB+v#XN!SI9?#Ubll5^QFn1E@V|1n5BD2StL7}Df?RuSf}rkw;x6(-eJAhvPaCx z4Zcs_irZ&$Cx0X%{<@sY<`mOQnW}wY1`7*?!bjj2!D_3)0Z3V|i&w6h%~vR9bkFdG zP2XU8UbkEK?Vc0MJ7e=ZNp(f0tRi*RWh6Tf=pB>zXVy|dS=1$y_d38RrGTT?PersU z*4|k-E=J;10MS2zGUIQoi9JDDYH_A4{INwE#OYAwM1ke`jKQs?o4KPhJYXW+Q(pry zPp5d0f*(|ngANUe#1J!NW)*l0%<9H^(~WvwYnatBihchq7Ih6GV;W$F*vWd7XV z3O&NP41xKX4mA6??Ef(LTYheJT=gscm_a>7x(UI2B2`b0z@4lmn^TWML*? zxoeBI6|kA3cjUbRkO{{*DWYZ|!H&w)&o7_l=mwi(xW|&Cr8c&ETw3 zf7RaoJs&a~@L&^(n1fCHwc-!Sy|2kkIk4w_5i;@4t0N@5>$I?iP%tm5Qm12Bm5|_7 zoI=Fx*XoK(;gfQz4aYpg0CKNQU(KdY#5ta-cU>pwj_l*SWl4 zPc?eJbnP>LVhHFX+8Z?Yb+>MVg^E$emFndbZN8?(@Xncf8YhAzPc>jh$&){y#kRIW zuHTO5;N&6pBzo$zj0VG5eWFX=&Vd?^zDm28IBE1!O(e>f3P6IDq%#Qui;(-@GF1h} zyzXc-2N`0jgYr}d{i)8I;~GE1y*4uaaYBxIWDddz%9ymxtNl?sArd#+UzeYK6Sp?} zYSrYrzi@nA-}}_BJ7+e+WqU}Ph{OR98Z&v+o&!5pS(AypPN$gKtVYA9pETy!hO__y zMMsr(lBvp})3of@$-=_hJ;fjVeJ=>z|EAkhMx>=~W!tpf2Fd7NK4XrysN^|aWxn?| zm^dxIM?}+_mb~%k^c4FEw_#QdF8egw`^XV(o@&-RccTUH<#fSf{6bbP7{#wv*`3Fc z^%5fHHe-4jsI|R_9d7V@L$=zmCZdMhs%g67(Z?o9XNr~^>rxCKF6>ozD(kIk8E3cI zH_`Ub^;!;aMDFPOm0KfvN%_#kg@0(?{E!eh+pE@Es=#!)-ej@Dl1A;&CGqZ?pO$h0 zEcV2?xAC@J@I?mV1EtAMjX8_aqBK!tDnmC;)QnQmlcWeq%?MmKra>)4gNm)z0Rh+* z@@u#(%9Q8^Cm%mi=fFqoKRsh6mu9!#|7_Zy)r>N~p3P+~SH{sHHt&R724z3Gvp)F8 zn!jJ+rs=SHx~7Xmigg{t6qDTBFXC_8cU%hHzqRSZbXr&4iyrirB{lru(^P~7Tt==k zf6S5DZLk~MUCI()Rd-26?3~+kS_#hp>X?vNJ<8VVa*;dJ8Fk>5#slpl2qE-3qJ?oNme7D9FF2>SFVmtny7d zUF>+3`d2zlL^6Bv5L#!+XhIF8IbRdN*j8aeLg~Ey+W$B|6lf2i_)zn5lFQNSrYmV( zJ5jX$yWn3DbD%^0H>}J7sG#8Og`W5B27#q=d;%jGi8X|itmuq8*;`A;E@L@Ba#3r^W64z@FD6=&#nxE=GHaJUp`67x)<{u0KMEf%Arj zh@!qr>2*h2nDp#3UikD(%)xRchBuMemBTUZ54JG#x&iN}pnz_#*5C-!KDd57H63A1 zJNrL}?-YIhb?eP5iV*+$rOzL{h)9^e_U3!Y?K9Lxzfwh6k{tqPk3~HDU*73wW+Qih zGk$iP>_j=c{!33qZr(k{!^Hvv{!^pnm{o&HrN~~N#o*7z!BR!>J&u2PmLf#bDeiT- z0PksGQ;Z+Ua+nhVv8a$eTtFSHBz5}N8@`z_Umwoh-s{?qg7E)c>%Obj z9ARq?Od8JtiwWmO#Z8!&<~(&7%67ZcxLsU~${}6O75J?ekhYzn8$2p9w>}stPA8!x zT89*v%3ry2vc~+^%@Xn$VPv`Tm7z?!G`Ng8XN;!X-RD-tE*nU$z zk$0)ZH2<5YeG~VoPEC{Lsa!###)T)?XXG}vhM`Wj$EC~rDLc&*Ol%8wMdF)m@Ncey zU1)gK>ECU18YQnjYF_Gb2bPOlFXAga;pT(w;Rs7koJkP9cz3v>?0!*11}k#64%m`e zjej(G@EwMz-2$m^;rRpntEz+rL?sr?^6my7zfQre*b}VTz z>Z*gJ9*T)`$P?SF9SEeZ5SEL4hY%9c6rqcJFI;v*&19a13Xm|>G6-9x{!;f_ogPLe}-9mTXiZ+0cLrl~CK4E>pfgUr=9|C!v`PBLVtcTtts zsH;e2)H$+p;jCWqLqRr8oS75)On`yA4u?&w6)^M=zCC-`{B-DCyDV5vSczQ z+7h4JG6NRttU$IktS^}b_kj;nL930xmSETY=nT2R9N7UmgV5s3)PFF#Zg%Txq)Oe+ zLo<)99O0|7BW^P7&;iM{)H(lv<95Op8n@KEY3Mk=FcKSeKgY1^$WlH(gZ7BT!lJ>7 zJoVyOMf2=~I`6&!HUZQ5P8I3-zYdqjO9BtN1LOzv34=r)ML?J#+@wF3yb|psNwIzZ@gf`_-k)rah&M!^W%u{Y99?l z0cx*r_e!a7WF8#2oATE?8Ll3GIdcE)tCZ}rVYWJyUJ_dlp|QP=U?I2u>4!p7Ha^rx zYoVm{fN-afY5Zw3dFkwJeM6PbL}_ChQ+B?z=uSId*m=-vr17-wd>Oi^^;)v`=f)8R?V)?Y!YEKRHuH zZv_2n{ZHg^gHoW{t<2f&xZw}R*_cH1iaOTqn_aZww3s3;zVeY8d%p1}L!byc<@EG| z+oG|gnxH?b+q;ME4=TS#05p+ieb0nc2KGpq>}3P9aKA3r;7q#)M~-H;{rO_V5T1pv zVi9w&aBkY43srM4ztmZrIDQicwrPrZ?vD~3?F^8|5f_*ZgJD65WwV`$E(eXG$Y#&T z&Lz^;LLf~NyWC_~@pDPxKkHG1zx-=om6ZAIYdZSRu=g}u!IsCfO%j80rTE7_84W^z zVx_T1U<(}@Rzd_mh7z6&;&5Ndnlw$$NC>8bI!mdf70#IwLZ z7WB?N*)T!`tv`J1UCrVPNjb@CQ?L7rtGld((rckK7sZDMhoX=L6k=zqm+A4c>>t3s z?SJ@Yl=hWS8c5%GR1|S3;_sZw|6$~mF^LL}lo*S^4p(b>n#m^h%A^}wztRqy_V#S( zr)Ki!`-vDyvhi?D>P#t-w*LKYYdaC3J=$lB^!@zTMGs4t@OXZ2q%k)lLUSnO10_Z4 zdpj(GoBTEntA9Q}`p6USaQ1?O77f$Cu>p#nJ2VpsH_v)0C?qVd|L+MI{8x6%|56O% z|9=yP=$gAsE)H|!H$;A#N$V_^ibjD}pwW_tOjd^>TrL)|inRj?K|ZmFZ}h&bQk?76 z$Sw14@013wM(%5x8zTJii3JmY8PB&Wy_3I~DJj0~V?61qg8b>a^>(vg@*o>RGL?M%Bx&2p)D8j&((W&KxoC8D|9RHszyCb&T~l8Tow`c@bXKo$ zk7oQO26ir`MD>B6mv4#wp1CUiZEWYfHN!BNb#Q9n>L#Zp+Pofbc<=!tNI?;o^8!Uk zD4Eyj1g;HfNSbE3V79~nm8l0Zpw9#*{Eo*8j-JO{@TD;zd%<+s>T{ScdxTc7ajdoL^UqRQb1)a(Vkk;+8jFYy=^puvdQbUHf`Ut*3!4$0=GC-iB^)K&RJH&B=^ zO@`%hYywY3)GFfQLH91)0u`J69Op7#6@c{6HyiU&fB&#|2q~s5bV|C9st3HhYR9wsn^#pII z_r#LC49eDDkOFpkc@BT}B8CO}0bA$Yv)tGaaV0NL?31G=Hnw`_<8p|>dAYCcQ=>q> zniY`hjUFe!b@K*X=B-1l!7}v%53CB!KQxfNOx0Et0NzOU9*E`ie?vYb%1f&%>vi=3 z{9#QdNDe=SZo^J=`mJ;F); zrh!{el1o3sdN#fDci{!@T}C9LZM8{LVbYXtM3uu&*!Xk4r`)U2sOX-9`nq+KbdDTQZ>Q^iEI#dhYA z%bw}!#+3Z6&;e=dnY*dcj`&~POj9!I%cZVNE9i9tH~rbxXKg-4>y_u)tF}p|wxm@C zD;RQb#hM4^UsCHbJrv;FRGlJ@A7~KGrRx;mW}ydrj*)x{gwO$>{LntRg?rz)d1QZO z{!HdEl$~XiGk}a3-!eEH6;Ky>N=Y>_I*F!D^^97WcOi#%k{&-AG<7XgTX-T8{96Qq zQ8%|eB62g{PNsIyIiN|BdZx4Q1RpD{6m~J1gC~@9H!m-KnJ9&{6v8+W;{Dn|d}GFk zBs0j1Nd~ppNpzs0IEhv0<;%$!<3!is6M-I)F3v;fz1vSFj}%_Z*p;KOcP;1)@TI9s9V z%wjq7Qb0BI(k=Ec(pUA?5oIgp&9vv7Vyqcg7BKg11;&cft)%wC&6{4_eEfm$&~sO^ za^p)0x&(l0ft9eZFjol-`UR%OaWON-5Ai5e2B$w5PH>#b#;{KrU7O5E{rqxLqqx1V zpGnNFN?1pu(9^o9B2NM@mG`TG4Lov~F6wJp2U<>+F4NQ=iQn(KE2L|y@^)cuT0?R{ zU!_@-)XKH}WmvDkVa27(&o69f@;9wd9eLNenWQ`SS;%M1P_Nw7!T_FKL<|f3C6k!L z8wQt>kEOkR1wyW*HOrX`)EWa3Z%_PU3U~V195)nRPV8=Sqt=Nd%S8FTas0rb7$(i(7x4N2l7)&F%D#plQsV~AiM54XH4{H+G-rXYX486?BV)KA;;p>?&GcF zt77rgJtmir7kUzMgyBa&wssDisgUCWg_S062XAjf%^MvPg+tY~`=#h5r?X$!i^{-NebpA204qGsggAac&sADBFn5b6K70+NYaaC#ctUF(3XD zL6@6~S;^l$1d6WDSSjvP4yiitO%^PKR68{*Tt%Vs!l+_Id-f6r>3O&riwiN&F(Khj z-cCK?<%mkB{G0XlQF})j$b~`)Q{r*0ziz3pQun}s*|5{#BvRX2cz)mpyP}||oG9lI z+K~AE6!+XjOD0i5k(6huwEt_Me3W3=^bgd%mWwX+^Jkw1C%Hb;7%*+I^Qj6xAufku zl&(67h5fMOvCu@P3^ZZW$2L}x2J!jxs!nEXwwP0-$+B?t`g|R8X6293p%qxaxG=J~ z7ry(&?L#lwkwkl#7_&}*pKp zy>;^k8D{djG`Q_sk==xcLwmnsq84`iec!Gp1IEjLW!}R8iG+LmVVdVF-C!%!Z5!il zK8(2i3%H$|)-2b;V5)6UQtd#{eClbFtvj-z$*^7a*I_033x2=p-P8pImy}twJ&X*5 z_u(QPZDZJepc2f)?x^o8US_vcXlXs>&{dO2fE-&kxT(Ce9SFfL!nR#;9|Brv=t|>* zSxn`Enhj^sdrha20xb=hGqTo@pRxhI%fU>8(Lw!tg}r#G{oR%(b4N26`yJy^abH|I zC{Xy9%;erUxGBJSZ+C1r1vh)j;#PwB*$y+gaFs@KKXG5peX^yAZ)B5R;6QO^Y#0wb z)WpU023C_&PpWO}zPV#$0}rjX0duan=a`^9H@7l?)cKZ`b_R#f)v{~EGdeP%yYc92 z zUi?Hy2Rlns|0Z~$qfBXk2WB34A59;)u5IrFy8}wmfmgPt9~=!{3vRZDNlWg&bRV}` z?|t<2cixPHsmLI%x};q^^S3#(i&}6DT6#n7>eO!;oD90MosX>2FJh7kd7t1Qo?&jT zKkt5tUic6ne9SsydDnNqXXPry|I}CYh;u2&4dXnQvb{T=3TtDw8()4>d1Q-^$tb_( z%!Kfc5>nBTit?A%EWKDbU*4{NGt1m&OPzZw@itlM>n4afeDvQ-`}|5}K!~OC7&riG zS??6g?$IeubragYZ}a#tW9XY+-u>?G!%a>@_0W*WPNPj68Ro4Ohu@b)nI4oOG|vMA3{r+Eo;@$yzm| z6%QAC;M;4U!$v}zTdl2@OMK=7R|CmDymtFyAIq#~lQZ109Hfi`RZ?ZNX6ML!R$P@X zXK)d|ERAg5TuwY*3+^e9^0#CT*aan?*JrO?H}hi<{s27q;JKKmRFv_=?3SccgbGP~+py=2Zm3FlPXm zooH9Q{!dxT-Hkmyp2FCn0EFGODz(&)MJp}C`mwrB|ZvdPGr^H9mFgMU7sR7zTK z@cRf1r~Vz@cHE00c;3!yO^1ArRi6!ng8TVx#<%_71Y5f}eFucT&o^*diN3S^SW zDWFlcXK7PyO^_AZF$ewbu0#m+`Nw-dQGM^U)*#ouk`u|OS0M@YiWW&BGPu^8^3N^o zj+G~~p;r09wm3S{4DC;5)urrfK429S{f6amy{2d2T@?ZV*n(YD6f(oNM>93U-4_g9 zy760m_6K^kro!frUAVvK45NM0(LX*XRy+n7uiVBo2|+U@)sH#+WFN=79;D&`)fx+mKjevukg)C{cW_d!hhIoafH2)T`yOh!?3Z zZ2!m&I`H75DLWoj3fs*ZJ*E+7g1e=is|$h+O&WN&UT+WAX%f|RzgSApY8uN?9~`i6 z?}U!ua0&2BRGgksZ)~40u1ajMnJ~<`tQ!I4cwjcT%_paKabz56Y&DX-k&Cc2qlVTuqj=0sKRwEe)>Z61cu4_*Vhm;v%qGegIk8Of zMY?2r$$m-=-liiZV|sVD-!;1>WPZ2WUXW9?=ESf@Lh%Aw7FVgTBN6B|o#hI9u|f+e z{a~$}PPuZ90iU8mt^St%t)Z&;!||j9mQu0rZK-)IUW)NO{8(PU%aL9L4Q>CaRKoxs)%9s!j-=MbA|YGoA^P z6PTPJgz#!U#$xT>1eMFnh?7a@xvWBn8;SU&(Eojgk@K$muJkDGttOhf}3_FoUr_AoU(C3C zQ}^I8ag(x3C05U@@QP9^#yhw=(Q1U4o%j6Zhe+R->p`#;)DMn~ywe&*3`u7g!l8Y+XZCwH_3k)^YVR=X+m{%fGMN*sj*Uq6C4<%zI}Ws|`3{ zQ=IfVG`o}YmSt|`vj6C+ZA_ZF1&MN0+8Ibwk4q>DdK-y*thz_4ZFIXGI?sJmv*+>rs zoF1~$WF!}l>nb|+&B2}4J9f41aEmjdKN;{dmaR;g+REKpp=EUd)nQ7^b~1)6h(cjrr}Jt=5tLi5!yAdl}cbrlfY5XP3XIX7ZUG zIFd8f@027865--fRf-S;&m>+#`D!|0BFz_J-%J+;qB6yV4@mAB&KGZVH@qTvrQa0C zis40MKLBHtR~5nDu5~%=JVD>4N|o%n?bQTG8C7~S1#*fMvBL{N_nMfM4d^QUygXd< zw5ik>^NiPx(430&0v*$n!F)Xfk(cN6jkpUTawMVkWwN!KWb1GsfE9k?4pZwdFcA@3 zzrbB?pJ5ksEy~R3$L)6IrLs=Ccf1y?xGrA6hfV^yu^?o6afqKf7?d zbm!7My>5k$0Dr%!>bTJ1J3h(JYh9f_#pxxz`HFyP`_@|%2E8+V_XhXr8FfD&sGp*y zdMrfh7M63{MG^)G2|VKbqoIw3h~tAY+g8iSBmB zI8UHpqMK7G+tJDmC@8STE|JOVi*Dx8vs-z3geKfk{)FTsYKqo!-J+RwxDVQJ=B-XK z@z7x?TcFTu@?O`n-Kjo*InEhg0y7wg>2hkmH+aH3QXR}$y!YE4Wi_2+YzuE{|Q(@}+pB&7KQXM7@1Srzei zPcrTtMCwp?-dbFK|Ev>O0{H9#2I_h9Ossc*s~v_F^rz{>3AbIV5zmg?l+?U?8Va<) z=(drN&^piJh$KG4_ewxbp?hpE>zV|JWyZv(Mec!YxgEaz0yExf=?8Yj?yINEm+e>I zRo?eIQdQia=z4rB0tYAzqCC~y4wW3q9f3iBxiQD4sxit)#qJTC%JBy9fOD=>t=yGPeUeD9E*Z#sZJ*a61>l~xf5rkfQ9zo!?w z1x6l0pR6y?7G9XwkSeTzw$)mgazWXIOjMIJ<1v95;ac zljKi-%S{U6b=d^;W|>q7rIC}DBhtB@{w*&NeMUAp9}|b;3H^JLs6?-oY#N?vs9HWp zt>>u2k5(#m{~P~NiX!?<37FY8)NY8W#9a#I&O%L051C|h%{y-O%@@d6Jbd~e(XgZr zq5p+SmH$3U^ly64SyWBb#ljTU2)XXwe6W_!CPjTt{_XugB{FGYOxGN9lj+$M6xv$e=P3vg0o1=J@sc5JXNgGK zSb7S=yNX*BT82|n*99a-&xvv&bpGaz{BJAx`R~d2pA|&Q(&RLip|48bys%?!eE2pl zs}UHSV~UoUXYThx!fb00^ee-x|E8&gY`6N|X1vf7B}2pvuf$$KLZc}{6m<|ud0t7U z@x?m$i-!@SJc2V@we`y8_$rFNZ!5M#%Ji!cjoEqq{^_yM%_@z!wI;OEyXhgkAO z+<773^O?W$b(fGR-^I2Nkk{QfE}U*5(;~T^gN7XcyJD!ahQB)0%bXzX@j5Rb?l=Q! zxOiToeV4w_UB52x4%OQYl0fzWcx~IYFc1ZNb-C!AL~k;xG;|&_$4uYPfMtvmSa2PR zcwc@rRP*xcm9Df4*{Z5ZyiT=q`YIkNrjE-!H>|Xx%^sg^XP=(lN}|%TXD*1Ei;YCRT~t^PbUk_JdCJ z7-U-Vn^$(&(i7;5KX{s>6vR)2qTlLhi z65Nub%!hE{=|r<9^sQ2QGM8qQm)ICtt83xb2tk~-6khmMhv9;Sg-G*p25Dgt*FOpv zS^g|lgku=10aU*@w5o;}zOxNB8H;XT+ZcRtL!vZnmuI>nZKJ)d);u=vssKlOa&G0t zLqCE<>FZ_Y!$E0=U5NP6t`eFya+RqDBb0t>1laW$13wO(yY5e;zf>NcF3{Si66)9e z9jU3DPa8dV??kF4tt^p*-BI4sqa2a(jeOW`Prt}WRn2FE`{8bO0j1KY%@6O?wPIP) z|DhB|q%x_xM$sdj&G61u`Lbh0dRpAWLLGW&iO?*vB;owCAdz6ruUv6<=0EOU^@Ohh z2j}(U4Ag}oM@A3|!1m2|iw-;dtSUkGe>yLvA?dXd@}atQrAACb(pq5IBG}6=?vHtL zV>mz_1Gn~@eRFy~!I1|o0G{&!fY5FITh}7*3Vce*tkGN074O;`irSJHHt^2|s`r)r`n#JH2k!6bXi5CcEyNUF01xtk~Dcge1oUJ(n( z0ZLJEq;6B8+2<)G;}inmYesEMqbjzF0qrtrs^DOLTp!#tH6HZaz2hu?HbZE?^G^Y!?&grZj zZ8XP)OY1xB7QJZ?cE0NOG+-6u5N^$A5ao$qABl54aZxsSHW}Y_?ef&nW1kcI0a!CM zAs@I*xc~%sy<12t&|W+0`xLb*$vF936?bV`Nfq~{q`m4atJ~1-SuFx?%A8!|DgP$1 z$%@W@cv&LwBzNG#s8&(XH1eGu32q?y#vx!NxG>i#>&*j~74{NxB>TNfZf_1@7m@db(bUc-b|36*IdHksTS?6s8VleFix|3~jge#5q>-nVEGJ{c?{#`--% zY<$a8+)1d+n}W$=>^_VTRX$8rPIi){y%AO&-0TqR1!iKB9`R}F<}(y@>$(onUY_3< z!qdekIn0K?Qm?+)-ta-m$^9AdewcNNeRiG{c>cVE(i^#pk{8CZI=32mx;m3Q?+kHo zEG9_{DY?=Y_wBD#mA&|}uv|_(`MgUgtqfsq={$O4!2s3A1eRA;0SO~>k3$0-$~zJ1AjNJKlzhgA7y*DWzDVlNxGivS9v(+W4l9h6`eO% z^2TdMn|h$>{BK+fqt`VC(#~BbSHwK*L3=1#+Um=)+QICh zyYJGyZBtD9#uh*5c)3qhuS1vL?+O*S($(|568Ag@ z^^ZbiBo@u2^|x)%GsHY}-eMj`R}jY#F5Y#+9^w%OBC|v$!qn1vwKRU#qC|?LC~N^I zBZZF3cTvS&ZVOwL$DgRG6T@Nuv-4M9637((>HTzz+Tg0E>c+p!E7ougHTegSsec`p9?3s_Rm=X>RvX$PW9vV)za_Wl#^UBB;b@bk1C;?m6TY0j#h3?TerfIcF7tId%@pQunOl^< zKM`MkRjZf1=S-iKHkd*j8)g!vj#){;Ks?}j$MAz^1Du}k#@mC+*{F3p z4ni-xrX1E=hS$nIP^Wo|y&ff>E+gSwznhNP62ZJ?p^gb15&V$iXC}Fudb{>%YGuh% zoY#A8k?lw)mh-cW*pMr3Uv^rnP88`!4A`A5Uw38rs1{$bCf=o`U`i8_d&O8iGu4IC zicfGQLTk`d>>QdNDqsAp)c|3UKT`sVL^|L+hKTw7j_|SEsgL|f#O+3DN_SVl`6)$e zw!{n$@dYRM*Usr7LrR`)?zMU)zvRF1=>UFdfyugFo1CY~0>?w}UpAz}oVvN*Mu#XS z=LL{F1$C}>kQuOt=4pO~_$axE%uZK4Amm%|6c66hDdAAjVYQ;|hHW}uZJ98=vmJxZ zHl}Mz!+zsFb8i+C{?nzns@Cct@rhJ;+vf+-vj8h|>emfM&6-_{Mtc@sWTFki-#<8P zQr%N^K}|h><6@_)NPNV&p*|o1=2Hx&(F*=T_$j`XjMZN~{5FtNM~hF70$8lv^!gwY zI>IN`_hQ_rY~p;Oyfbi{t8i>nzMyBE=h}9$UHF&%PKr-f`)}2r($`xDE>;ke&_X#% zm>`8!nT8(;5C@A~9u9grORPX*p4aGoEH9l#qO^zYmr%a_A0b@aRo(^T-F1OAJgAo< ztdA3mGt?iu94+&9hot&=WJL^aiK3!W=UKy06pE;&oJ zv~Mk$j-&60o&P;zA`t@&EX-T-Gp-H zy$`f??4_Yy#Id)X$hA14*<^e2S}~(Sxc6Z#W?M``?M)B`{!L@3eR$P9==iVa^cV+M zN18X9=A4Dn7H`4i8k+0ZTHX5Jyr~d;R%`SRcf0$H#E<(l&OE7%zW}#R4XVHF3JfUHZji`*s++m;Smd=uqY3y7qJJGM;< z7ET`e|JWtoK=rrOy1?lNA{h#b9}A>&7B3VoFJBBX0)QcRyJvX7#l!T8 z`)@;{w53{gO+ODLgWVYghJFwUI@a0f#nRq6CF)O|;X~(?z9>ztOF>EwQbnQ#5tgnS z3E-DQcB8Z>@_`yw*VL=oFB?YB1`f`jw?3It*qu41bxNyzpCh+lgM1{H-8T5uU1gQ{ zL`AjRoYtigz`BJ$SNUj;wpXo#HBYGnRrbQkpI(h3NaKyQoeLAV&$9QR_q*tERm7L{ zdY**k5TJ#a%J!%D?2eeD_Wrj6J=_}X%W`B zZ*Y%YGsPaFpZp2YKdYcDkS$q+Yx;$IWG5V?3C|Aef6HXQFDKf;Xxp@}sB>cZ{d8lu zr0mNmt;hxB19kq_l7v0W4AX`EuJV%!$6a3=*zyxBWXOFip}3jS)^mZJe)EBwqO~LN zeOl5j=Qm610G}+Bfk7m2n|v@yX8jo_5{l7p0=@e&OQvP*_Y#5L&aMmsZK_;1$XpWB zV-(~?(Pz}aQ#U!2Gv4vYprkXg=buPnuBtQ#M#b@2?lL!bIZv_>Z~lVvaBp5WX0k?Ex)B-Hb|6Qci75TwpVdDIYOaL0zheBp(TD6zV&k z{X;ui(9OmyMZ{Kie@o*;>*q--_6l)9q_Fq(ETw$R_;F(lJ=|g|C-?LL^3_M$+^GWn zh){>~#M`c?qa?jsICmB;wL7u)jXXQ{IBoNvDd$?{r0fC##-JpQav19jh73UBYtF+ zeeJ(Yw!br80#Q$IFvgXU6Oz?m-(EopPm^Pk)I4G^7)vlGWr(Dz!7iJ)Q(YvNYOH@x zRL4YWDzAFHT-P8~Z&#n`v-C>1$Y-^TEidxd86y_F4p#&?P}Ng06ga~&<{e(?`@SVn zUkDgC&WBRFGbGz~D>YYq|EkfFaEr$Fr+ck_*f)cc1;!p;Ecst-^a3{20zF8izb*-O z^VK4)O7R>qnvZv`st=M6M?k+*9sLm$7hyqHS{W>89)q35fDW|I)sk&0HKF2il{(B zk$`|m6+)CMHPY`7-}mm$ot-IO#tn`QHSEQvgFnXdMUlHxnk!0|v~WZVlw_*m*zaLNK7-`zc9m62yQ-9aDb{%k^{B4M7|8~_ z{Km4XJb)5F#S6-(%>Q@R@!#I_zqN+{VEl;`khxJJzT4mb-a|Bwzo{dCd>W*E9{n$R zQb0i)3&t~=+f}cJ?$@3MW~i_RAQhVj>niQD(DR>>?mCozgyugQo$`Nqcg0&k>ipjP zjj!nK9jzR$%!X4qskE=&y`u$X->noiuh$Z282!6vC_hF%9%#xNR6dlYto84 z>4l*DB0@Bn>970jP1| zGoC@yP9qRq(4U6G$>8glDr^#tNIG0Z%8wd`B3&gstw~TfClDxuPtG72T@$jVhv( z#jdR&HK^K=2FHAcEQebPc?Afy6_=tW=cdWg%?(780(sjdP%0`(Xv_~m5>!d<2ixeqsPC_h#|h8)i)FzZYMV0;iU+# zuk>qNG-!yPIryj9s78>Y%S4d|S#T)?Ksu*Y{imAy9gC@Ln9uuM@7|piW_1wdO1{M~ ze$L_?HGnD${356=Tt(o9wo^{_-P+5Tb!zsGf0|DKMf*!?NOoE&?OrGR{Ub90GY6s0 z^#=cz0^3sC)NARzscCDhbQEJpPa{re1?g*i^hTtJn3o6G*BSfN9IxkzXrPSj@Ss^C zZNM{#HQ~Slv+IW`iDD)17B9aj9AS{?E}bg7*8M9~Z3GEwUJeqZv1l=Yi)4X0ltr(gpXIOS}Y;%BJ zyTHa06`JlRuiDY|Ff{6`8(q}G_4*a-1EuHwG_2jj3e&qgv~LFseUfW00W?+lEG$g6 zaSSni-9uUW$=0aCNv5K4)ye)zcPLa<{Z2gn?j!jz(L-^+wNlr3Q+IUBQN|5$Zh6ht zvz)=(N-IjccGWuyd7ydndc9n#WtD0Hr6j0~`ePlkaKBqR`5lrzNiVb(_~g^RVN}>aSE`am-R^H?ssb?9h+}w7jn+Fz7><~e21jt>p5&M7RYNg>L~S2Sw}VESs+$R zNI0<4sdal-&>My_=;DU2Ec@SAQO5N)=~*}(09cp_9adG_S{)<8!x3Ap3XokyAf)Uo z7i2=5{cE<&Z?@NGD)`D@2@(>oGP_x7_5^;EoC1n`ptKPDHZfZIj60xN_3Aer-9XE+ z~oA#=!I1FhyBm2*9m^jWakp z3z!K5Cwi@e+1{Siy(Y!m5%6NxR?a1*>VPc0{|P~Nlv2-KY1pRK`h_;=UKDic5b(3X zZ$GoGWlOJlpWB_}e@|B-X?am_MB^Bg8FDn*Y#=a97k+y;SvB6PX=HXrC$BC-~TE zW>OY(&&Tyw&%~j;y-8J~=n#IAKNC;fB!F?lBYbzuQdsa^9lx1f9Gx}&dd9~z1cqWc z_BLZJis&0l3XOpO7K6K%zeCQj162~iC}WYfmb+=NestwFY$(i>$ylOtlgmw{t0(E! zVh>tOUzC(u)~ul8=Y6Z{fLar19a)ttm@H?oN)C?OI4cC!!6)4pUi~6^X%$kXg=L2I zu8+(yqnQ%e2N$_sz0;>gwwgHu-I|t;8_BpvFhcMr=E-3+Da4zYsWF({3&bOpw zIIXr>#*Rh@@Hopi)y>CDn-Q#*r;0N+d~8!|KuVpiNUCeyYx;cufbfv7?hp&XAakZ!3%xUqQDV$^5+$Sbut8H*uPkWi!)K&CNFcGt36pUAHCu*s~1N|bqj6Y2FM(5E>~ z-2uJJ5|hwe*Py zv*3rJJn_r${+|h#uRtXl6{7>v3oXYF=QqnV*JTB_kwD?dkCED~j+_52)ptyLRb4#Y zTI|CGp^xxfe{k!EGBl&{RUORZFm8uNK|pm>ceDIjrQtTHUk8JnX15Fj3!Q#$i@ejc>D`>JF}{9eT-ztz;4GTs5hoS$a#& z<0oNYRNkhA!6OOZwaZD9Z!6~p`xwM^xzRq^2`Qz5@oqNL3vj&|D7e|4Qmmn z_|dW^s;4g`w&=5OE)SSY{m<82?i+xb7w!0crfYan_Q!TI>STI7+&={}=`XKzlJ(Be z2w%TvwHF#!*UH0mMk36Mr+%@&*}IdV0SyeBE|WRzU+7zW)am0d7C)03gBjC=PW%FW>h=>#2Rq-sNSIZ1E$Txvv zR$kXj3-?zmx(N?3UfC-5)K=~45&4P90u34NeMIozZ1dRO0S=a;7xysVOav1Jqa8Hp)u9mv?=mF&$+hfrB66`E8(#au+j>4DW z`Wpz4E#?@!H?+;Nl!i60aLzM&_$46p9B`b=k0JlmmmcaC%CQHNBWLPQRrQmWLSBqn zqQg7)H7rM^MN3A?pKmQL2QEOi_fRTTx4H6v#hGRH^TmayZPdN>AG1X$UGlOuX9Rw= z^u^Lfze`+=e{8^dj^rz?spsYUC-03etOY0=PW+ggY{XLC-yQcK3CUV2d-6)ZVcu9! z!tm?&obz2|U;21Ia!~HCa;EWMo-5c@BJh=&$9qvF+;CM(pWdY(7>={UAMn>GLCi^X zUlnsgb;z99EI zS!NX$KjCibl$KtZ)d&Jc@9MzcN|p_n*YuFlO7og1nZAhF;-IMWdgprnC)uo+y{nJq zgS8bJ%;&S;D30nTr6Hy2Jp1W^ZEv6D)+%^-FcZKQoV9aDnRV97fHtV*zSYZq*Obt% zR!oA@iGan+N;B7fChOcXpr0J-l-I|xI6szf(DaRdZicCyCtc*7 zoP$KEq)lr%%?reK`Ml3Xzpnz?x)qb5c*7E=ma-`80rkXwnT@#J0x#957C!@W%a!q}LFQb-0IaKB>oEsa`m9u5!+0H}@yjCVz!bSu9Gb zMg?I1Vn}iUF(}3Kv<6zj7qVR*1E5E~Kh0vM?ar9(C*)X_<^oyuRES3A+#`c>(hl0$ z#V`P8;vF_nP7RzR)By7VpSp{+%U3wxf4PtH%qZB!YKx}-M00-0Sb~RH!aPAbSQw{V zb;Ak0moQMLGR-Mbicj_GFL4S5GZV`byO}`C4_oTWJr*dFF9}>JWK_Z(hd`{ZN_8Wb zd=Hu7C%pFufXk-xl};5kA!6^?lWebkpxSlydCZ1Sb{M__VAE>~S?Qryx~s7YE)H^o z_o8-!otxnD4fT2~p~@}-&Fw1eq{OL41Ge`AB!n*H(@5p-$EWM-KQKx)$XU!lQzvy- zCuF)Ok8Z)eWLu@Uq*`?BMNkLwtcuV+t^QCuM_OICl=halh4aE5+<-N?N56-wj(75* zev&G-`xriB4<7L0*IKsLlr`70;ztFp2iCks+5eiKW1K*J3g&rLSN zh42oloO_02(-mz9g_0PHTmjS}vOeNH&6I?YhU0T0=|~rWOGOBNoCU*O!VC+jcMEAT zo_=eZ+A@$!^+4nS)KTjcRs`4Bo->{%P)z=Kj&Y&%E%`FbtQX+%1l(>2WW!khj2u!i(K6AI6f5j4LZ8{*pK zq6%}w9!1qZnKxEW$pMA8exH8wsI%o_zBn*4Hk>#7UxeiUGhgyI(MW83{4xfla6pc~ zrVN!IUOWq8FA%Nj$OV!wXa6K$X8#(&6BmE8h|Uo31$QB!f&Bn*oqsc?XEWrlRLzIt ZlS@_ix6S4{Z!w%c0@g54uTr&-_y+|B6pa7? literal 0 HcmV?d00001 diff --git a/articles/search/vector-search-how-to-create-index.md b/articles/search/vector-search-how-to-create-index.md index 573c19a496..d66d9a3b0d 100644 --- a/articles/search/vector-search-how-to-create-index.md +++ b/articles/search/vector-search-how-to-create-index.md @@ -30,13 +30,13 @@ This article explains the workflow and uses REST for illustration. Once you unde ## Prerequisites -+ Azure AI Search, in any region and on any tier. Most existing services support vector search. For services created before January 2019, there's a small subset that can't create a vector index. In this situation, a new service must be created. If you're using integrated vectorization (skillsets that call Azure AI), Azure AI Search must be in the same region as Azure OpenAI or Azure AI services. ++ Azure AI Search, in any region and on any tier. On services created before January 2019, there's a small subset that can't create a vector index. If this applies to you, create a new service to use vectors. For indexing workloads that include integrated vectorization (skillsets that call Azure AI), Azure AI Search must be in the same region as Azure OpenAI or Azure AI services. -+ [Pre-existing vector embeddings](vector-search-how-to-generate-embeddings.md) or use [integrated vectorization](vector-search-integrated-vectorization.md), where embedding models are called from the indexing pipeline. ++ You must have [pre-existing vector embeddings](vector-search-how-to-generate-embeddings.md) to upload to the index, or you can use [integrated vectorization](vector-search-integrated-vectorization.md), where embedding models are called from a skillset in an indexer pipeline. -+ You should know the dimensions limit of the model used to create the embeddings. Valid values are 2 through 3072 dimensions. In Azure OpenAI, for **text-embedding-ada-002**, the length of the numerical vector is 1536. For **text-embedding-3-small** or **text-embedding-3-large**, the vector length is 3072. ++ You should know the dimensions limit of the model used to create the embeddings so that you can assign that limit to the vector field. Integrated vectorization supports a finite number of embedding models. For **text-embedding-ada-002**, dimensions are fixed at 1536. For **text-embedding-3-small** or **text-embedding-3-large**, the vector length ranges from 1 to 1536 and 3072, respectively. -+ You should also know what the supported similarity metrics are. For Azure OpenAI, similarity is [computed using `cosine`](/azure/ai-services/openai/concepts/understand-embeddings#cosine-similarity). ++ You should also know what similarity metric to use. For embedding models on Azure OpenAI, similarity is [computed using `cosine`](/azure/ai-services/openai/concepts/understand-embeddings#cosine-similarity). + You should be familiar with [creating an index](search-how-to-create-search-index.md). The schema must include a field for the document key, other fields you want to search or filter, and other configurations for behaviors needed during indexing and queries. @@ -50,9 +50,9 @@ Make sure your documents: 1. Provide vector data (an array of single-precision floating point numbers) in source fields. - Vector fields contain an array generated by embedding models, one embedding per field, where the field is a top-level field (not part of a nested or complex type). For the simplest integration, we recommend the embedding models in [Azure OpenAI](https://aka.ms/oai/access), such as **text-embedding-ada-002** for text documents or the [Image Retrieval REST API](/rest/api/computervision/2023-02-01-preview/image-retrieval/vectorize-image) for images. + Vector fields contain an array generated by embedding models, one embedding per field, where the field is a top-level field (not part of a nested or complex type). For the simplest integration, we recommend the embedding models in [Azure OpenAI](https://aka.ms/oai/access), such as a **text-embedding-3** model for text documents or the [Image Retrieval REST API](/rest/api/computervision/2023-02-01-preview/image-retrieval/vectorize-image) for images. - If you can take a dependency on indexers and skillsets, consider using [integrated vectorization](vector-search-integrated-vectorization.md) that encodes images and textual content during indexing. Your field definitions are for vector fields, but incoming source data can be text or images, represented as vector arrays created during indexing. + If you can take a dependency on indexers and skillsets, consider using [integrated vectorization](vector-search-integrated-vectorization.md) that encodes images and textual content during indexing. Your field definitions are for vector fields, but incoming source data can be text or images, which are converted to vector arrays during indexing. 1. Provide other fields with human-readable content for the query response, and for hybrid query scenarios that include full text search or semantic ranking in the same request. @@ -69,7 +69,7 @@ A vector configuration specifies the parameters used during indexing to create " If you choose HNSW on a field, you can opt in for exhaustive KNN at query time. But the other direction doesn’t work: if you choose exhaustive, you can’t later request HNSW search because the extra data structures that enable approximate search don’t exist. -A vector configuration also specifies quantization methods for reducing vector size: +Optionally, a vector configuration also specifies quantization methods for reducing vector size: + Scalar + Binary (available in 2024-07-01 only and in newer Azure SDK packages) diff --git a/articles/search/vector-search-index-size.md b/articles/search/vector-search-index-size.md index 969c69366e..2aeaa87008 100644 --- a/articles/search/vector-search-index-size.md +++ b/articles/search/vector-search-index-size.md @@ -10,7 +10,7 @@ ms.custom: - build-2024 - ignite-2024 ms.topic: conceptual -ms.date: 09/19/2024 +ms.date: 01/09/2025 --- # Vector index size and staying under limits @@ -27,7 +27,7 @@ For each vector field, Azure AI Search constructs an internal vector index using + Vector index size is measured in bytes. -+ Vector quotas are based on memory constraints. All searchable vector indexes must be loaded into memory. At the same time, there must also be sufficient memory for other runtime operations. Vector quotas exist to ensure that the overall system remains stable and balanced for all workloads. ++ Vector quotas are based on memory constraints. For vector indexes created using the Hierarchical Navigable Small World (HNSW) algorithm, searchable vector indexes reside in memory. At the same time, there must also be sufficient memory for other runtime operations. Vector quotas exist to ensure that the overall system remains stable and balanced for all workloads. If you use exhaustive KNN algorithm, indexes are loaded into memory only at query time.W + Vector indexes are also subject to disk quota, in the sense that all indexes are subject disk quota. There's no separate disk quota for vector indexes. @@ -67,11 +67,24 @@ A request for vector metrics is a data plane operation. You can use the Azure po ### [**Portal**](#tab/portal-vector-quota) -Usage information can be found on the **Overview** page's **Usage** tab. Portal pages refresh every few minutes so if you recently updated an index, wait a bit before checking results. +#### Vectors per index + +To get vector index size per index, select **Search management** > **Indexes** to view a list of indexes, along with the document count, the size of in-memory vector indexes, and total index size as stored on disk. + +Recall that vector quota is based on memory constraints. For vector indexes created using the HNSW algorithm, all searchable vector indexes are loaded into memory. For indexes created using the exhaustive KNN algorithm, vector indexes aren't loaded until query time. There's no memory residency requirement for exhaustive KNN indexes. + +The following screenshot shows two versions of the same vector index. One version is created using HNSW algorithm, where the vector graph is memory resident. Another version is created using exhaustive KNN algorithm. With exhaustive KNN, there's no specialized in-memory vector index, so the portal shows 0 MB for vector index size. Those vectors still exist and are counted in overall storage size, but they don’t occupy the in-memory resource that the vector index size metric is tracking. + +:::image type="content" source="media/vector-search-index-size/vector-index-size-by-algorithm.png" lightbox="media/vector-search-index-size/vector-index-size-by-algorithm.png" alt-text="Screenshot of the index portal page showing vector index size based on different algorithms."::: + +#### Vectors per service + +To get vector index size for the search service as a whole, select the **Overview** page's **Usage** tab. Portal pages refresh every few minutes so if you recently updated an index, wait a bit before checking results. The following screenshot is for an older Standard 1 (S1) search service, configured for one partition and one replica. + Storage quota is a disk constraint, and it's inclusive of all indexes (vector and nonvector) on a search service. + + Vector index size quota is a memory constraint. It's the amount of memory required to load all internal vector indexes created for each vector field on a search service. The screenshot indicates that indexes (vector and nonvector) consume almost 460 megabytes of available disk storage. Vector indexes consume almost 93 megabytes of memory at the service level. From b18a401bc9773057147548fedde29f27cee0e76a Mon Sep 17 00:00:00 2001 From: Aaron Hill Date: Fri, 10 Jan 2025 10:29:35 -0800 Subject: [PATCH 2/7] updating rest api --- .../ai-services/openai/includes/use-your-data-rest.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/articles/ai-services/openai/includes/use-your-data-rest.md b/articles/ai-services/openai/includes/use-your-data-rest.md index 2827c5d4bc..bf88d10a3e 100644 --- a/articles/ai-services/openai/includes/use-your-data-rest.md +++ b/articles/ai-services/openai/includes/use-your-data-rest.md @@ -5,7 +5,7 @@ author: aahill ms.author: aahi ms.service: azure-ai-openai ms.topic: include -ms.date: 03/07/2024 +ms.date: 01/10/2025 --- [!INCLUDE [Set up required variables](./use-your-data-common-variables.md)] @@ -20,7 +20,7 @@ To trigger a response from the model, you should end with a user message indicat > There are several parameters you can use to change the model's response, such as `temperature` or `top_p`. See the [reference documentation](../reference.md#completions-extensions) for more information. ```bash -curl -i -X POST $AZURE_OPENAI_ENDPOINT/openai/deployments/$AZURE_OPENAI_DEPLOYMENT_ID/chat/completions?api-version=2024-02-15-preview \ +curl -i -X POST $AZURE_OPENAI_ENDPOINT/openai/deployments/$AZURE_OPENAI_DEPLOYMENT_ID/chat/completions?api-version=2024-10-21 \ -H "Content-Type: application/json" \ -H "api-key: $AZURE_OPENAI_API_KEY" \ -d \ @@ -31,8 +31,11 @@ curl -i -X POST $AZURE_OPENAI_ENDPOINT/openai/deployments/$AZURE_OPENAI_DEPLOYME "type": "azure_search", "parameters": { "endpoint": "'$AZURE_AI_SEARCH_ENDPOINT'", - "key": "'$AZURE_AI_SEARCH_API_KEY'", - "index_name": "'$AZURE_AI_SEARCH_INDEX'" + "index_name": "'$AZURE_AI_SEARCH_INDEX'", + "authentication": { + "type": "api_key", + "key": "'$AZURE_AI_SEARCH_API_KEY'" + } } } ], From adb00644d83666351c0d307204e111787853f604 Mon Sep 17 00:00:00 2001 From: Aaron Hill Date: Fri, 10 Jan 2025 10:33:12 -0800 Subject: [PATCH 3/7] rest api example --- articles/ai-services/openai/includes/use-your-data-rest.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/articles/ai-services/openai/includes/use-your-data-rest.md b/articles/ai-services/openai/includes/use-your-data-rest.md index bf88d10a3e..1098b17a22 100644 --- a/articles/ai-services/openai/includes/use-your-data-rest.md +++ b/articles/ai-services/openai/includes/use-your-data-rest.md @@ -84,7 +84,8 @@ curl -i -X POST $AZURE_OPENAI_ENDPOINT/openai/deployments/$AZURE_OPENAI_DEPLOYME "prompt_tokens": 3779, "completion_tokens": 105, "total_tokens": 3884 - } + }, + "system_fingerprint": "fp_65792305e4" } ``` From a3f8a9d69abd5cc44aec111b084060fae08cc106 Mon Sep 17 00:00:00 2001 From: Jordan Bean <84806584+jordanbean-msft@users.noreply.github.com> Date: Fri, 10 Jan 2025 12:34:12 -0600 Subject: [PATCH 4/7] Update configure-managed-network.md Fixed typo in call to `az ml workspace create --provision-network` command --- articles/ai-studio/how-to/configure-managed-network.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/articles/ai-studio/how-to/configure-managed-network.md b/articles/ai-studio/how-to/configure-managed-network.md index 18525a9bb6..1572d583cc 100644 --- a/articles/ai-studio/how-to/configure-managed-network.md +++ b/articles/ai-studio/how-to/configure-managed-network.md @@ -634,7 +634,7 @@ During hub creation, select __Provision managed network proactively at creation_ The following example shows how to provision a managed virtual network during hub creation. The `--provision-network-now` flag is in preview. ```azurecli -az ml workspace create -n myworkspace -g my_resource_group --kind hub --managed-network AllowInternetOutbound --provision-network-now +az ml workspace create -n myworkspace -g my_resource_group --kind hub --managed-network AllowInternetOutbound --provision-network-now true ``` The following example shows how to provision a managed virtual network. @@ -654,7 +654,7 @@ az ml workspace show -n my_ai_hub_name -g my_resource_group --query managed_netw The following example shows how to provision a managed virtual network during hub creation. The `--provision-network-now` flag is in preview. ```azurecli -az ml workspace create -n myworkspace -g my_resource_group --managed-network AllowInternetOutbound --provision-network-now +az ml workspace create -n myworkspace -g my_resource_group --managed-network AllowInternetOutbound --provision-network-now true ``` The following example shows how to provision a managed virtual network: From 108ca94e1f0ff1df178326410dcd4cc13d59aad2 Mon Sep 17 00:00:00 2001 From: Heidi Steen <1612796+HeidiSteen@users.noreply.github.com> Date: Fri, 10 Jan 2025 10:36:20 -0800 Subject: [PATCH 5/7] Robert's feedback --- articles/search/vector-search-index-size.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/articles/search/vector-search-index-size.md b/articles/search/vector-search-index-size.md index 2aeaa87008..f745610a68 100644 --- a/articles/search/vector-search-index-size.md +++ b/articles/search/vector-search-index-size.md @@ -67,17 +67,17 @@ A request for vector metrics is a data plane operation. You can use the Azure po ### [**Portal**](#tab/portal-vector-quota) -#### Vectors per index +#### Vector size per index -To get vector index size per index, select **Search management** > **Indexes** to view a list of indexes, along with the document count, the size of in-memory vector indexes, and total index size as stored on disk. +To get vector index size per index, select **Search management** > **Indexes** to view a list of indexes and the document count, the size of in-memory vector indexes, and total index size as stored on disk. -Recall that vector quota is based on memory constraints. For vector indexes created using the HNSW algorithm, all searchable vector indexes are loaded into memory. For indexes created using the exhaustive KNN algorithm, vector indexes aren't loaded until query time. There's no memory residency requirement for exhaustive KNN indexes. +Recall that vector quota is based on memory constraints. For vector indexes created using the HNSW algorithm, all searchable vector indexes are permanently loaded into memory. For indexes created using the exhaustive KNN algorithm, vector indexes are loaded in chunks, sequentially, during query time. There's no memory residency requirement for exhaustive KNN indexes. The lifetime of the loaded pages in memory is similar to text search and there are no other metrics applicable to exhaustive KNN indexes other than total storage. The following screenshot shows two versions of the same vector index. One version is created using HNSW algorithm, where the vector graph is memory resident. Another version is created using exhaustive KNN algorithm. With exhaustive KNN, there's no specialized in-memory vector index, so the portal shows 0 MB for vector index size. Those vectors still exist and are counted in overall storage size, but they don’t occupy the in-memory resource that the vector index size metric is tracking. :::image type="content" source="media/vector-search-index-size/vector-index-size-by-algorithm.png" lightbox="media/vector-search-index-size/vector-index-size-by-algorithm.png" alt-text="Screenshot of the index portal page showing vector index size based on different algorithms."::: -#### Vectors per service +#### Vector size per service To get vector index size for the search service as a whole, select the **Overview** page's **Usage** tab. Portal pages refresh every few minutes so if you recently updated an index, wait a bit before checking results. From 1eea3b53b1eef4c043b404d7008e1e10b58b6f0a Mon Sep 17 00:00:00 2001 From: Shilpa Sharma Date: Fri, 10 Jan 2025 11:26:31 -0800 Subject: [PATCH 6/7] Fixed typo --- articles/search/vector-search-index-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/search/vector-search-index-size.md b/articles/search/vector-search-index-size.md index f745610a68..1a4a6789d7 100644 --- a/articles/search/vector-search-index-size.md +++ b/articles/search/vector-search-index-size.md @@ -27,7 +27,7 @@ For each vector field, Azure AI Search constructs an internal vector index using + Vector index size is measured in bytes. -+ Vector quotas are based on memory constraints. For vector indexes created using the Hierarchical Navigable Small World (HNSW) algorithm, searchable vector indexes reside in memory. At the same time, there must also be sufficient memory for other runtime operations. Vector quotas exist to ensure that the overall system remains stable and balanced for all workloads. If you use exhaustive KNN algorithm, indexes are loaded into memory only at query time.W ++ Vector quotas are based on memory constraints. For vector indexes created using the Hierarchical Navigable Small World (HNSW) algorithm, searchable vector indexes reside in memory. At the same time, there must also be sufficient memory for other runtime operations. Vector quotas exist to ensure that the overall system remains stable and balanced for all workloads. If you use exhaustive KNN algorithm, indexes are loaded into memory only at query time. + Vector indexes are also subject to disk quota, in the sense that all indexes are subject disk quota. There's no separate disk quota for vector indexes. From cb333a45c55dcbda0bc0c8eb93677b9e3dca2d94 Mon Sep 17 00:00:00 2001 From: mrbullwinkle <31510320+mrbullwinkle@users.noreply.github.com> Date: Fri, 10 Jan 2025 15:32:30 -0500 Subject: [PATCH 7/7] update --- articles/ai-services/openai/api-version-deprecation.md | 7 +++++-- articles/ai-services/openai/includes/api-surface.md | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/articles/ai-services/openai/api-version-deprecation.md b/articles/ai-services/openai/api-version-deprecation.md index 797cf83093..9b62f584c3 100644 --- a/articles/ai-services/openai/api-version-deprecation.md +++ b/articles/ai-services/openai/api-version-deprecation.md @@ -5,7 +5,7 @@ services: cognitive-services manager: nitinme ms.service: azure-ai-openai ms.topic: conceptual -ms.date: 01/08/2024 +ms.date: 01/10/2024 author: mrbullwinkle ms.author: mbullwin recommendations: false @@ -25,7 +25,10 @@ This article is to help you understand the support lifecycle for the Azure OpenA Azure OpenAI API latest release: - Inference: [2024-12-01-preview](reference-preview.md) -- Authoring: [2024-12-01-preview](https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring/preview/2024-10-01-preview/azureopenai.json) +- Authoring: [2024-10-01-preview](https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring/preview/2024-10-01-preview/azureopenai.json) + +> [!IMPORTANT] +> For features that are part of the dataplane authoring API such as batch, fine-tuning, and assistants files, continue to use API version `2024-10-01-preview` to take advantage of the latest preview features. This version contains support for the latest Azure OpenAI features including: diff --git a/articles/ai-services/openai/includes/api-surface.md b/articles/ai-services/openai/includes/api-surface.md index 80acca3393..ea8fe27a03 100644 --- a/articles/ai-services/openai/includes/api-surface.md +++ b/articles/ai-services/openai/includes/api-surface.md @@ -22,7 +22,7 @@ Each API surface/specification encapsulates a different set of Azure OpenAI capa | API | Latest preview release | Latest GA release | Specifications | Description | |:---|:----|:----|:----|:---| | **Control plane** | [`2024-06-01-preview`](/rest/api/aiservices/accountmanagement/operation-groups?view=rest-aiservices-accountmanagement-2024-06-01-preview&preserve-view=true) | [`2024-10-01`](/rest/api/aiservices/accountmanagement/deployments/create-or-update?view=rest-aiservices-accountmanagement-2024-10-01&tabs=HTTP&preserve-view=true) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/resource-manager/Microsoft.CognitiveServices) | Azure OpenAI shares a common control plane with all other Azure AI Services. The control plane API is used for things like [creating Azure OpenAI resources](/rest/api/aiservices/accountmanagement/accounts/create?view=rest-aiservices-accountmanagement-2023-05-01&tabs=HTTP&preserve-view=true), [model deployment](/rest/api/aiservices/accountmanagement/deployments/create-or-update?view=rest-aiservices-accountmanagement-2023-05-01&tabs=HTTP&preserve-view=true), and other higher level resource management tasks. The control plane also governs what is possible to do with capabilities like Azure Resource Manager, Bicep, Terraform, and Azure CLI.| -| **Data plane - authoring** | `2024-12-01-preview` | `2024-10-21` | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring) | The data plane authoring API controls [fine-tuning](/rest/api/azureopenai/fine-tuning?view=rest-azureopenai-2024-08-01-preview&preserve-view=true), [file-upload](/rest/api/azureopenai/files/upload?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [ingestion jobs](/rest/api/azureopenai/ingestion-jobs/create?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [batch](/rest/api/azureopenai/batch?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) and certain [model level queries](/rest/api/azureopenai/models/get?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) +| **Data plane - authoring** | `2024-10-01-preview` | `2024-10-21` | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring) | The data plane authoring API controls [fine-tuning](/rest/api/azureopenai/fine-tuning?view=rest-azureopenai-2024-08-01-preview&preserve-view=true), [file-upload](/rest/api/azureopenai/files/upload?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [ingestion jobs](/rest/api/azureopenai/ingestion-jobs/create?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [batch](/rest/api/azureopenai/batch?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) and certain [model level queries](/rest/api/azureopenai/models/get?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) | **Data plane - inference** | [`2024-12-01-preview`](/azure/ai-services/openai/reference-preview#data-plane-inference) | [`2024-10-21`](/azure/ai-services/openai/reference#data-plane-inference) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference) | The data plane inference API provides the inference capabilities/endpoints for features like completions, chat completions, embeddings, speech/whisper, on your data, Dall-e, assistants, etc. | ## Authentication