Skip to content

Commit

Permalink
Updated MM video example (#10722)
Browse files Browse the repository at this point in the history 
* updated example, pkg breakage due to refactor errors fixed

* typos

* nb clean

* output restored

* formatting

* typo
  • Loading branch information
@raghavdixit99
raghavdixit99 authored Feb 14, 2024
1 parent 3111da4 commit f7b3fee
Showing 1 changed file with 51 additions and 88 deletions.
139 changes: 51 additions & 88 deletions docs/examples/multi_modal/multi_modal_video_RAG.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@
"outputs": [],
"source": [
"%pip install llama-index-multi-modal-llms-openai\n",
"%pip install llama-index-vector-stores-lancedb"
"%pip install llama-index-vector-stores-lancedb\n",
"%pip install llama-index-embeddings-clip"
]
},
{
Expand All @@ -49,6 +50,7 @@
"outputs": [],
"source": [
"%pip install llama_index ftfy regex tqdm\n",
"%pip install -U openai-whisper\n",
"%pip install git+https://github.com/openai/CLIP.git\n",
"%pip install torch torchvision\n",
"%pip install matplotlib scikit-image\n",
Expand All @@ -57,7 +59,8 @@
"%pip install pytube\n",
"%pip install pydub\n",
"%pip install SpeechRecognition\n",
"%pip install ffmpeg-python"
"%pip install ffmpeg-python\n",
"%pip install soundfile"
]
},
{
Expand All @@ -67,7 +70,6 @@
"outputs": [],
"source": [
"from moviepy.editor import VideoFileClip\n",
"from pydub import AudioSegment\n",
"from pathlib import Path\n",
"import speech_recognition as sr\n",
"from pytube import YouTube\n",
Expand Down Expand Up @@ -230,46 +232,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Moviepy - Writing frames ./mixed_data/frame%04d.png.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Moviepy - Done writing frames ./mixed_data/frame%04d.png.\n",
"MoviePy - Writing audio in ./mixed_data/output_audio.wav\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"MoviePy - Done.\n",
"Text data saved to file\n",
"Audio file removed\n"
]
}
],
"outputs": [],
"source": [
"try:\n",
" metadata_vid = download_video(video_url, output_video_path)\n",
Expand Down Expand Up @@ -420,7 +383,7 @@
{
"data": {
"text/markdown": [
"**Node ID:** 2b0dab05-1469-48a2-9f36-2103458ba252<br>**Similarity:** 0.742850661277771<br>**Text:** The basic function underlying a normal distribution, aka a Gaussian, is e to the negative x squared. But you might wonder why this function? Of all the expressions we could dream up that give you s...<br>"
"**Node ID:** bda08ef1-137c-4d69-9bcc-b7005a41a13c<br>**Similarity:** 0.7431071996688843<br>**Text:** The basic function underlying a normal distribution, aka a Gaussian, is e to the negative x squared. But you might wonder why this function? Of all the expressions we could dream up that give you s...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
Expand All @@ -432,7 +395,7 @@
{
"data": {
"text/markdown": [
"**Node ID:** 6856d43b-9978-4882-a1ff-09d11913157b<br>**Similarity:** 0.7337126731872559<br>**Text:** This step is actually pretty technical, it goes a little beyond what I want to talk about here. Often use these objects called moment generating functions, that gives you a very abstract argument t...<br>"
"**Node ID:** 7d6d0f32-ce16-461b-be54-883241252e50<br>**Similarity:** 0.7335695028305054<br>**Text:** This step is actually pretty technical, it goes a little beyond what I want to talk about here. Often use these objects called moment generating functions, that gives you a very abstract argument t...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
Expand All @@ -444,7 +407,7 @@
{
"data": {
"text/markdown": [
"**Node ID:** 27e3d1f6-4b30-4087-a93d-5484edc814d3<br>**Similarity:** 0.7068501114845276<br>**Text:** This is the important point. All of the stuff that's involving s is now entirely separate from the integrated variable. This remaining integral is a little bit tricky. I did a whole video on it. It...<br>"
"**Node ID:** 519fb788-3927-4842-ad5c-88be61deaf65<br>**Similarity:** 0.7069740295410156<br>**Text:** The essence of what we want to compute is what the convolution between two copies of this function looks like. If you remember, in the last video, we had two different ways to visualize convolution...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
Expand All @@ -456,7 +419,7 @@
{
"data": {
"text/markdown": [
"**Node ID:** 320ed2fc-eeaa-48a3-a216-b90e0316b1a8<br>**Similarity:** 0.7065496444702148<br>**Text:** The essence of what we want to compute is what the convolution between two copies of this function looks like. If you remember, in the last video, we had two different ways to visualize convolution...<br>"
"**Node ID:** f265c3fb-3c9f-4f36-aa2a-fb15efff9783<br>**Similarity:** 0.706935465335846<br>**Text:** This is the important point. All of the stuff that's involving s is now entirely separate from the integrated variable. This remaining integral is a little bit tricky. I did a whole video on it. It...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
Expand Down Expand Up @@ -503,63 +466,63 @@
"name": "stdout",
"output_type": "stream",
"text": [
"('The video titled \"A pretty reason why Gaussian + Gaussian = Gaussian\" by '\n",
" '3Blue1Brown covers several aspects of the Gaussian function, also known as '\n",
" 'the normal distribution. Here are the key points discussed in the video:\\n'\n",
"('The video by 3Blue1Brown, titled \"A pretty reason why Gaussian + Gaussian = '\n",
" 'Gaussian,\" covers several aspects of the Gaussian function, also known as '\n",
" \"the normal distribution. Here's a summary of the key points discussed in the \"\n",
" 'video:\\n'\n",
" '\\n'\n",
" '1. **Central Limit Theorem**: The video begins by discussing the central '\n",
" 'limit theorem, which states that the sum of multiple copies of a random '\n",
" 'variable tends to look like a normal distribution. As the number of summed '\n",
" 'variable tends to look like a normal distribution. As the number of '\n",
" 'variables increases, the approximation to a normal distribution becomes '\n",
" 'better.\\n'\n",
" '\\n'\n",
" '2. **Convolution of Random Variables**: The process of adding two random '\n",
" 'variables is mathematically represented by a convolution of their respective '\n",
" 'distributions. The video explains how to visualize the convolution operation '\n",
" 'using two methods, with a focus on the second method involving diagonal '\n",
" 'slices.\\n'\n",
" 'distributions. The video explains the concept of convolution and how it is '\n",
" 'used to find the distribution of the sum of two random variables.\\n'\n",
" '\\n'\n",
" '3. **Gaussian Function**: The Gaussian function is more complex than just '\n",
" '\\\\(e^{-x^2}\\\\). The full formula includes a normalization factor to ensure '\n",
" 'the area under the curve is one, making it a valid probability distribution. '\n",
" 'The standard deviation (\\\\(\\\\sigma\\\\)) is used to describe the spread of the '\n",
" 'distribution, and the mean (\\\\(\\\\mu\\\\)) can be included to shift the center '\n",
" 'of the distribution.\\n'\n",
" '\\\\( e^{-x^2} \\\\). The full formula includes a scaling factor to ensure the '\n",
" 'area under the curve is 1 (making it a valid probability distribution), a '\n",
" 'standard deviation parameter \\\\( \\\\sigma \\\\) to describe the spread, and a '\n",
" 'mean parameter \\\\( \\\\mu \\\\) to shift the center. However, the video focuses '\n",
" 'on centered distributions with \\\\( \\\\mu = 0 \\\\).\\n'\n",
" '\\n'\n",
" '4. **Convolution of Two Gaussians**: The video explores what happens when '\n",
" 'you add two normally distributed random variables, which is equivalent to '\n",
" 'computing the convolution of two Gaussian functions. The author presents a '\n",
" 'visual approach to understand this calculation by exploiting the rotational '\n",
" 'symmetry of the graph of \\\\(e^{-x^2}\\\\).\\n'\n",
" '4. **Visualizing Convolution**: The video presents a visual method to '\n",
" 'understand the convolution of two Gaussian functions using diagonal slices '\n",
" 'on the xy-plane. This method involves looking at the probability density of '\n",
" 'landing on a point (x, y) as \\\\( f(x) \\\\times g(y) \\\\), where f and g are '\n",
" 'the two distributions being convolved.\\n'\n",
" '\\n'\n",
" '5. **Rotational Symmetry and Slices**: The video demonstrates that the graph '\n",
" 'of \\\\(e^{-x^2} \\\\cdot e^{-y^2}\\\\) is rotationally symmetric around the '\n",
" 'origin, which is a unique property of Gaussian functions. By examining '\n",
" 'diagonal slices of this graph, the author shows how to compute the area '\n",
" 'under these slices, which corresponds to the convolution of the two '\n",
" 'functions.\\n'\n",
" '5. **Rotational Symmetry**: A key property of the Gaussian function is its '\n",
" 'rotational symmetry, which is unique to bell curves. This symmetry is '\n",
" 'exploited in the video to simplify the calculation of the convolution. By '\n",
" 'rotating the graph 45 degrees, the computation becomes easier because the '\n",
" 'integral only involves one variable.\\n'\n",
" '\\n'\n",
" '6. **Resulting Distribution**: The convolution of two Gaussian functions is '\n",
" 'shown to be another Gaussian function. This is a special property because '\n",
" 'convolutions typically result in a different kind of function. The video '\n",
" 'explains that this property is key to understanding why the Gaussian '\n",
" 'function is the universal shape approached by the central limit theorem.\\n'\n",
" '6. **Result of Convolution**: The video demonstrates that the convolution of '\n",
" 'two Gaussian functions is another Gaussian function. This is a special '\n",
" 'property because convolutions typically result in a different kind of '\n",
" 'function. The standard deviation of the resulting Gaussian is \\\\( \\\\sqrt{2} '\n",
" '\\\\times \\\\sigma \\\\) if the original Gaussians had the same standard '\n",
" 'deviation.\\n'\n",
" '\\n'\n",
" '7. **Standard Deviation of the Result**: When reintroducing the constants '\n",
" 'for a normal distribution, the video concludes that the convolution of two '\n",
" 'normal distributions with mean 0 and standard deviation \\\\(\\\\sigma\\\\) '\n",
" 'results in another normal distribution with a standard deviation of '\n",
" '\\\\(\\\\sqrt{2} \\\\cdot \\\\sigma\\\\).\\n'\n",
" '7. **Proof of Central Limit Theorem**: The video explains that the '\n",
" 'convolution of two Gaussians being another Gaussian is a crucial step in '\n",
" 'proving the central limit theorem. It shows that the Gaussian function is a '\n",
" 'fixed point in the space of distributions, and since all distributions with '\n",
" 'finite variance tend towards a single universal shape, that shape must be '\n",
" 'the Gaussian.\\n'\n",
" '\\n'\n",
" '8. **Implications for the Central Limit Theorem**: The video emphasizes that '\n",
" 'the computation of the convolution of two Gaussians is fundamental to the '\n",
" 'central limit theorem. It shows that the Gaussian distribution is a fixed '\n",
" 'point in the space of distributions, which is why it is the shape that '\n",
" 'emerges from the central limit theorem.\\n'\n",
" '8. **Connection to Pi**: The video also touches on the connection between '\n",
" 'the Gaussian function and the number Pi, which appears in the formula for '\n",
" 'the normal distribution.\\n'\n",
" '\\n'\n",
" 'Throughout the video, the author provides visual examples and explanations '\n",
" 'to help viewers understand the mathematical concepts involved in the '\n",
" 'Gaussian function and its properties related to probability and statistics.')\n"
" 'The video aims to provide an intuitive geometric argument for why the sum of '\n",
" 'two normally distributed random variables is also normally distributed, and '\n",
" 'how this relates to the central limit theorem and the special properties of '\n",
" 'the Gaussian function.')\n"
]
}
],
Expand Down

0 comments on commit f7b3fee

Please sign in to comment.