transcript_summarization_final.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.text_splitter import NLTKTextSplitter, RecursiveCharacterTextSplitter\n",
    "from langchain.chains.summarize import load_summarize_chain\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "load_dotenv()\n",
    "# os.environ[\"OPENAI_API_KEY\"] = your_key_here\n",
    "\n",
    "with open(\"transcript_medium.txt\", encoding=\"utf-8\") as f:\n",
    "    transcript = f.read()\n",
    "\n",
    "# transcript[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10202"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat_llm = ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\", request_timeout=600)\n",
    "chat_llm.get_num_tokens(transcript)  # 10,202 tokens; exceeds gpt-3.5's max 4,096 tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is Atomic Soul, where we connect with the visionary mind shaping the future of the tech industry.\n",
      "\n",
      "I'm your host Jepson Taylor and my mission is to fight for you, our listeners, as we chase the ideas that will help us stay ahead in a world of rapid innovation.\n",
      "\n",
      "We'll journey through the realms of tech, AI, and the human brain, discovering how to live a more satisfying life in the face of relentless progress.\n",
      "\n",
      "Let's forge a path into the future together.\n",
      "\n",
      "Sol, is that how you pronounce your name?\n",
      "\n",
      "It is.\n",
      "\n",
      "It is.\n",
      "\n",
      "And you're one of the first to get it right.\n",
      "\n",
      "I've heard Sal, Sul, Sol.\n",
      "\n",
      "Oh, yeah.\n",
      "\n",
      "Well, it's very fitting because the name of this podcast is Atomic Soul.\n",
      "\n",
      "I promise that's not why I tried to invite you onto this podcast.\n",
      "\n",
      "But I met you in Riyadh a couple of weeks ago at LEAP.\n",
      "\n",
      "Lots of people there.\n",
      "\n",
      "It was it was a crazy conference.\n",
      "\n",
      "And I think in a good way.\n",
      "\n",
      "But I would love for the listeners to understand what has your journey been to get to where you are now.\n",
      "\n",
      "And you're also chemie.\n",
      "\n",
      "Yeah.\n",
      "\n",
      "Also a chemical engineer.\n",
      "\n",
      "Yeah.\n",
      "\n",
      "I love that.\n",
      "\n",
      "So what is the Sol story?\n",
      "\n",
      "Oh, geez.\n",
      "\n",
      "I always envy individuals who had a plan and everything fell into place in according to that plan, because I'm like, it just sounds so much easier than the path I've had to go through.\n",
      "\n",
      "But there's this amazing diagram that I saw both on LinkedIn and on Instagram, Jeff, where it's like some people think the road to success is linear and it just goes up and forward.\n",
      "\n",
      "When in actuality, it's just a bunch of squiggles and you have a bunch of dips and peaks and valleys and all of the above.\n",
      "\n",
      "And that's kind of been my journey.\n",
      "\n",
      "I always say I was at the wrong place the wrong time.\n",
      "\n",
      "And I took the job no one wanted.\n",
      "\n",
      "And it was way back when I would say.\n",
      "\n",
      "Yeah, the late 90s, early 2000s, I actually didn't know what I wanted to do.\n",
      "\n",
      "I majored in chemistry, took all the chemical engineering courses.\n",
      "\n",
      "I sort of played water polo.\n",
      "\n",
      "I found out that I was a great athlete in high school, but barely mediocre at the collegiate level.\n",
      "\n",
      "And so I was training four to six hours a day and swimming.\n",
      "\n",
      "But I was always second string.\n",
      "\n",
      "And I had to talk with my coach and I'll never forget her, Mo.\n",
      "\n",
      "And she's just like, listen, you're good, but you're not good enough.\n",
      "\n",
      "And, you know, I love the way you show up every day, your enthusiasm, your energy.\n",
      "\n",
      "You're you're very, you know, passion full, if you will, and purpose led.\n",
      "\n",
      "But you're five, three and a half, five, four, you know, on your best day.\n",
      "\n",
      "And everyone else is like five, nine, five, ten.\n",
      "\n",
      "So she's like, you can continue to train with us, but you're just never going to be the first string.\n",
      "\n",
      "You're always going to be second string.\n",
      "\n",
      "As a sophomore year in college, I was like, OK, maybe professional sports isn't in my my trajectory.\n",
      "\n",
      "And then oddly enough, I randomly fell into rugby.\n",
      "\n",
      "Kathy Flores, rest in peace.\n",
      "\n",
      "She she found me on Bancroft Way.\n",
      "\n",
      "And she's like, have you ever tried rugby?\n",
      "\n",
      "I was like, what on earth is rugby?\n",
      "\n",
      "And they're like, football without pads.\n",
      "\n",
      "I'm like, why would women ever play football without pads?\n",
      "\n",
      "It's horrible.\n",
      "\n",
      "But turns out that I absolutely love the sport.\n",
      "\n",
      "I was built for the sport because I was five, three.\n",
      "\n",
      "I had a lower center of gravity.\n",
      "\n",
      "It was hard to tackle.\n",
      "\n",
      "And I had these amazing ball handling skills because what I learned in water polo, which is treading in water, I now had stable ground.\n",
      "\n",
      "And so it was a sport that I naturally picked up.\n",
      "\n",
      "And it was probably the first life lesson I ever had of knowing your shelf life.\n",
      "\n",
      "What was a deficiency and a weakness in water polo turned out to be a fantastic strength of mine in rugby.\n",
      "\n",
      "And so I've always carried that along with me as this weird trajectory of mine continued.\n",
      "\n",
      "Long story short, I love sports, turned into becoming a professional.\n",
      "\n",
      "I didn't know what I wanted to do, and I took a bunch of random jobs.\n",
      "\n",
      "And, you know, I got into the data and analytics and data science and digital and AI space completely accidentally.\n",
      "\n",
      "And after taking these odd jobs, the one pivot that happened was as there was this boutique agency that I was working for and I was a partner in charge of business development.\n",
      "\n",
      "And we were up against Accenture at the time.\n",
      "\n",
      "And we really wanted this deal in Southern California.\n",
      "\n",
      "But the problem was, is that the CIO, who we had a great relationship with, said, I'd love nothing more than to give you guys this business.\n",
      "\n",
      "You show up every day, you're present, you obviously know your stuff.\n",
      "\n",
      "But that's a risk on my end.\n",
      "\n",
      "You've got to give me something, anything that shows you guys are better fit than a larger organization like Accenture, which from our perspective would be, you know, risk free.\n",
      "\n",
      "So we said, OK. And what ended up happening was is the firm shipped me off to Texas for three months to become MDM, SAP certified.\n",
      "\n",
      "It was a very technical course.\n",
      "\n",
      "Not many people were certified back then.\n",
      "\n",
      "And turns out I passed the exam and I was one of 11 in the nation.\n",
      "\n",
      "So we come back.\n",
      "\n",
      "My boss, who was a CEO, comes back, who has a relationship with the CIO and says, All right, we have something.\n",
      "\n",
      "So it was not only in charge of business development, which you enjoy working with her, but she's now SAP certified and she's one of 11.\n",
      "\n",
      "And there's no one else in Accenture that has it.\n",
      "\n",
      "And we won the deal.\n",
      "\n",
      "So that kind of started the data journey.\n",
      "\n",
      "Now you say wrong place, wrong time.\n",
      "\n",
      "And I got voluntold to go get certified and pass.\n",
      "\n",
      "Otherwise, we were going to lose the deal.\n",
      "\n",
      "But I went from being an individual contributor because I I fell into that world and I oddly enough loved it to managing a small team, then a group, then the division, then creating practices and frameworks and hiring talent and being a mentor.\n",
      "\n",
      "And I got my first break at IBM Watson when Jay Bolasimo went over.\n",
      "\n",
      "And I really wanted to be a part of the Watson group.\n",
      "\n",
      "I thought I was doing amazing, amazing things.\n",
      "\n",
      "And it started two key industries and both in health care and in finance.\n",
      "\n",
      "What year is this?\n",
      "\n",
      "This was 2011 was when Watson beat Ken Jennings in Jeopardy.\n",
      "\n",
      "And that's when IBM said, OK, we're double downing and we're building a team and we're taking a Watson to market.\n",
      "\n",
      "And so I transferred over around 2012.\n",
      "\n",
      "So you've got this girl for all intents and purposes who played sports.\n",
      "\n",
      "One coach tells her, you know, she's great, but not good enough.\n",
      "\n",
      "The other coach is like, you're going to be a natural.\n",
      "\n",
      "Let's keep going.\n",
      "\n",
      "And then having to switch gears from sports to being a professional, but not knowing what I wanted to do and taking a bunch of jobs to them being shipped off into Texas to go get MDM certified with a ton of pressure because we were going to lose the deal if we didn't.\n",
      "\n",
      "To then joining IBM and starting as a person who knows a space.\n",
      "\n",
      "But I really didn't.\n",
      "\n",
      "I was learning on the job as well.\n",
      "\n",
      "Typical consulting, you'll learn what you earn.\n",
      "\n",
      "But my aptitude and my hustle and just my wanting this to know the space sort of built the trajectory.\n",
      "\n",
      "And so when IBM chose to take Watson to market, I was really fortunate that there was a leader that I had worked for in the SAP division who went over to Watson and he brought me with him.\n",
      "\n",
      "And that's kind of what changed everything.\n",
      "\n",
      "And then from there, it was a series of successes and failures and lessons learned because that space at the time was just not defined whatsoever.\n",
      "\n",
      "Well, ultimately, after about 70 years at IBM, I left to go join Ernst and Young and Royal Caribbean was my key, my key client.\n",
      "\n",
      "So I was living in New York City, traveling to Miami, helping them with their four year digital transformation roadmap and not digital transformation the way we define it in 2023.\n",
      "\n",
      "This was 2016.\n",
      "\n",
      "Like this was the first generation of digital transformation.\n",
      "\n",
      "So nothing on the cloud yet.\n",
      "\n",
      "Barely.\n",
      "\n",
      "We were still on massive server farms.\n",
      "\n",
      "Like, remember the world of the Teradata's?\n",
      "\n",
      "Well, that used to be cool.\n",
      "\n",
      "It used to be cool.\n",
      "\n",
      "Yes.\n",
      "\n",
      "Cloud operations wasn't cloud strategy.\n",
      "\n",
      "Those things weren't even languages told.\n",
      "\n",
      "And I remember 2015, 16 when I joined Royal as a full time employee.\n",
      "\n",
      "It was like the first gen of even developing data lakes.\n",
      "\n",
      "That was a new concept.\n",
      "\n",
      "We hadn't learned the do's and the don'ts of data lakes and the notion of it can easily become the data ocean.\n",
      "\n",
      "Everyone was just starting off and me and my leadership team at the time.\n",
      "\n",
      "We were one of the first teams to actually build it.\n",
      "\n",
      "And it was an amazing adventure.\n",
      "\n",
      "And again, I had another great leader.\n",
      "\n",
      "His name was Jay Schneider.\n",
      "\n",
      "I've had two great leaders, both named Jay, oddly enough.\n",
      "\n",
      "And I learned a lot from him about product development and being maniacally focused on the consumer experience and design and the way you present things fundamentally matters.\n",
      "\n",
      "So you take someone with my background that was mostly on data and analytics and engineering and data science, and then the lessons learned around AI deployments.\n",
      "\n",
      "And now I get to learn around product development, platform development, UI, UX design, graphic design.\n",
      "\n",
      "And that just completely shifted my mindset as well.\n",
      "\n",
      "So as my career trajectory moved on to Sony Music, then Mark Pharmaceuticals, Estee Lauder and so on and so forth.\n",
      "\n",
      "Every job has taught me what never to do again and what I'd love to continue to do.\n",
      "\n",
      "And here's where we are now.\n",
      "\n",
      "And with every job you have, mistakes are made, failures are realized and lessons are learned.\n",
      "\n",
      "I think that's something I celebrate later in my career.\n",
      "\n",
      "The seasoned realist, not the naive optimist.\n",
      "\n",
      "But at one time, at one point in our careers, you had to be the naive optimist.\n",
      "\n",
      "Absolutely.\n",
      "\n",
      "That's half the time how you make the mistakes.\n",
      "\n",
      "Yeah.\n",
      "\n",
      "So you realize doing the right thing isn't what's going to help you always be successful.\n",
      "\n",
      "And no one ever teaches you that.\n",
      "\n",
      "So doing the right thing for the company isn't necessarily what defines your success or how people perceive your success within an organization.\n",
      "\n",
      "And so you come in with the competency and a subject matter expertise.\n",
      "\n",
      "So you have the DQ, the digital quotient.\n",
      "\n",
      "You combine it with your IQ, but the EQ part you develop over time and you realize how complicated it is to navigate that.\n",
      "\n",
      "And those are the gotchas that no one can teach you.\n",
      "\n",
      "You have to experience it for good or for bad.\n",
      "\n",
      "Yeah.\n",
      "\n",
      "Well, and I was telling you that story.\n",
      "\n",
      "I didn't get the EQ side on the business.\n",
      "\n",
      "And I had that awkward interaction with my CEO saying, we know you're smart.\n",
      "\n",
      "That's why we hired you.\n",
      "\n",
      "And that it was a terrible thing.\n",
      "\n",
      "Like in the meeting, it was that I was helpless to know.\n",
      "\n",
      "I don't know what's going on.\n",
      "\n",
      "Yeah.\n",
      "\n",
      "Yeah, it's true.\n",
      "\n",
      "And even with now, like as I have leaders report into me, really brilliant individuals.\n",
      "\n",
      "Right.\n",
      "\n",
      "So I hire a VP of data sciences, and I've had to do that several times over.\n",
      "\n",
      "And it's a really smart group of individuals, and I can't do what they can do.\n",
      "\n",
      "But sometimes understanding the EQ in the situation or understanding the chess moves and why people are saying what they're saying.\n",
      "\n",
      "And they really don't mean it, but they're doing it for the following reasons.\n",
      "\n",
      "That is really hard to explain.\n",
      "\n",
      "And unless you've tripped over those mine, like those landmines, you wouldn't know that that's what's happening.\n",
      "\n",
      "And so I try and even with my leadership team, coach them as much as possible without telling them the answers.\n",
      "\n",
      "But half the time, you just have to go through it.\n",
      "\n",
      "And speaking of going through it, digital transformation comes with change management antibodies.\n",
      "\n",
      "So when you go into an organization, if you have a brilliant thing on the board with your team, maybe explain that how what advice would you give to new CDOs as they're going down this path?\n",
      "\n",
      "A new org, one that's behind, and they need to bring them into the future, but they're dealing with it.\n",
      "\n",
      "And they need to bring them into the future, but they're dealing with.\n",
      "\n",
      "What advice would you give them?\n",
      "\n",
      "Make sure you have a backbone and not a wishbone.\n",
      "\n",
      "I like that.\n",
      "\n",
      "The CEO gig is tough.\n",
      "\n",
      "Everyone and no one owns data.\n",
      "\n",
      "And so part of your job is relationship building.\n",
      "\n",
      "And you've got to get in with the business because they drive the P&L at the end of the day.\n",
      "\n",
      "The role of a CDO function, while we may contribute to the P, for the majority of it, we're a big fat L. We're a cost center.\n",
      "\n",
      "And so understanding what their strategic priorities are, understanding what they value, understanding the business language is probably the most important thing that no one also teaches you because they're like, OK, data is a mess.\n",
      "\n",
      "It's fragmented.\n",
      "\n",
      "It's it's disparate.\n",
      "\n",
      "Let's consolidate.\n",
      "\n",
      "Let's create the semantic layer.\n",
      "\n",
      "Let's enable data science as a service, analytics as a service.\n",
      "\n",
      "Let's.\n",
      "\n",
      "Yeah, yeah, yeah.\n",
      "\n",
      "That's great.\n",
      "\n",
      "But I call that the back end defense.\n",
      "\n",
      "There's an offensive play and there's a defensive play.\n",
      "\n",
      "Doing all of that stuff are the fundamentals and that's defensive play.\n",
      "\n",
      "But what's going to gain and I don't want to use the word notoriety, but what's going to help the business pay attention and keep their focus and have you continue to be a seat at the table is when you play when you develop your offensive playbook.\n",
      "\n",
      "And that means you're sitting at the table with them and understanding what are their drivers?\n",
      "\n",
      "Is it top line growth?\n",
      "\n",
      "Is it increasing margins?\n",
      "\n",
      "Is it operational excellence and driving more efficiency so they can do more with less and better accuracy?\n",
      "\n",
      "Figure out their language.\n",
      "\n",
      "What's important to them and develop an offensive playbook that meets their needs while you're figuring out everything else on the back end.\n",
      "\n",
      "So for new CDO, I would say it's it's you know, you're going to have to wear your big boy pants and girls.\n",
      "\n",
      "Like, that's just part of it.\n",
      "\n",
      "It is an exercise in patience and resilience.\n",
      "\n",
      "You got to have a balance between an offensive playbook and a defensive playbook.\n",
      "\n",
      "And you must understand the business language and and if you want to have a seat at the table.\n",
      "\n",
      "Otherwise, you're always going to be referred to as a cost center and someone who's aren't they the technical people, aren't they the IT people?\n",
      "\n",
      "Yeah.\n",
      "\n",
      "And that's that's scary.\n",
      "\n",
      "One of the things I was thinking about when you're talking about the defensive play is I think sometimes as employees, we think all the activity we're doing matters.\n",
      "\n",
      "So if I'm working for you and I'm saying, look, I worked 100 hours a week over and over again, look at all the work I'm doing.\n",
      "\n",
      "But if you're not mapping it back to the business, no one cares how many hours you're working.\n",
      "\n",
      "So I think that that's a different mindset that comes as you kind of grow up to the ranks, I think.\n",
      "\n",
      "Very true.\n",
      "\n",
      "I know some phenomenal CDOs who actually don't build anything.\n",
      "\n",
      "And they're just great storytellers.\n",
      "\n",
      "And the business likes them because they're friendly and they're more functional than technical.\n",
      "\n",
      "But if you talk about what have they tangibly built, what value have they created, what new capability has been introduced that wasn't there before?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://python.langchain.com/en/latest/modules/indexes/text_splitters.html\n",
    "\n",
    "# NLTK\n",
    "nltk_splitter = NLTKTextSplitter(chunk_size=15000, chunk_overlap=1000)\n",
    "nltk_texts = nltk_splitter.split_text(transcript)\n",
    "print(nltk_texts[0])\n",
    "len(nltk_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is Atomic Soul, where we connect with the visionary mind shaping the future of the tech industry. I'm your host Jepson Taylor and my mission is to fight for you, our listeners, as we chase the ideas that will help us stay ahead in a world of rapid innovation. We'll journey through the realms of tech, AI, and the human brain, discovering how to live a more satisfying life in the face of relentless progress. Let's forge a path into the future together. Sol, is that how you pronounce your name? It is. It is. And you're one of the first to get it right. I've heard Sal, Sul, Sol. Oh, yeah. Well, it's very fitting because the name of this podcast is Atomic Soul. I promise that's not why I tried to invite you onto this podcast. But I met you in Riyadh a couple of weeks ago at LEAP. Lots of people there. It was it was a crazy conference. And I think in a good way. But I would love for the listeners to understand what has your journey been to get to where you are now. And you're also chemie. Yeah. Also a chemical engineer. Yeah. I love that. So what is the Sol story? Oh, geez. I always envy individuals who had a plan and everything fell into place in according to that plan, because I'm like, it just sounds so much easier than the path I've had to go through. But there's this amazing diagram that I saw both on LinkedIn and on Instagram, Jeff, where it's like some people think the road to success is linear and it just goes up and forward. When in actuality, it's just a bunch of squiggles and you have a bunch of dips and peaks and valleys and all of the above. And that's kind of been my journey. I always say I was at the wrong place the wrong time. And I took the job no one wanted. And it was way back when I would say. Yeah, the late 90s, early 2000s, I actually didn't know what I wanted to do. I majored in chemistry, took all the chemical engineering courses. I sort of played water polo. I found out that I was a great athlete in high school, but barely mediocre at the collegiate level. And so I was training four to six hours a day and swimming. But I was always second string. And I had to talk with my coach and I'll never forget her, Mo. And she's just like, listen, you're good, but you're not good enough. And, you know, I love the way you show up every day, your enthusiasm, your energy. You're you're very, you know, passion full, if you will, and purpose led. But you're five, three and a half, five, four, you know, on your best day. And everyone else is like five, nine, five, ten. So she's like, you can continue to train with us, but you're just never going to be the first string. You're always going to be second string. As a sophomore year in college, I was like, OK, maybe professional sports isn't in my my trajectory. And then oddly enough, I randomly fell into rugby. Kathy Flores, rest in peace. She she found me on Bancroft Way. And she's like, have you ever tried rugby? I was like, what on earth is rugby? And they're like, football without pads. I'm like, why would women ever play football without pads? It's horrible. But turns out that I absolutely love the sport. I was built for the sport because I was five, three. I had a lower center of gravity. It was hard to tackle. And I had these amazing ball handling skills because what I learned in water polo, which is treading in water, I now had stable ground. And so it was a sport that I naturally picked up. And it was probably the first life lesson I ever had of knowing your shelf life. What was a deficiency and a weakness in water polo turned out to be a fantastic strength of mine in rugby. And so I've always carried that along with me as this weird trajectory of mine continued. Long story short, I love sports, turned into becoming a professional. I didn't know what I wanted to do, and I took a bunch of random jobs. And, you know, I got into the data and analytics and data science and digital and AI space completely accidentally. And after taking these odd jobs, the one pivot that happened was as there was this boutique agency that I was working for and I was a partner in charge of business development. And we were up against Accenture at the time. And we really wanted this deal in Southern California. But the problem was, is that the CIO, who we had a great relationship with, said, I'd love nothing more than to give you guys this business. You show up every day, you're present, you obviously know your stuff. But that's a risk on my end. You've got to give me something, anything that shows you guys are better fit than a larger organization like Accenture, which from our perspective would be, you know, risk free. So we said, OK. And what ended up happening was is the firm shipped me off to Texas for three months to become MDM, SAP certified. It was a very technical course. Not many people were certified back then. And turns out I passed the exam and I was one of 11 in the nation. So we come back. My boss, who was a CEO, comes back, who has a relationship with\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# RecursiveCharacterTextSplitter\n",
    "recursive_splitter = RecursiveCharacterTextSplitter(\n",
    "    separators=[\n",
    "        \"\\n\\n\",\n",
    "        \"\",  # https://github.com/hwchase17/langchain/issues/1663#issuecomment-1469161790\n",
    "    ],\n",
    "    chunk_size=5000,\n",
    "    chunk_overlap=500,\n",
    ")\n",
    "recursive_texts = recursive_splitter.split_text(transcript)\n",
    "print(recursive_texts[0])\n",
    "len(recursive_texts)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Will go with NLTK text splitter as sentence structure is maintained when text is split into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now we have 4 documents and the first one has 3340 tokens\n"
     ]
    }
   ],
   "source": [
    "docs = nltk_splitter.create_documents([transcript])\n",
    "docs[0]\n",
    "num_docs = len(docs)\n",
    "num_tokens_first_doc = chat_llm.get_num_tokens(docs[0].page_content)\n",
    "print(\n",
    "    f\"Now we have {num_docs} documents and the first one has {num_tokens_first_doc} tokens\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the 4 documents will be summarized using `map_prompt`. Think of it as four summaries.  \n",
    "In this prompt, I'm trying to find a balance - a summary yet not too condensed, since a final summary will take place later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_prompt = \"\"\"\n",
    "The following is part of an intimate conversation between two individuals:\n",
    "\"{text}\"\n",
    "You are tasked to take notes - these will be used as teaching materials.\n",
    "Document any major life events, career and achievements, beliefs and values, \n",
    "impact and legacy, personal traits and characteristics, and any other information that\n",
    "would be essential when creating these lessons. \n",
    "These notes should be detailed and include relevant quotes where possible (they\n",
    "have to be full sentences).\n",
    "Exclude any mention of sponsors.\n",
    "Do not hallucinate or make up any information.\n",
    "NOTES AND QUOTES:\n",
    "\"\"\"\n",
    "map_prompt_template = PromptTemplate(template=map_prompt, input_variables=[\"text\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The four summaries will then be summarized together into one final summary using `combine_prompt`.  \n",
    "Here I explictly state to summarize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_prompt = \"\"\"\n",
    "You are an expert in writing biographies and capturing the motivations, emotions, \n",
    "successes, failures, and everyday realities of individuals.\n",
    "From the following text delimited by triple backquotes, write a concise summary that \n",
    "contains multiple sections. \n",
    "```{text}```\n",
    "Each section is an insightful life lesson. If lessons are similar, combine the sections.\n",
    "Supplement each section with quotes that best captures the essence of the lesson.\n",
    "Once all sections are complete, combine sections based on lesson similarity and overlaps. \n",
    "Rank each section for its insight, depth, transformative potential and practicality.\n",
    "Sections containing more details and quotes should be ranked higher.\n",
    "From this ranking, select the top 10 sections and discard the rest. \n",
    "Don't include the final ranking in the summary.\n",
    "Use bullet points for each section.\n",
    "Use a tone that reflects empathy, authenticity, understanding, and encouragement.\n",
    "Do not hallucinate or make up any information.\n",
    "CONCISE SUMMARY WITH MAXIMUM 10 SECTIONS:\n",
    "\"\"\"\n",
    "\n",
    "combine_prompt_template = PromptTemplate(\n",
    "    template=combine_prompt, input_variables=[\"text\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Sol's Journey: A Series of Squiggles\n",
      "- Sol's life and career have been filled with ups and downs, and she describes it as a series of squiggles rather than a linear path to success.\n",
      "- \"My journey has been a series of squiggles, not a straight line.\"\n",
      "- \"Life is not a straight line, it's a series of squiggles.\"\n",
      "- \"It's okay to have a squiggly path, as long as you keep moving forward.\"\n",
      "\n",
      "2. Importance of Emotional Intelligence in the Business World\n",
      "- Sol emphasizes the importance of EQ (emotional intelligence) in navigating the complexities of the business world.\n",
      "- \"Emotional intelligence is key in the business world.\"\n",
      "- \"EQ is just as important as IQ in the business world.\"\n",
      "- \"Understanding and managing emotions is crucial for success in the business world.\"\n",
      "\n",
      "3. Mapping Activities Back to Business and Creating Value\n",
      "- Sol believes in mapping the work employees do back to the business and creating value.\n",
      "- \"Every activity should be mapped back to the business and create value.\"\n",
      "- \"Creating value should be the focus of every task.\"\n",
      "- \"Don't just focus on the number of hours worked, focus on creating value.\"\n",
      "\n",
      "4. Importance of Being a Great Storyteller\n",
      "- Sol highlights the importance of being a great storyteller in the role of a Chief Data Officer (CDO).\n",
      "- \"Being a great storyteller is essential for a CDO.\"\n",
      "- \"Telling a compelling story is key to gaining support and buy-in.\"\n",
      "- \"A great CDO is a great storyteller.\"\n",
      "\n",
      "5. Patience and Understanding in AI and Digital Transformation\n",
      "- Sol emphasizes the importance of patience and understanding when working with business executives who may not fully grasp the potential of AI and digital transformation.\n",
      "- \"Patience and understanding are crucial when introducing AI and digital transformation.\"\n",
      "- \"Not everyone will immediately understand the potential of AI, and that's okay.\"\n",
      "- \"Take the time to educate and explain the benefits of AI and digital transformation.\"\n",
      "\n",
      "6. The Potential of AI to Save Brain Cycles and Create Capacity\n",
      "- Sol discusses the potential of AI to save wasted brain cycles and create capacity for more valuable and complex tasks.\n",
      "- \"AI has the potential to save wasted brain cycles and free up capacity.\"\n",
      "- \"Automating routine tasks with AI allows for more focus on valuable and complex tasks.\"\n",
      "- \"AI can be a game-changer in terms of efficiency and productivity.\"\n",
      "\n",
      "7. Balancing Personal and Professional Responsibilities\n",
      "- Sol acknowledges the struggle of balancing personal and professional responsibilities.\n",
      "- \"Finding a balance between personal and professional responsibilities is a constant challenge.\"\n",
      "- \"It's important to prioritize self-care and personal well-being.\"\n",
      "- \"Don't neglect your personal life in pursuit of professional success.\"\n",
      "\n",
      "8. Constant Drive to Learn and Stay Mentally Fit\n",
      "- Sol has a constant drive to learn and stay mentally fit.\n",
      "- \"Never stop learning and growing.\"\n",
      "- \"Stay mentally fit by constantly seeking new knowledge and experiences.\"\n",
      "- \"A curious mind is a powerful tool.\"\n",
      "\n",
      "9. Building Relationships and Understanding the Market\n",
      "- Sol believes in the importance of building relationships and understanding the players in the market space.\n",
      "- \"Building relationships is key to success in any industry.\"\n",
      "- \"Understanding the market and its players gives you a competitive edge.\"\n",
      "- \"Networking and relationship-building should be a priority.\"\n",
      "\n",
      "10. Overcoming Self-Doubt and Embracing Change\n",
      "- Sol has experienced moments of self-doubt but has learned to embrace change and thrive on exploration.\n",
      "- \"Overcome self-doubt by embracing change and taking risks.\"\n",
      "- \"Don't let self-doubt hold you back from pursuing your goals.\"\n",
      "- \"Embrace change and see it as an opportunity for growth.\"\n",
      "\n",
      "Note: The sections have been ranked based on their transformative potential and practicality.\n"
     ]
    }
   ],
   "source": [
    "custom_summary_chain = load_summarize_chain(\n",
    "    llm=chat_llm,\n",
    "    chain_type=\"map_reduce\",\n",
    "    map_prompt=map_prompt_template,\n",
    "    combine_prompt=combine_prompt_template,\n",
    "    verbose=False,\n",
    ")\n",
    "\n",
    "custom_output = custom_summary_chain.run(docs)\n",
    "print(custom_output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}