Skip to content

Latest commit

 

History

History
257 lines (221 loc) · 31.9 KB

llm.md

File metadata and controls

257 lines (221 loc) · 31.9 KB
Raw

Section 7 : Large Language Model: Landscape

Large Language Models (in 2023)

  1. Change in perspective is necessary because some abilities only emerge at a certain scale. Some conclusions from the past are invalidated and we need to constantly unlearn intuitions built on top of such ideas.
  2. From first-principles, scaling up the Transformer amounts to efficiently doing matrix multiplications with many, many machines.
  3. Further scaling (think 10000x GPT-4 scale). It entails finding the inductive bias that is the bottleneck in further scaling.

Evolutionary Tree of Large Language Models

A Taxonomy of Natural Language Processing

  • An overview of different fields of study and recent developments in NLP. doc / ref [24 Sep 2023]

    Exploring the Landscape of Natural Language Processing Research ref [20 Jul 2023]

    NLP taxonomy

    Distribution of the number of papers by most popular fields of study from 2002 to 2022

Open-Source Large Language Models

  • The Open Source AI Definition [28 Oct 2024]
  • The LLM Index: A list of large language models (LLMs)
  • Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
  • LLM Collection: promptingguide.ai
  • Huggingface Open LLM Learboard
  • ollam: ollama-supported models
  • KoAlpaca: Alpaca for korean [Mar 2023] GitHub Repo stars
  • Pythia: How do large language models (LLMs) develop and evolve over the course of training and change as models scale? A suite of decoder-only autoregressive language models ranging from 70M to 12B parameters git [Apr 2023] GitHub Repo stars
  • OLMo: Truly open language model and framework to build, study, and advance LMs, along with the training data, training and evaluation code, intermediate model checkpoints, and training logs. git [Feb 2024] / OLMoE: fully-open LLM leverages sparse Mixture-of-Experts [Sep 2024] GitHub Repo stars GitHub Repo stars
  • Open-Sora: Democratizing Efficient Video Production for All [Mar 2024] GitHub Repo stars
  • Jamba: AI21's SSM-Transformer Model. Mamba + Transformer + MoE [28 Mar 2024]
  • Meta (aka. Facebook)
    1. Most OSS LLM models have been built on the Llama / ref / git GitHub Repo stars GitHub Repo stars
    2. Llama 2: 1) 40% more data than Llama. 2)7B, 13B, and 70B. 3) Trained on over 1 million human annotations. 4) double the context length of Llama 1: 4K 5) Grouped Query Attention, KV Cache, and Rotary Positional Embedding were introduced in Llama 2 [18 Jul 2023] demo
    3. Llama 3: 1) 7X more data than Llama 2. 2) 8B, 70B, and 400B. 3) 8K context length [18 Apr 2024]
    4. MEGALODON: Long Sequence Model. Unlimited context length. Outperforms Llama 2 model. [Apr 2024] GitHub Repo stars
    5. Llama 3.1: 405B, context length to 128K, add support across eight languages. first OSS model outperforms GTP-4o. [23 Jul 2024]
    6. Llama 3.2: Multimodal. Include text-only models (1B, 3B) and text-image models (11B, 90B), with quantized versions of 1B and 3B [Sep 2024]
    7. NotebookLlama: An Open Source version of NotebookLM [28 Oct 2024]
  • Google
    1. Gemma: Open weights LLM from Google DeepMind. git / Pytorch git [Feb 2024] GitHub Repo stars GitHub Repo stars
    2. Gemma 2 2B, 9B, 27B ref: releases [Jun 2024]
    3. DataGemma [12 Sep 2024] / NotebookLM: LLM-powered notebook. free to use, not open-source. [12 Jul 2023]
  • Qualcomm
    1. Qualcomm’s on-device AI models: Bring generative AI to mobile devices [Feb 2024]
  • xAI
    • xAI is an American AI company founded by Elon Musk in March 2023
    1. Grok: 314B parameter Mixture-of-Experts (MoE) model. Released under the Apache 2.0 license. Not includeded training code. Developed by JAX git [17 Mar 2024] GitHub Repo stars
    2. Grok-2 and Grok-2 mini [13 Aug 2024]
  • Databricks
    1. DBRX: MoE, open, general-purpose LLM created by Databricks. git [27 Mar 2024] GitHub Repo stars
  • Apple
    1. OpenELM: Apple released a Transformer-based language model. Four sizes of the model: 270M, 450M, 1.1B, and 3B parameters. [April 2024]
    2. Apple Intelligence Foundation Language Models: 1. A 3B on-device model used for language tasks like summarization and Writing Tools. 2. A large Server model used for language tasks too complex to do on-device. [10 Jun 2024]
  • Microsoft
    1. phi-series: cost-effective small language models (SLMs) x-ref
  • NVIDIA
    1. Nemotron-4 340B: Synthetic Data Generation for Training Large Language Models [14 Jun 2024]
  • Mistral
    • Founded in April 2023. French tech.
    1. open-weights models (Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, NeMo) and optimized commercial models (Mistral Small, Mistral Medium, Mistral Large) ref
    2. NeMo: 12B model with 128k context length that outperforms LLama 3 8B [18 Jul 2024]
  • Groq
    • Founded in 2016. low-latency AI inference H/W. American tech.
    1. Llama-3-Groq-Tool-Use: a model optimized for function calling [Jul 2024]
  • Alibaba
    1. Qwen series > Qwen2: 29 languages. 5 sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B. [Feb 2024] GitHub Repo stars
  • Cohere
    • Founded in 2019. Canadian multinational tech.
    1. Command R+: The performant model for RAG capabilities, multilingual support, and tool use. [Aug 2024]
  • Deepseek
    • Founded in 2023, is a Chinese company dedicated to AGI.
    1. A list of models: git
  • Tencent
    • Founded in 1998, Tencent is a Chinese company dedicated to various technology sectors, including social media, gaming, and AI development.
    • Hunyuan-Large: An open-source MoE model with open weights. [4 Nov 2024] git GitHub Repo stars
  • GPT for Domain Specific x-ref
  • MLLM (multimodal large language model) x-ref
  • Large Language Models (in 2023) x-ref
  • Llama variants emerged in 2023
    • Falcon LLM Apache 2.0 license [Mar 2023]
    • Alpaca: Fine-tuned from the LLaMA 7B model [Mar 2023]
    • vicuna: 90% ChatGPT Quality [Mar 2023]
    • dolly: Databricks [Mar 2023]
    • Cerebras-GPT: 7 GPT models ranging from 111m to 13b parameters. [Mar 2023]
    • Koala: Focus on dialogue data gathered from the web. [Apr 2023]
    • StableVicuna First Open Source RLHF LLM Chatbot [Apr 2023]
    • Upstage's 70B Language Model Outperforms GPT-3.5: ref [1 Aug 2023]

LLM for Domain Specific

  • AlphaFold3: Open source implementation of AlphaFold3 [Nov 2023] / OpenFold: PyTorch reproduction of AlphaFold 2 [Sep 2021] GitHub Repo stars GitHub Repo stars
  • BioGPT: [cnt]: Generative Pre-trained Transformer for Biomedical Text Generation and Mining git [19 Oct 2022] GitHub Repo stars
  • Galactica: A Large Language Model for Science [16 Nov 2022]
  • TimeGPT: The First Foundation Model for Time Series Forecasting git [Mar 2023] GitHub Repo stars
  • BloombergGPT: A Large Language Model for Finance [30 Mar 2023]
  • Huggingface StarCoder: A State-of-the-Art LLM for Code: git [May 2023]
  • FrugalGPT: LLM with budget constraints, requests are cascaded from low-cost to high-cost LLMs. git [9 May 2023] GitHub Repo stars
  • Code Llama: Built on top of Llama 2, free for research and commercial use. ref / git [24 Aug 2023] GitHub Repo stars
  • MechGPT: Language Modeling Strategies for Mechanics and Materials git [16 Oct 2023] GitHub Repo stars
  • MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers [27 Nov 2023]
  • EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain [30 Jan 2024]
  • SaulLM-7B: A pioneering Large Language Model for Law [6 Mar 2024]
  • Devin AI: Devin is an AI software engineer developed by Cognition AI [12 Mar 2024]
  • DeepSeek-Coder-V2: Open-source Mixture-of-Experts (MoE) code language model [17 Jun 2024] GitHub Repo stars
  • Qwen2-Math: math-specific LLM / Qwen2-Audio: large-scale audio-language model [Aug 2024] / Qwen 2.5-Coder [18 Sep 2024] GitHub Repo stars GitHub Repo stars GitHub Repo stars
  • Chai-1: a multi-modal foundation model for molecular structure prediction [Sep 2024] GitHub Repo stars
  • Prithvi WxC: In collaboration with NASA, IBM is releasing an open-source foundation model for Weather and Climate ref [20 Sep 2024]
  • AlphaChip: Reinforcement learning-based model for designing physical chip layouts. [26 Sep 2024]
  • OpenCoder: 1.5B and 8B base and open-source Code LLM, supporting both English and Chinese. [Oct 2024] GitHub Repo stars

MLLM (multimodal large language model)

  • Understanding Multimodal LLMs:💡Two main approaches to building multimodal LLMs: 1. Unified Embedding Decoder Architecture approach; 2. Cross-modality Attention Architecture approach. [3 Nov 2024]

    mllm

  • Multimodal Foundation Models: From Specialists to General-Purpose Assistants: [cnt]: A comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities. Specific-Purpose 1. Visual understanding tasks 2. Visual generation tasks General-Purpose 3. General-purpose interface. [18 Sep 2023]

  • Awesome Multimodal Large Language Models: Latest Papers and Datasets on Multimodal Large Language Models, and Their Evaluation. [Jun 2023] GitHub Repo stars

  • CLIP: [cnt]: CLIP (Contrastive Language-Image Pretraining), Trained on a large number of internet text-image pairs and can be applied to a wide range of tasks with zero-shot learning. git [26 Feb 2021] GitHub Repo stars

  • BLIP-2 [30 Jan 2023]: [cnt]: Salesforce Research, Querying Transformer (Q-Former) / git / ref / 📺 / BLIP: [cnt]: git [28 Jan 2022] GitHub Repo stars

    • Q-Former (Querying Transformer): A transformer model that consists of two submodules that share the same self-attention layers: an image transformer that interacts with a frozen image encoder for visual feature extraction, and a text transformer that can function as both a text encoder and a text decoder.
    • Q-Former is a lightweight transformer which employs a set of learnable query vectors to extract visual features from the frozen image encoder. It acts as an information bottleneck between the frozen image encoder and the frozen LLM.

  • TaskMatrix, aka VisualChatGPT: [cnt]: Microsoft TaskMatrix git; GroundingDINO + SAM / git [8 Mar 2023] GitHub Repo stars GitHub Repo stars

  • GroundingDINO: [cnt]: DINO with Grounded Pre-Training for Open-Set Object Detection git [9 Mar 2023] GitHub Repo stars

  • LLaVa: [cnt]: Large Language-and-Vision Assistant git [17 Apr 2023]

    • Simple linear layer to connect image features into the word embedding space. A trainable projection matrix W is applied to the visual features Zv, transforming them into visual embedding tokens Hv. These tokens are then concatenated with the language embedding sequence Hq to form a single sequence. Note that Hv and Hq are not multiplied or added, but concatenated, both are same dimensionality.
    • LLaVA-1.5: [cnt]: is out! git: Changing from a linear projection to an MLP cross-modal. [5 Oct 2023] GitHub Repo stars

  • MiniGPT-4 & MiniGPT-v2: [cnt]: Enhancing Vision-language Understanding with Advanced Large Language Models git [20 Apr 2023]

  • openai/shap-e Generate 3D objects conditioned on text or images [3 May 2023] git GitHub Repo stars

  • Drag Your GAN: [cnt]: Interactive Point-based Manipulation on the Generative Image Manifold git [18 May 2023] GitHub Repo stars

  • Video-ChatGPT: [cnt]: a video conversation model capable of generating meaningful conversation about videos. / git [8 Jun 2023] GitHub Repo stars

  • moondream: an OSS tiny vision language model. Built using SigLIP, Phi-1.5, LLaVA dataset. [Dec 2023] GitHub Repo stars

  • MiniCPM-V: MiniCPM-Llama3-V 2.5: A GPT-4V Level Multimodal LLM on Your Phone [Jan 2024] GitHub Repo stars

  • mini-omni2: ref: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities. [15 Oct 2024] GitHub Repo stars

  • Vision capability to a LLM ref [22 Aug 2023]

    • The model has three sub-models:

      1. A model to obtain image embeddings
      2. A text model to obtain text embeddings
      3. A model to learn the relationships between them

    • This is analogous to adding vision capability to a LLM.

  • Meta (aka. Facebook)

    1. facebookresearch/ImageBind: [cnt]: ImageBind One Embedding Space to Bind Them All git [9 May 2023] GitHub Repo stars
    2. facebookresearch/segment-anything(SAM): [cnt]: The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. git [5 Apr 2023] GitHub Repo stars
    3. facebookresearch/SeamlessM4T: [cnt]: SeamlessM4T is the first all-in-one multilingual multimodal AI translation and transcription model. This single model can perform speech-to-text, speech-to-speech, text-to-speech, and text-to-text translations for up to 100 languages depending on the task. ref [22 Aug 2023]
    4. Chameleon: Early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. The unified approach uses fully token-based representations for both image and textual modalities. [16 May 2024]
    5. Models and libraries

  • Microsoft

    1. Language Is Not All You Need: Aligning Perception with Language Models Kosmos-1: [cnt] [27 Feb 2023]
    2. Kosmos-2: [cnt]: Grounding Multimodal Large Language Models to the World [26 Jun 2023]
    3. Kosmos-2.5: [cnt]: A Multimodal Literate Model [20 Sep 2023]
    4. BEiT-3: [cnt]: Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks [22 Aug 2022]
    5. TaskMatrix.AI: [cnt]: TaskMatrix connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. [29 Mar 2023]
    6. Florence-2: Advancing a unified representation for various vision tasks, demonstrating specialized models like CLIP for classification, GroundingDINO for object detection, and SAM for segmentation. ref [10 Nov 2023]
    7. LLM2CLIP: Directly integrating LLMs into CLIP causes catastrophic performance drops. We propose LLM2CLIP, a caption contrastive fine-tuning method that leverages LLMs to enhance CLIP. [7 Nov 2024]

  • Google

    1. Gemini 1.5: 1 million token context window, 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. [Feb 2024]
    2. Foundation Models: Gemini, Veo, Gemma etc.

  • Anthrophic

    1. Claude 3 Opus, the largest version of the new LLM, outperforms rivals GPT-4 and Google’s Gemini 1.0 Ultra. Three variants: Opus, Sonnet, and Haiku. [Mar 2024]

  • Apple

    1. 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities. [13 Jun 2024]

  • Benchmarking Multimodal LLMs.

    • LLaVA-1.5 achieves SoTA on a broad range of 11 tasks incl. SEED-Bench.
    • SEED-Bench: [cnt]: Benchmarking Multimodal LLMs git [30 Jul 2023] GitHub Repo stars

  • Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models ref [25 Sep 2024]

  • Optimizing Memory Usage for Training LLMs and Vision Transformers: When applying 10 techniques to a vision transformer, we reduced the memory consumption 20x on a single GPU. ref / git [2 Jul 2023] GitHub Repo stars

Generative AI Landscape

Model Description Strengths Weaknesses
GANs Two neural networks, a generator and a discriminator, work together. The generator creates synthetic samples, and the discriminator distinguishes between real and generated samples. Unsupervised learning, able to mimic data distributions without labeled data, and are versatile in applications like image synthesis, super-resolution, and style transfer Known for potentially unstable training and less diversity in generation.
VAEs Consists of an encoder and a decoder. The encoder maps input data into a low-dimensional representation, and the decoder reconstructs the original input data from this representation. e.g, DALLE Efficient at learning latent representations and can be used for tasks like data denoising and anomaly detection, in addition to data generation. Dependent on an approximate loss function.
Diffusion Models Consists of forward and reverse diffusion processes. Forward diffusion adds noise to input data until white noise is obtained. The reverse diffusion process removes the noise to recover the original data. e.g, Stable Diffusion Capable of producing high-quality, step-by-step samples. Multi-step (often 1000) generation process.