Code

pull#1

Code

Open

codybum wants to merge 156 commits intoinnovationcore:maininnovationcore/lorax:mainfrom predibase:mainpredibase/lorax:mainCopy head branch name to clipboard

+25,682-19,204

Commits

Commits on Jul 3, 2024

• Infer dtype from model config when not explicitly specified (#534)

  arnavgarg1

  authored
  24cb494

  Copy full SHA for 24cb494

Commits on Jul 9, 2024

• Lorax NER (#531)

  magdyksaleh

  authored
  a3ad209

  Copy full SHA for a3ad209

Commits on Jul 12, 2024

• Fix gemma2 (#539)

  Infernaught

  authored
  35f666a

  Copy full SHA for 35f666a

Commits on Jul 15, 2024

• no warm up (#540)

  magdyksaleh

  authored
  2dd5277

  Copy full SHA for 2dd5277

Commits on Jul 17, 2024

• Preload adapters during init (#543)

  Show description for 5c25e26
  

  tgaddair

  and

  noyoshi

  authored
  5c25e26

  Copy full SHA for 5c25e26

Commits on Jul 19, 2024

• Move kv cache allocation to router to ensure correct block allocation (#545)

  tgaddair

  authored
  5a7a1be

  Copy full SHA for 5a7a1be

• Fix : compile bug causing models to error with 'lora' key not found (#547)

  ajtejankar

  authored
  1adc076

  Copy full SHA for 1adc076

• Tokenize inputs in router (#548)

  tgaddair

  authored
  452ac73

  Copy full SHA for 452ac73

Commits on Jul 23, 2024

• Add support for Llama 3 rotary embeddings (#551)

  tgaddair

  authored
  240079b

  Copy full SHA for 240079b

• Apply chat template in router to properly validate input length (#538)

  tgaddair

  authored
  59631a0

  Copy full SHA for 59631a0

• Fix: short circuit download, load, offload for preloaded adapters (#552)

  tgaddair

  authored
  07addea

  Copy full SHA for 07addea

Commits on Jul 25, 2024

• Update PyTorch, CUDA, vLLM, and Bitsandbytes (#553)

  ajtejankar

  authored
  5cefe6e

  Copy full SHA for 5cefe6e

Commits on Jul 26, 2024

• Fix the attention bug caused by upgrading vLLM (#555)

  ajtejankar

  authored
  2e81331

  Copy full SHA for 2e81331

• Allow eager_prefill to be set in Helm chart (#557)

  bdalal

  authored
  15a38d5

  Copy full SHA for 15a38d5

• Added missing nvidia-ml-py package (#558)

  tgaddair

  authored
  d1a4d09

  Copy full SHA for d1a4d09

Commits on Jul 30, 2024

• Support FP8 for Mistral (#559)

  Show description for 91ef7a8
  

  ajtejankar

  and

  tgaddair

  authored
  91ef7a8

  Copy full SHA for 91ef7a8

Commits on Jul 31, 2024

• Move NER output formatting to server (#561)

  magdyksaleh

  authored
  efa4bff

  Copy full SHA for efa4bff

• (WIP) Support FP8 for LLaMa (#562)

  ajtejankar

  authored
  ea5d74b

  Copy full SHA for ea5d74b

Commits on Aug 2, 2024

• parse headers for errored requests (#564)

  noyoshi

  authored
  3b2cc05

  Copy full SHA for 3b2cc05

• handle folders for predibase (#565)

  noyoshi

  authored
  e594ce0

  Copy full SHA for e594ce0

• feat: support lazy loading the lora module for reducing the loading p… (#434)

  thincal

  authored
  2e47e77

  Copy full SHA for 2e47e77

Commits on Aug 5, 2024

• Fix LM head interaction with Medusa (#567)

  tgaddair

  authored
  8454a82

  Copy full SHA for 8454a82

Commits on Aug 7, 2024

• bump version (#569)

  noyoshi

  authored
  387e17e

  Copy full SHA for 387e17e

Commits on Aug 8, 2024

• Fix adapter mask when using speculative decoding + LM head LoRA (#570)

  tgaddair

  authored
  6a3d65b

  Copy full SHA for 6a3d65b

Commits on Aug 9, 2024

• enable mistral nemo (#568)

  magdyksaleh

  authored
  da37b5e

  Copy full SHA for da37b5e

• Doc updates for Medusa training (#544)

  arnavgarg1

  authored
  25fd9ca

  Copy full SHA for 25fd9ca

Commits on Aug 12, 2024

• include prompt and generated tokens as part of logs (#574)

  noyoshi

  authored
  2e5d97c

  Copy full SHA for 2e5d97c

Commits on Aug 14, 2024

• add support for classification in bert (#573)

  magdyksaleh

  authored
  a3f91a6

  Copy full SHA for a3f91a6

Commits on Aug 16, 2024

• Support classify batch (#577)

  magdyksaleh

  authored
  12dc740

  Copy full SHA for 12dc740

• Fix outlines compatibility with speculative decoding (#578)

  tgaddair

  authored
  ca2e643

  Copy full SHA for ca2e643

Commits on Aug 17, 2024

• Adding longrope for serve Phi-3 (#576)

  huytuong010101

  authored
  bb2cdd6

  Copy full SHA for bb2cdd6

Commits on Aug 21, 2024

• Added flashinfer support (#580)

  tgaddair

  authored
  76c6e71

  Copy full SHA for 76c6e71

• Add prefix caching (#581)

  tgaddair

  authored
  1d2b514

  Copy full SHA for 1d2b514

• Install flashinfer in Docker (#582)

  tgaddair

  authored
  1095b52

  Copy full SHA for 1095b52

• Pin flashinfer

  tgaddair

  authored
  53cd493

  Copy full SHA for 53cd493

Commits on Aug 22, 2024

• Fix qwen lora (#585)

  magdyksaleh

  authored
  f365466

  Copy full SHA for f365466

• Add "pbase" to adapter_source docstrings (#583)

  alexsherstinsky

  authored
  c3c95da

  Copy full SHA for c3c95da

Commits on Aug 26, 2024

• Add Llava Next (VLM) (#586)

  tgaddair

  authored
  20c544f

  Copy full SHA for 20c544f

• feat : use --no-cache-dir flag to pip in dockerfiles to save space (#587)

  Show description for 551b5be
  Rajpratik71

  authored
  551b5be

  Copy full SHA for 551b5be

• Add prerequisites to readme (#584)

  csabakecskemeti

  authored
  d1c02ba

  Copy full SHA for d1c02ba

• add new agnostic health endpoint (#588)

  Show description for 3a74c25
  

  magdyksaleh

  and

  tgaddair

  authored
  3a74c25

  Copy full SHA for 3a74c25

• Add missing configs (#590)

  magdyksaleh

  authored
  335a786

  Copy full SHA for 335a786

Commits on Aug 27, 2024

• Address rust compiler warnings (#589)

  magdyksaleh

  authored
  65d8529

  Copy full SHA for 65d8529

• support bge-base-en-v1.5 (#593)

  magdyksaleh

  authored
  b1578d2

  Copy full SHA for b1578d2

Commits on Sep 6, 2024

• Fix ner entity merging (#596)

  magdyksaleh

  authored
  39b6c8e

  Copy full SHA for 39b6c8e

• Support FlashInfer for BERT (#597)

  tgaddair

  authored
  a8ca5cb

  Copy full SHA for a8ca5cb

Commits on Sep 10, 2024

• Speed up NER inference (#598)

  magdyksaleh

  authored
  19301c7

  Copy full SHA for 19301c7

Commits on Sep 12, 2024

• Fix class ner (#602)

  magdyksaleh

  authored
  b119ca1

  Copy full SHA for b119ca1

Commits on Sep 13, 2024

• Disable healthcheck tracing and add metrics to classify + classify_batch endpoints (#603)

  magdyksaleh

  authored
  654a31d

  Copy full SHA for 654a31d

• Added launcher args for preloaded_adapter_source and backend (#604)

  tgaddair

  authored
  a446ae4

  Copy full SHA for a446ae4

Commits on Sep 17, 2024

• Fix classify and classify_batch for Python client (#608)

  tgaddair

  authored
  a6b60e9

  Copy full SHA for a6b60e9

Commits on Sep 18, 2024

• Parallelize tokenization for /classify_batch and remove block allocator for non-causal LMs (#609)

  tgaddair

  authored
  9919ae1

  Copy full SHA for 9919ae1

• Fix dependencies to address high urgency dependabot alerts (#612)

  magdyksaleh

  authored
  66c5b9c

  Copy full SHA for 66c5b9c

• upgrade poetry (#613)

  magdyksaleh

  authored
  69d712b

  Copy full SHA for 69d712b

Commits on Sep 19, 2024

• add num inputs to metrics (#615)

  magdyksaleh

  authored
  b11ff18

  Copy full SHA for b11ff18

Commits on Sep 25, 2024

• Fix deps4 (#614)

  Show description for 7a75a94
  

  magdyksaleh

  and

  arnavgarg1

  authored
  7a75a94

  Copy full SHA for 7a75a94

Commits on Sep 27, 2024

• Add done message to openai endpoints (#618)

  magdyksaleh

  authored
  38bb892

  Copy full SHA for 38bb892

Commits on Sep 28, 2024

• Add --predibase-api-token CLI arg (#617)

  joseph-predibase

  authored
  38483b9

  Copy full SHA for 38483b9

Commits on Sep 30, 2024

• Added Mllama (#619)

  tgaddair

  authored
  3584a30

  Copy full SHA for 3584a30

Commits on Oct 1, 2024

• Flash mllama (#622)

  tgaddair

  authored
  99fdc2a

  Copy full SHA for 99fdc2a

• Remove LD_PRELOAD from Docker and improve error message (#623)

  tgaddair

  authored
  ccf6b64

  Copy full SHA for ccf6b64

• support MRL embeddings for qwen2 (#621)

  magdyksaleh

  authored
  c754415

  Copy full SHA for c754415

Commits on Oct 2, 2024

• add label to id this as a lorax image (#626)

  noyoshi

  authored
  641a74f

  Copy full SHA for 641a74f

• Fix CUDA graph compilation (#627)

  tgaddair

  authored
  e3f7d6e

  Copy full SHA for e3f7d6e

• Fix CUDA graphs for Medusa (#628)

  tgaddair

  authored
  fee2ffc

  Copy full SHA for fee2ffc

• Fix retrace message (#629)

  tgaddair

  authored
  6c413b9

  Copy full SHA for 6c413b9

Commits on Oct 3, 2024

• Added ranks 96 and 128 to BGMV kernel (#630)

  tgaddair

  authored
  b9294ed

  Copy full SHA for b9294ed

• Merge weights (#600)

  Show description for 0670556
  

  magdyksaleh

  and

  tgaddair

  authored
  0670556

  Copy full SHA for 0670556

• Fix prefix plumbing and BGMV compiler dimensions (#631)

  tgaddair

  authored
  99891a0

  Copy full SHA for 99891a0

Commits on Oct 4, 2024

• Fix punica kernel compilation (#632)

  tgaddair

  authored
  a1230a5

  Copy full SHA for a1230a5

• Fix FlashInfer when not using prefix caching (#633)

  tgaddair

  authored
  d513ee8

  Copy full SHA for d513ee8

Commits on Oct 7, 2024

• Fix cuda graph tracing without lora ranks (#634)

  tgaddair

  authored
  0c1cec2

  Copy full SHA for 0c1cec2

Commits on Oct 8, 2024

• pass correct stuff to predibase-reporter (#635)

  magdyksaleh

  authored
  f1ef0ee

  Copy full SHA for f1ef0ee

Commits on Oct 15, 2024

• Look for language model lm head (#640)

  Infernaught

  authored
  2a22063

  Copy full SHA for 2a22063

• Return n choices for chat completions API (#638)

  tgaddair

  authored
  bea8834

  Copy full SHA for bea8834

• Fix llava_next for llama 3.2 vision cross attention states (#641)

  tgaddair

  authored
  4c84b4f

  Copy full SHA for 4c84b4f

Commits on Oct 16, 2024

• Add --disable-sgmv flag (#639)

  joseph-predibase

  authored
  3818e1a

  Copy full SHA for 3818e1a

• Fix compile for qwen-2.5-32b (#645)

  tgaddair

  authored
  8ac729b

  Copy full SHA for 8ac729b

• Enhance Structured Output Interface (#644)

  GirinMan

  authored
  4fb4d69

  Copy full SHA for 4fb4d69

• try using arc runner for build (#646)

  noyoshi

  authored
  c8f361e

  Copy full SHA for c8f361e

• Added backwards compatible field to OpenAI json_object API (#648)

  tgaddair

  authored
  974c2b2

  Copy full SHA for 974c2b2

• change runner 2 (#650)

  magdyksaleh

  authored
  d9ed1a6

  Copy full SHA for d9ed1a6

• feat: Function calling with output schema enforcement (#536)

  Show description for 418b9fa
  

  jeffreyftang

  and

  tgaddair

  authored
  418b9fa

  Copy full SHA for 418b9fa

Commits on Oct 21, 2024

• Chunked prefill (#653)

  tgaddair

  authored
  6c5ca67

  Copy full SHA for 6c5ca67

Commits on Oct 22, 2024

• Fix PREDIBASE_API_TOKEN env var being thrown away (#654)

  joseph-predibase

  authored
  71ca771

  Copy full SHA for 71ca771

Commits on Oct 23, 2024

• Prompt prefix caching for multi-LoRA (#655)

  tgaddair

  authored
  373c3e6

  Copy full SHA for 373c3e6

Commits on Oct 29, 2024

• Support FP8 KV Cache (#652)

  ajtejankar

  authored
  2ff1c71

  Copy full SHA for 2ff1c71

Commits on Oct 30, 2024

• Fix absent fp8_kv property on llama and qwen models (#662)

  ajtejankar

  authored
  c2441e2

  Copy full SHA for c2441e2

Commits on Oct 31, 2024

• Support for Embeddings with XLM-RoBERTa and Adapters (#656)

  jfhetzer

  authored
  3f108f7

  Copy full SHA for 3f108f7

• Fix seqlen bug for sliding window models like Mistral v0.1 (#660)

  ajtejankar

  authored
  bd92e52

  Copy full SHA for bd92e52

Commits on Nov 1, 2024

• added metrics docs, updated links in main docs (#663)

  noyoshi

  authored
  b3944ad

  Copy full SHA for b3944ad

Commits on Nov 5, 2024

• Convert to Triton Punica kernels (#658)

  tgaddair

  authored
  902a68c

  Copy full SHA for 902a68c

Commits on Nov 6, 2024

• Fix sliding window + compile bug (#666)

  ajtejankar

  authored
  e03f989

  Copy full SHA for e03f989

Commits on Nov 7, 2024

• Set maximum grpc message receive size to 2GiB (#667)

  tgaddair

  authored
  a2c1fc1

  Copy full SHA for a2c1fc1

Commits on Nov 11, 2024

• Fix frequency_penalty and presence_penalty (#672)

  tgaddair

  authored
  ebd8d0d

  Copy full SHA for ebd8d0d

• Fix scores (remove debug code) (#673)

  tgaddair

  authored
  7078765

  Copy full SHA for 7078765

Commits on Nov 12, 2024

• Fix stella embeddings + Integration tests for lorax (#668)

  magdyksaleh

  authored
  429520c

  Copy full SHA for 429520c

• Only use sha tag for running int tests (#674)

  magdyksaleh

  authored
  9358d0b

  Copy full SHA for 9358d0b

• Fix int tests 2 (#675)

  magdyksaleh

  authored
  8f2cbf7

  Copy full SHA for 8f2cbf7

• Fix top_p to allow setting it to 1.0 (#676)

  magdyksaleh

  authored
  9082d36

  Copy full SHA for 9082d36

• Add support for adapter loading in mllama (#669)

  ajtejankar

  authored
  fdda45a

  Copy full SHA for fdda45a

Commits on Nov 13, 2024

• Always build and push image before running IT (#678)

  arnavgarg1

  authored
  c1eb3dd

  Copy full SHA for c1eb3dd

• Format fixes tool calling (#680)

  magdyksaleh

  authored
  fc2cebb

  Copy full SHA for fc2cebb

Commits on Nov 14, 2024

• Record number of skipped tokens in the response (#681)

  tgaddair

  authored
  0204c2e

  Copy full SHA for 0204c2e

Commits on Nov 15, 2024

• Only push main if int tests pass (#677)

  magdyksaleh

  authored
  2388096

  Copy full SHA for 2388096

• Fix lora loading and indexing bug in mllama (#682)

  ajtejankar

  authored
  7e8af35

  Copy full SHA for 7e8af35

• Remove bad check (#683)

  magdyksaleh

  authored
  e4394d6

  Copy full SHA for e4394d6

Commits on Nov 18, 2024

• Record TTFT and TPOT in response headers (#684)

  tgaddair

  authored
  80730b7

  Copy full SHA for 80730b7

Commits on Nov 19, 2024

• Add cli arg --speculation-max-batch-size (#686)

  tgaddair

  authored
  f32a11d

  Copy full SHA for f32a11d

• Use --predibase-api-token parameter when downloading (#687)

  joseph-predibase

  authored
  1220ce8

  Copy full SHA for 1220ce8

Commits on Nov 20, 2024

• Use predibase API token when downloading pbase files (#688)

  joseph-predibase

  authored
  25f71d2

  Copy full SHA for 25f71d2

• Pbase adapter source resolution (#689)

  magdyksaleh

  authored
  d6caec4

  Copy full SHA for d6caec4

• Launcher args for compile max batch size and rank (#690)

  tgaddair

  authored
  81d0173

  Copy full SHA for 81d0173

Commits on Nov 22, 2024

• fix: Make logprob field optional for response Pydantic validation (#692)

  jeffreyftang

  authored
  c0e5798

  Copy full SHA for c0e5798

Commits on Nov 25, 2024

• Bump tornado from 6.4.1 to 6.4.2 in /server (#695)

  Show description for 02ff9f3
  

  dependabot[bot]

  and

  jeffreyftang

  authored
  02ff9f3

  Copy full SHA for 02ff9f3

Commits on Nov 26, 2024

• Handle case where adapter exists but is still being trained (#691)

  tgaddair

  authored
  69e5306

  Copy full SHA for 69e5306

Commits on Nov 27, 2024

• Add support for returning alternative_tokens with return_k_alternatives (#697)

  ajtejankar

  authored
  8f51d0e

  Copy full SHA for 8f51d0e

• Speed up best_of when using prefix caching (#698)

  tgaddair

  authored
  da95224

  Copy full SHA for da95224

• Add support for targeting cross_attn layers in mllama (#693)

  ajtejankar

  authored
  c96ff88

  Copy full SHA for c96ff88

Commits on Dec 3, 2024

• Make MEMORY_WIGGLE_ROOM a custom arg (#701)

  magdyksaleh

  authored
  5e0b003

  Copy full SHA for 5e0b003

Commits on Dec 4, 2024

• Openai embedding endpoint (#702)

  magdyksaleh

  authored
  07b77db

  Copy full SHA for 07b77db

• fix alerts (#704)

  magdyksaleh

  authored
  79e4283

  Copy full SHA for 79e4283

Commits on Dec 6, 2024

• fix classification int test (#706)

  magdyksaleh

  authored
  61b40a8

  Copy full SHA for 61b40a8

Commits on Dec 9, 2024

• Added Http Status code documentation (#703)

  andrejTodoroski21

  authored
  482aa2b

  Copy full SHA for 482aa2b

• Remove restriction on using compile + sharding (#707)

  tgaddair

  authored
  63c5eb3

  Copy full SHA for 63c5eb3

Commits on Dec 10, 2024

• Fix decoder_input_details bug (#705)

  ajtejankar

  authored
  2af302d

  Copy full SHA for 2af302d

• Update lorax images (#709)

  wvaneaton

  authored
  2aac736

  Copy full SHA for 2aac736

• Fix integration tests (#708)

  Show description for 3b25a39
  

  magdyksaleh

  and

  Ubuntu

  authored
  3b25a39

  Copy full SHA for 3b25a39

• Update lorax logo (#710)

  wvaneaton

  authored
  9a63db1

  Copy full SHA for 9a63db1

• Fix integration tests2 (#712)

  Show description for 37123f4
  

  magdyksaleh

  and

  Ubuntu

  authored
  37123f4

  Copy full SHA for 37123f4

• Update index.md (#711)

  wvaneaton

  authored
  64c4891

  Copy full SHA for 64c4891

• Update index.md (#713)

  wvaneaton

  authored
  ea6a62a

  Copy full SHA for ea6a62a

Commits on Dec 13, 2024

• Fix FlashInfer + Medusa bug (#715)

  ajtejankar

  authored
  4d7f468

  Copy full SHA for 4d7f468

• some metrics and logging for batch classify (#714)

  magdyksaleh

  authored
  e314845

  Copy full SHA for e314845

Commits on Dec 15, 2024

• Fix stella model and client <> embedding compatibility (#717)

  magdyksaleh

  authored
  abf9f39

  Copy full SHA for abf9f39

Commits on Dec 16, 2024

• make tokeziner name required for dev ex (#718)

  magdyksaleh

  authored
  170aae1

  Copy full SHA for 170aae1

• Adds support for the Solar architecture (#719)

  arnavgarg1

  authored
  ddd1c2d

  Copy full SHA for ddd1c2d

Commits on Dec 17, 2024

• Memory estimate fixes (#720)

  magdyksaleh

  authored
  6dfb215

  Copy full SHA for 6dfb215

• fix and add test (#721)

  magdyksaleh

  authored
  69bb989

  Copy full SHA for 69bb989

Commits on Dec 23, 2024

• Fix residual summation for the Solar architecture (#723)

  arnavgarg1

  authored
  52710ea

  Copy full SHA for 52710ea

Commits on Dec 28, 2024

• Better health check (#724)

  magdyksaleh

  authored
  7dbabda

  Copy full SHA for 7dbabda

Commits on Jan 3, 2025

• Support JSON response for /metrics endpoint, add client method for metrics (#725)

  arnavgarg1

  authored
  7cd7d06

  Copy full SHA for 7cd7d06

• Add tokenizer-name arg to router launch in developer docs (#727)

  arnavgarg1

  authored
  0c486c9

  Copy full SHA for 0c486c9

• Fix lm_head with tied word embeddings for Qwen2 and Qwen2.5 (#728)

  arnavgarg1

  authored
  eef95f1

  Copy full SHA for eef95f1

Commits on Jan 6, 2025

• IBM Granite support (#730)

  jppgks

  authored
  269004f

  Copy full SHA for 269004f

Commits on Jan 8, 2025

• Adds support for BGMV kernels needed to run Phi-4 (#731)

  ajtejankar

  authored
  667e248

  Copy full SHA for 667e248

• Bump jinja2 from 3.1.4 to 3.1.5 in /server (#729)

  Show description for e36d9c8
  

  dependabot[bot]

  and

  arnavgarg1

  authored
  e36d9c8

  Copy full SHA for e36d9c8

Commits on Jan 9, 2025

• offline mode support (#735)

  noyoshi

  authored
  0f91a7e

  Copy full SHA for 0f91a7e

Commits on Jan 13, 2025

• Fix envvar plumbing when setting args to false (#739)

  tgaddair

  authored
  095e892

  Copy full SHA for 095e892

• improvements to the chart (#716)

  noyoshi

  authored
  7195d25

  Copy full SHA for 7195d25

Commits on Jan 14, 2025

• Fix incompiltibility between --revision and --adapter-id (#741)

  tgaddair

  authored
  73408b5

  Copy full SHA for 73408b5

• fix probes in chart (#740)

  noyoshi

  authored
  01c536f

  Copy full SHA for 01c536f

Commits on Jan 15, 2025

• Enable eager prefill by default (#732)

  ajtejankar

  authored
  9bab135

  Copy full SHA for 9bab135

Commits on Jan 17, 2025

• Stricter tracing inclusions (#738)

  noyoshi

  authored
  30c97cb

  Copy full SHA for 30c97cb

Commits on Jan 18, 2025

• fix block size in health check (#742)

  magdyksaleh

  authored
  223c554

  Copy full SHA for 223c554

Commits on Jan 22, 2025

• dyaminc scaling (Thanks ajinkia 😄 ) (#746)

  magdyksaleh

  authored
  05b2d63

  Copy full SHA for 05b2d63