Skip to content

Commit

Permalink
Merge pull request #17 from WukLab/cognify_post
Browse files Browse the repository at this point in the history
Center align and uniform width images
  • Loading branch information
reyna-abhyankar authored Nov 28, 2024
2 parents 2e7baad + 5c52c74 commit df3ca2e
Showing 1 changed file with 21 additions and 10 deletions.
31 changes: 21 additions & 10 deletions content/posts/cognify.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ Building high-quality, cost-effective generative AI applications is challenging
---
Author: Yiying Zhang, Reyna Abhyankar, Zijian He

![logo-light](/images/cognify/cognify-light.jpg)
<p align="center">
<img src="https://github.com/GenseeAI/cognify/blob/main/cognify.jpg?raw=true" alt="Cognify logo">
</p>

**TL;DR:** Building high-quality, cost-effective generative AI applications is challenging due to the absence of systematic methods for tuning, testing, and optimization. We introduce **Cognify**, a tool that automatically enhances generation quality and reduces costs for generative AI workflows, including those written with LangChain, DSPy, and annotated Python. Built on a novel foundation of hierarchical, workflow-level optimization, Cognify delivers up to a **48% improvement in generation quality** and up to **9x cost reduction**. Cognify is publicly available at [https://github.com/GenseeAI/cognify](https://github.com/GenseeAI/cognify).

Expand Down Expand Up @@ -54,25 +56,28 @@ We introduce Cognify, a comprehensive, multi-objective gen-AI workflow optimizer
</tr>
</table> -->

<p align="center">
<img src="https://github.com/WukLab/wuklab_sysml/blob/main/static/images/cognify/output_figure.png?raw=true" width=800>
</p>

![results](/images/cognify/output_figure.png)


## The Secret Sauce: Holistic Workflow Hyperparameter Tuning

The core idea of Cognify is to perform optimizations at the workflow level instead of at each individual workflow component. Since upstream components’ generation highly affects how downstream components perform, optimizing components in isolation can negatively impact both final generation quality and workflow execution costs. Instead, Cognify optimizes the entire workflow by experimenting with various combinations of cogs applied across workflow components and assessing the effectiveness of these combinations based on the quality of the final output.



![arch](/images/cognify/architecture.png)
<p align="center">
<img src="https://github.com/WukLab/wuklab_sysml/blob/main/static/images/cognify/architecture.png?raw=true" width=800>
</p>


***Cognify Overall Architecture***

A key challenge in holistic workflow optimization is the associated optimization cost, both in terms of monetary cost and optimization time. A simplistic approach would involve performing a grid search over every possible cog combination, leading to exponential optimization costs. To confront this challenge, Cognify employs two strategies. First, we treats a workflow as a “*grey box*” and cogs as hyper-parameters to the workflow. The grey-box approach is in between white boxes and black boxes where we analyze and utilize workflows’ internal structures but not what each workflow step does. We design a customized Bayesian Optimizer for tuning workflow hyperparameters (i.e., cogs) based on the grey-box information, which allows for efficient exploration of the cog space. Second, we categorize cogs into two distinct layers: an outer loop containing cogs that alter workflow structures (like adding or removing components or rearranging their order) and an inner loop containing cogs that do not affect workflow structures (like prompt tuning and model selection). This two-layer approach reduces the overall search space that our Bayesian Optimizer needs to navigate.


![optim](/images/cognify/search_gif_large.gif)
<p align="center">
<img src="https://github.com/WukLab/wuklab_sysml/blob/main/static/images/cognify/search_gif_large.gif?raw=true" width=800>
</p>


***Cognify Optimization Flow***
Expand Down Expand Up @@ -122,13 +127,19 @@ Below, we provide the generation results of a data visualization task for plotti
</table>
-->

![demo_all](/images/cognify/demos_all_white.png)

<p align="center">
<img src="https://github.com/WukLab/wuklab_sysml/blob/main/static/images/cognify/demos_all_white.png?raw=true" width=800>
</p>

![cognify](/images/cognify/cognify.png)

<p align="center">
<img src="https://github.com/WukLab/wuklab_sysml/blob/main/static/images/cognify/cognify.png?raw=true" width=800>
</p>

**Generated by Cognify-optimized workflow**
<p align="center">
<strong>Generated by Cognify-optimized workflow</strong>
</p>


## Get Started with Cognify
Expand Down

0 comments on commit df3ca2e

Please sign in to comment.