Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mobile friendliness for understanding-incontext blog post #30

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions _posts/2022-08-01-understanding-incontext.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,26 @@ draft: True
<script type="text/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_CHTML"></script>

<style>
.twitterblock {
width: 32%;
display: inline-block;
}

/* The block of code below tells the browsers what to do on a screen that has a width of 1000px or less */

@media (max-width: 1200px) {

.twitterblock {
width: 320px;
display: block; /* Stops it from floating */
margin: auto; /* Ensures that it is centered */
margin-bottom: 10px; /* Space between the stacked elements */

}

}
</style>

*In this post, we provide a Bayesian inference framework for in-context learning in large language models like GPT-3 and show empirical evidence for our framework, highlighting the differences from traditional supervised learning. This blog post primarily draws from the theoretical framework for in-context learning from [An Explanation of In-context Learning as Implicit Bayesian Inference](https://arxiv.org/abs/2111.02080) [^BI] and experiments from [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://arxiv.org/abs/2202.12837) [^RRD].*

Expand Down Expand Up @@ -47,11 +67,11 @@ Two examples of in-context learning, where a language model (LM) is given a list
**What can in-context learning do?** On many benchmark NLP benchmarks, in-context learning is competitive with models trained with much more labeled data and is state-of-the-art on LAMBADA (commonsense sentence completion) and TriviaQA (question answering). Perhaps even more exciting is the array of applications that in-context learning has enabled people to spin up in just a few hours, including writing code from natural language descriptions, helping with app design mockups, and generalizing spreadsheet functions:

{% figure %}
<div style="display: inline-block; width: 33%"> <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Here&#39;s a sentence describing what Google&#39;s home page should look and here&#39;s GPT-3 generating the code for it nearly perfectly. <a href="https://t.co/m49hoKiEpR">pic.twitter.com/m49hoKiEpR</a></p>&mdash; Sharif Shameem (@sharifshameem) <a href="https://twitter.com/sharifshameem/status/1283322990625607681?ref\_src=twsrc%5Etfw">July 15, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div>
<div class="twitterblock"> <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Here&#39;s a sentence describing what Google&#39;s home page should look and here&#39;s GPT-3 generating the code for it nearly perfectly. <a href="https://t.co/m49hoKiEpR">pic.twitter.com/m49hoKiEpR</a></p>&mdash; Sharif Shameem (@sharifshameem) <a href="https://twitter.com/sharifshameem/status/1283322990625607681?ref\_src=twsrc%5Etfw">July 15, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div>

<div style="display: inline-block; width: 33%"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">This changes everything. 🤯<br><br>With GPT-3, I built a Figma plugin to design for you.<br><br>I call it &quot;Designer&quot; <a href="https://t.co/OzW1sKNLEC">pic.twitter.com/OzW1sKNLEC</a></p>&mdash; jordan singer (@jsngr) <a href="https://twitter.com/jsngr/status/1284511080715362304?ref\_src=twsrc%5Etfw">July 18, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div>
<div class="twitterblock"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">This changes everything. 🤯<br><br>With GPT-3, I built a Figma plugin to design for you.<br><br>I call it &quot;Designer&quot; <a href="https://t.co/OzW1sKNLEC">pic.twitter.com/OzW1sKNLEC</a></p>&mdash; jordan singer (@jsngr) <a href="https://twitter.com/jsngr/status/1284511080715362304?ref\_src=twsrc%5Etfw">July 18, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div>

<div style="display: inline-block; width: 31%"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">=GPT3()... the spreadsheet function to rule them all.<br><br>Impressed with how well it pattern matches from a few examples.<br><br>The same function looked up state populations, peoples&#39; twitter usernames and employers, and did some math. <a href="https://t.co/W8FgVAov2f">pic.twitter.com/W8FgVAov2f</a></p>&mdash; Paul Katsen (@pavtalk) <a href="https://twitter.com/pavtalk/status/1285410751092416513?ref_src=twsrc%5Etfw">July 21, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></div>
<div class="twitterblock"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">=GPT3()... the spreadsheet function to rule them all.<br><br>Impressed with how well it pattern matches from a few examples.<br><br>The same function looked up state populations, peoples&#39; twitter usernames and employers, and did some math. <a href="https://t.co/W8FgVAov2f">pic.twitter.com/W8FgVAov2f</a></p>&mdash; Paul Katsen (@pavtalk) <a href="https://twitter.com/pavtalk/status/1285410751092416513?ref_src=twsrc%5Etfw">July 21, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></div>
{% endfigure %}

In-context learning allows users to quickly build models for a new use case without worrying about fine-tuning and storing new parameters for each task. It typically requires very few training examples to get a prototype working, and the natural language interface is intuitive even for non-experts.
Expand Down