From 3e0a52daf7855723bf6b58332ca777d546fdaed1 Mon Sep 17 00:00:00 2001 From: Philipp Trachsel Date: Tue, 10 Dec 2024 15:09:24 +0100 Subject: [PATCH 1/2] draft --- topics/w6-live-programming.md | 67 ++++++++++++++++++++++++++++++++--- 1 file changed, 63 insertions(+), 4 deletions(-) diff --git a/topics/w6-live-programming.md b/topics/w6-live-programming.md index f2af36d..bb11c89 100644 --- a/topics/w6-live-programming.md +++ b/topics/w6-live-programming.md @@ -1,11 +1,70 @@ # W6 - Live Programming +## SnipPy: Small-step live programming by example -### Reading List +SnipPy is a tool that combines two user assisting programming paradigms: live programming and programming by example. +Live programming refers to programming environments that provide near-instant feedback, enabling programmers to immediately observe the effects of code changes. This can range from simple tools like REPLs (Read-Eval-Print Loops) to systems like PythonTutor (Week 4) or Sketch-n-sketch (Week 5), which offer visualizations and interactive debugging. On the other hand, Programming by example focuses on generating code by taking input/output pairs as demonstrations. A synthesizer analyzes these pairs and attempts to infer a program that produces the desired results -#### Required Reading +In SnipPy, live programming is facilitated by projection boxes, which dynamically display the current state of program variables beside each line of code. Programmers can step through the code line by line and observe how the program state changes in real time. This visualization is a powerful aid in writing code, debugging, and understanding new code, as it allows developers to reason visually about how their code manipulates data. (INSERT EXAMPLE IMAGE FOR PROJECTION BOXES) +Building upon this snippy now integrates programming by example into projection boxes. On a new line of code, the user can provide input/output pairs into a projection box and the tool will try to synthesize a matching expression. + +The inner workings of the synthesizer are quite easy to understand. It has an abstract grammar that +represents the possible expressions that can be generated (INSERT EXAMPLES OR EXAMPLE IMAGE). +Note that it is limited to a subset of Python - it can only operate on +integer and string types and functions whose arguments and return values are of these types. +The synthesizer then recursively generates expressions by expanding the grammar rules, until it finds an expression that matches the types of the input/output pairs. If expression correctly matches the actual pairs, it is returned to the user, otherwise the synthesizer will continue searching for a correct expression until it runs out of time. +To ensure meaningful results, the synthesizer uses a number of heuristics to guide the search, for details see the paper.(NOTE FOR REVIEW: I think going into more technical detail here is not fitting for a blogpost - what do you think?) + +Now that we know how SnipPy works, let’s dive into a few examples to see where it shines and where it might run into some challenges: First, lets consider a program that abbreviates names. We give the synthesizer the input/output pairs +- "Adam Smith" -> "A.S." +- "Augusta Ada King" -> "A.A.K." +- "Alan Turing" -> "A.T." + +```python +#Synthesized code: +abbreviation = ".".join([var[0] for var in name.split(" ")]) +``` +This is a concise and correct solution to the problem. +Next, let’s consider an example where we extract the domain name from an email address. Again we provide the synthesizer with input/output pairs: +- user1@example.com -> example.com +- john@ethz.ch -> ethz.ch +- jane@gmail.com -> gmail.com +- food@mensacorp.org -> mensacorp.org + +```python +#Synthesized code:¨ + splitchar = "@" + domain = email[1:-1][email.find(splitchar):-1] + email[-2:len(email)] +``` +This solution, while less elegant than the first one, is still correct. However we can see that the synthesizer has some limitations. First of all, the splitchar has to be user-defined, else synthesis fails. Secondly, the solution for some reason first removes the first and last character of the email before extracting the domain, and then adds them back in. This is obviously unnecessary and is an artifact of the synthesizers enumeration based approach - which is not able to reason about the logic of the problem. +Lastly, lets consider a program that takes a list of integers and returns the last element. +- [1, 3, 5, 6] -> 6 +- [3, 2, 7, 5] -> 5 + +```python +#Synthesized code: + last_elem = len(input_list) + 2 // min(input_list) +``` +What? This is absolutely not what we were looking for. (a very simple solution would be ```input_list[-1]```) When doing the math, we can see that the expression is indeed correct for our input/output pairs. However, it is not a correct solution to the problem. This shows that the synthesizer is very bad at using list indeces. Diving deeper into more examples could reveal even more limitations of the synthesizer. + +While SnipPy presents an innovative fusion of live programming and programming by example, its current synthesizer +struggles to meet the practical needs of developers. +However, the landscape of code synthesis has changed quite dramatically since the publication of this tool. +New large language models like GPT, BERT or Claude are able to reason about code and problems in a much more accurate way than the limited enumeration approach of SnipPy. It would be interesting to see how the interface of SnipPy could be combined with an LLM based synthesizer. +Tools like github copilot function in a similar way - where the user provides a prompt and the tool is able to generate code in-editor. + +A promising step in this direction is LEAP, a tool that integrates LLM-based code synthesis with live programming. LEAP generates multiple code completions for a given problem, allowing users to preview the options. Upon selection, LEAP displays the chosen code with corresponding projection boxes based on sample inputs, which visualize the program state. Unlike SnipPy, LEAP primarily uses projection boxes for code verification rather than problem specification. This shift aligns better with the strengths of LLMs, which can reason about prior code context and high-level problem requirements. + +In conclusion, SnipPy demonstrates the potential of integrating live programming with programming by example, while its limitations reveal opportunities for more advanced synthesis techniques to enhance developer tools. + +### Sources +#### SnipPy * [Snippy](https://snippy.goto.ucsd.edu/editor): Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020, October). Small-step live programming by example. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (pp. 614-626). -#### Optional Reading +#### Live Programming +* [Projection Boxes](https://cseweb.ucsd.edu/\~lerner/papers/projection-boxes-chi2020.pdf): Lerner, S. (2020, April). Projection boxes: On-the-fly reconfigurable visualization for live programming. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-7). +* [PythonTutor](https://pythontutor.com/): Guo, P. J. (2013, March). Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education (pp. 579-584). +* [Sketch-n-sketch](https://ravichugh.github.io/sketch-n-sketch/): Hempel, B., Lubin, J., & Chugh, R. (2019, October). Sketch-n-sketch: Output-directed programming for svg. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (pp. 281-292). -* [Projection Box](https://cseweb.ucsd.edu/\~lerner/papers/projection-boxes-chi2020.pdf): Lerner, S. (2020, April). Projection boxes: On-the-fly reconfigurable visualization for live programming. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-7). +#### LLM based code synthesis combined with live programming +* [LEAP](https://doi.org/10.1145/3613904.3642495): Kasra Ferdowsi, Ruanqianqian (Lisa) Huang, Michael B. James, Nadia Polikarpova, and Sorin Lerner. 2024. Validating AI-Generated Code with Live Programming. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 143, 1–8. https://doi.org/10.1145/3613904.3642495 \ No newline at end of file From 9634075506928991ba9d593415e6409adc58f623 Mon Sep 17 00:00:00 2001 From: Philipp Trachsel Date: Tue, 31 Dec 2024 17:32:44 +0100 Subject: [PATCH 2/2] final version - less tech focused and more open --- topics/w6-live-programming.md | 181 ++++++++++++++++++++++++---------- 1 file changed, 131 insertions(+), 50 deletions(-) diff --git a/topics/w6-live-programming.md b/topics/w6-live-programming.md index bb11c89..65b45f4 100644 --- a/topics/w6-live-programming.md +++ b/topics/w6-live-programming.md @@ -1,70 +1,151 @@ -# W6 - Live Programming -## SnipPy: Small-step live programming by example +# W6 – Live Programming +## SnipPy: Small-Step Live Programming by Example -SnipPy is a tool that combines two user assisting programming paradigms: live programming and programming by example. -Live programming refers to programming environments that provide near-instant feedback, enabling programmers to immediately observe the effects of code changes. This can range from simple tools like REPLs (Read-Eval-Print Loops) to systems like PythonTutor (Week 4) or Sketch-n-sketch (Week 5), which offer visualizations and interactive debugging. On the other hand, Programming by example focuses on generating code by taking input/output pairs as demonstrations. A synthesizer analyzes these pairs and attempts to infer a program that produces the desired results +Live programming is a development paradigm that offers immediate feedback by showing how code behaves as it is written. +This real-time interaction allows developers to observe variable states, program output, +and the effects of changes instantly, which can improve understanding, debugging, and iteration. +The concept encompasses a range of tools, from simple REPLs (Read-Eval-Print Loops) to more advanced systems +like PythonTutor ([Week 4](https://pythontutor.com/)) and +Sketch-n-Sketch ([Week 5](https://ravichugh.github.io/sketch-n-sketch/)), +which provide visualizations and interactive debugging. -In SnipPy, live programming is facilitated by projection boxes, which dynamically display the current state of program variables beside each line of code. Programmers can step through the code line by line and observe how the program state changes in real time. This visualization is a powerful aid in writing code, debugging, and understanding new code, as it allows developers to reason visually about how their code manipulates data. (INSERT EXAMPLE IMAGE FOR PROJECTION BOXES) +Despite its benefits, live programming faces challenges such as cognitive overload from constant updates, +scalability to larger programs, and limited support for abstract or complex problem-solving. +These challenges have inspired tools that combine live programming with other paradigms to enhance usability +and functionality. -Building upon this snippy now integrates programming by example into projection boxes. On a new line of code, the user can provide input/output pairs into a projection box and the tool will try to synthesize a matching expression. +One such tool is **SnipPy**, which integrates live programming with programming by example (PBE). +By doing so, it not only offers real-time feedback through **projection boxes** but also uses **input/output pairs** +to synthesize code on the fly. This blog explores how SnipPy works, where it excels, and where it struggles, +while situating it in the broader context of live programming. -The inner workings of the synthesizer are quite easy to understand. It has an abstract grammar that -represents the possible expressions that can be generated (INSERT EXAMPLES OR EXAMPLE IMAGE). -Note that it is limited to a subset of Python - it can only operate on -integer and string types and functions whose arguments and return values are of these types. -The synthesizer then recursively generates expressions by expanding the grammar rules, until it finds an expression that matches the types of the input/output pairs. If expression correctly matches the actual pairs, it is returned to the user, otherwise the synthesizer will continue searching for a correct expression until it runs out of time. -To ensure meaningful results, the synthesizer uses a number of heuristics to guide the search, for details see the paper.(NOTE FOR REVIEW: I think going into more technical detail here is not fitting for a blogpost - what do you think?) +## How Live Programming Works -Now that we know how SnipPy works, let’s dive into a few examples to see where it shines and where it might run into some challenges: First, lets consider a program that abbreviates names. We give the synthesizer the input/output pairs -- "Adam Smith" -> "A.S." -- "Augusta Ada King" -> "A.A.K." -- "Alan Turing" -> "A.T." +The core idea of live programming is **immediacy**: as developers write code, the environment updates to reflect changes in the program state. +This approach eliminates the traditional compile-run-debug loop, providing continuous feedback. +Modern tools like **projection boxes** take this a step further by dynamically displaying variable +states next to the corresponding lines of code. +This helps developers visually track how data flows through their program in real time. +For example, in a tool like PythonTutor, users can step through code execution line by line and see +visualizations of how variables evolve. + +These capabilities make live programming particularly useful for: +- Debugging: Quickly identifying where and why code behaves unexpectedly. +- Learning: Helping new programmers understand how individual lines of code affect program behavior. +- Incremental Development: Allowing developers to experiment with changes and immediately observe the results. + +However, as program complexity increases, the constant feedback can become overwhelming, +and existing tools often struggle to scale their visualizations or interactivity effectively. + +## SnipPy: A Fusion of Live Programming and Programming by Example + +SnipPy builds upon live programming by integrating it with programming by example (PBE). This combination enables users to both observe program execution through projection boxes and generate new code expressions by providing input/output (I/O) pairs. + +### How It Works + +- **Projection Boxes:** SnipPy uses projection boxes to display variable states beside each line of code in real time. This allows developers to see how program data evolves as they step through the code. For example, if a loop iterates through a list, the projection box shows how the list’s elements are accessed and modified. (INSERT EXAMPLE IMAGE FOR PROJECTION BOXES) +- **Code Synthesis via I/O Pairs:** On a new line of code, users can provide I/O pairs in a projection box, and SnipPy’s synthesizer attempts to generate a Python expression that matches the given behavior. + +#### The Synthesizer + +The synthesizer operates on a restricted subset of Python, limited to integer and string types +and functions with compatible arguments and return values. Using an **abstract grammar**, +it recursively generates candidate expressions until it finds one that matches the I/O pairs. +If no match is found within a time limit, the synthesis fails. To improve efficiency, +the synthesizer uses heuristics to guide its search. + +### Examples + +Let’s explore a few examples to see SnipPy in action and examine its strengths and limitations. + +#### Example 1: Abbreviating Names + +Input/Output Pairs: +- `"Adam Smith"` → `"A.S."` +- `"Augusta Ada King"` → `"A.A.K."` +- `"Alan Turing"` → `"A.T."` + +**Synthesized Code:** ```python -#Synthesized code: abbreviation = ".".join([var[0] for var in name.split(" ")]) -``` -This is a concise and correct solution to the problem. -Next, let’s consider an example where we extract the domain name from an email address. Again we provide the synthesizer with input/output pairs: -- user1@example.com -> example.com -- john@ethz.ch -> ethz.ch -- jane@gmail.com -> gmail.com -- food@mensacorp.org -> mensacorp.org - +``` +This is a concise and correct solution that demonstrates SnipPy’s ability to handle string manipulations effectively. + +#### Example 2: Extracting Domain Names from Email Addresses + +Input/Output Pairs: +- `"user1@example.com"` → `"example.com"` +- `"john@ethz.ch"` → `"ethz.ch"` +- `"jane@gmail.com"` → `"gmail.com"` +- `"food@mensacorp.org"` → `"mensacorp.org"` + +**Synthesized Code:** ```python -#Synthesized code:¨ - splitchar = "@" - domain = email[1:-1][email.find(splitchar):-1] + email[-2:len(email)] -``` -This solution, while less elegant than the first one, is still correct. However we can see that the synthesizer has some limitations. First of all, the splitchar has to be user-defined, else synthesis fails. Secondly, the solution for some reason first removes the first and last character of the email before extracting the domain, and then adds them back in. This is obviously unnecessary and is an artifact of the synthesizers enumeration based approach - which is not able to reason about the logic of the problem. -Lastly, lets consider a program that takes a list of integers and returns the last element. -- [1, 3, 5, 6] -> 6 -- [3, 2, 7, 5] -> 5 +splitchar = "@" +domain = email[1:-1][email.find(splitchar):-1] + email[-2:len(email)] +``` +While this solution works, it is unnecessarily convoluted. For example, +it removes and re-adds parts of the string redundantly, +reflecting the limitations of SnipPy’s enumeration-based synthesis. + +#### Example 3: Returning the Last Element of a List + +Input/Output Pairs: +- `[1, 3, 5, 6]` → `6` +- `[3, 2, 7, 5]` → `5` +**Synthesized Code:** ```python -#Synthesized code: - last_elem = len(input_list) + 2 // min(input_list) -``` -What? This is absolutely not what we were looking for. (a very simple solution would be ```input_list[-1]```) When doing the math, we can see that the expression is indeed correct for our input/output pairs. However, it is not a correct solution to the problem. This shows that the synthesizer is very bad at using list indeces. Diving deeper into more examples could reveal even more limitations of the synthesizer. +last_elem = len(input_list) + 2 // min(input_list) +``` +This output, while mathematically correct for the given pairs, +clearly fails to generalize to the problem’s intent. The synthesizer struggles to reason about list indices, +a limitation of its grammar-based approach. + +## Limitations of SnipPy and Opportunities for Improvement + +While SnipPy demonstrates the potential of integrating live programming with PBE, +its limitations highlight broader challenges in live programming: + +1. **Cognitive Overload:** Projection boxes can clutter the interface in more complex programs, making it hard to focus. +2. **Quality of Synthesized Code:** The synthesizer often generates correct but inelegant solutions, or even brittle ones that fail for broader input ranges. +3. **Limited Scope:** The tool’s restricted grammar and data types reduce its utility for real-world applications. -While SnipPy presents an innovative fusion of live programming and programming by example, its current synthesizer -struggles to meet the practical needs of developers. -However, the landscape of code synthesis has changed quite dramatically since the publication of this tool. -New large language models like GPT, BERT or Claude are able to reason about code and problems in a much more accurate way than the limited enumeration approach of SnipPy. It would be interesting to see how the interface of SnipPy could be combined with an LLM based synthesizer. -Tools like github copilot function in a similar way - where the user provides a prompt and the tool is able to generate code in-editor. -A promising step in this direction is LEAP, a tool that integrates LLM-based code synthesis with live programming. LEAP generates multiple code completions for a given problem, allowing users to preview the options. Upon selection, LEAP displays the chosen code with corresponding projection boxes based on sample inputs, which visualize the program state. Unlike SnipPy, LEAP primarily uses projection boxes for code verification rather than problem specification. This shift aligns better with the strengths of LLMs, which can reason about prior code context and high-level problem requirements. +## Future Directions + +The advent of large language models (LLMs) like GPT or Claude offers a promising path forward. +These models can reason about broader code contexts and generate more accurate and elegant solutions +than enumeration-based synthesizers. + +One tool exploring this potential is **LEAP**, which combines LLM-based synthesis with live programming. +LEAP generates multiple code completions for user evaluation and visualizes the corresponding program state +using projection boxes. Unlike SnipPy, LEAP focuses more on validating and refining code rather than purely generating +it. This aligns better with LLMs’ strengths in reasoning about high-level problem context. + +## Conclusion + +SnipPy illustrates the exciting possibilities of combining live programming with programming by example, +but its limitations underscore the need for more advanced synthesis techniques. Looking ahead, +integrating LLMs into live programming tools could enhance their usability and effectiveness, +allowing developers to harness real-time feedback without being hindered by the system’s constraints. + +As tools like LEAP demonstrate, the future of live programming lies in striking a balance between automation, +interactivity, and developer control. By addressing current challenges, +these tools can unlock new possibilities for faster, smarter, and more intuitive coding experiences. + -In conclusion, SnipPy demonstrates the potential of integrating live programming with programming by example, while its limitations reveal opportunities for more advanced synthesis techniques to enhance developer tools. ### Sources + #### SnipPy -* [Snippy](https://snippy.goto.ucsd.edu/editor): Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020, October). Small-step live programming by example. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (pp. 614-626). +- [Snippy](https://snippy.goto.ucsd.edu/editor): Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020, October). *Small-step live programming by example*. In *Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology* (pp. 614-626). #### Live Programming -* [Projection Boxes](https://cseweb.ucsd.edu/\~lerner/papers/projection-boxes-chi2020.pdf): Lerner, S. (2020, April). Projection boxes: On-the-fly reconfigurable visualization for live programming. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-7). -* [PythonTutor](https://pythontutor.com/): Guo, P. J. (2013, March). Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education (pp. 579-584). -* [Sketch-n-sketch](https://ravichugh.github.io/sketch-n-sketch/): Hempel, B., Lubin, J., & Chugh, R. (2019, October). Sketch-n-sketch: Output-directed programming for svg. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (pp. 281-292). +- [Projection Boxes](https://cseweb.ucsd.edu/~lerner/papers/projection-boxes-chi2020.pdf): Lerner, S. (2020, April). *Projection boxes: On-the-fly reconfigurable visualization for live programming*. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems* (pp. 1-7). +- [PythonTutor](https://pythontutor.com/): Guo, P. J. (2013, March). *Online python tutor: embeddable web-based program visualization for cs education*. In *Proceedings of the 44th ACM technical symposium on Computer science education* (pp. 579-584). +- [Sketch-n-Sketch](https://ravichugh.github.io/sketch-n-sketch/): Hempel, B., Lubin, J., & Chugh, R. (2019, October). *Sketch-n-sketch: Output-directed programming for svg*. In *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology* (pp. 281-292). -#### LLM based code synthesis combined with live programming -* [LEAP](https://doi.org/10.1145/3613904.3642495): Kasra Ferdowsi, Ruanqianqian (Lisa) Huang, Michael B. James, Nadia Polikarpova, and Sorin Lerner. 2024. Validating AI-Generated Code with Live Programming. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 143, 1–8. https://doi.org/10.1145/3613904.3642495 \ No newline at end of file +#### LLM-based Code Synthesis Combined with Live Programming +- [LEAP](https://doi.org/10.1145/3613904.3642495): Ferdowsi, K., Huang, R., James, M. B., Polikarpova, N., & Lerner, S. (2024). *Validating AI-Generated Code with Live Programming.* In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24)*. Association for Computing Machinery, New York, NY, USA, Article 143, 1–8. \ No newline at end of file